Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

When I run df it shows the root device is full.

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  9.4G     0 100% /

I looked at the inode usage and there is pretty much space available for root device

Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1               640K    103K    538K   16% /

But, when I run the du command, it shows I have used only 2G out of 9.9G.

ip-XXX-XXX-XXX-XXX:/$ du -xh --max-depth=1
14M ./etc
4.0K ./mnt
96K ./tmp
3.5M ./bin
0   ./sys
964K ./boot
4.0K ./srv
0   ./dev
55M ./lib
25M ./root
1.1G ./usr
4.0K ./opt
846M ./var
4.3M ./sbin
23M ./home
16K ./lost+found
0   ./proc
2.0G .

It just driving me crazy and interesting too. This is big problem for us since the root disk / is full and some of the function in our site is failing.

Please help me resolve (also understand) this problem.

Thanks.

share|improve this question

migrated from stackoverflow.com Jun 30 '11 at 8:48

This question came from our site for professional and enthusiast programmers.

2  
See linux free disk space confusion and other +df +du questions at Unix & Linux. –  Gilles Jun 30 '11 at 7:20
    
@Gilles like you said, I ran du -x / and I see only 2G is used and I calculated the inode size which is 160M. It helped me understand the stuff but I just want to resolve this problem. –  RakeshS Jun 30 '11 at 7:46
    
@Gilles help me find the culprit, thanks. –  RakeshS Jun 30 '11 at 7:47
    
Did you run du as root? Otherwise it can only report on the files you can access. –  Gilles Jun 30 '11 at 7:59
    
@Gilles I am running as root –  RakeshS Jun 30 '11 at 8:02
show 2 more comments

4 Answers

up vote 4 down vote accepted

When files are deleted in *nix, they continue to live on disk (and take up disk space) for as long as a process has them open. It's fairly common to take advantage of this to "secure" temp files by creating them with a small size, deleting them, and then using the deleted file to store data without having to worry about other processes (easily) getting access to it, so the amount of space in deleted files can grow pretty large if, say, a temp database or multimedia editing session is being handled in this way. Another possibility for how you could have so much "lost" space would be if the system has been upgraded (multiple times) without rebooting or restarting programs, resulting in all your old .so libraries being held open by programs which were started prior to the upgrade and are still running.

df sees the space used by these files because it just looks at how much space is allocated on the device, but du doesn't see them because there aren't any corresponding directory entries.

"Hidden" used space like this can only be freed up when processes that have deleted files open close them. You can find these processes with the fuser command and terminate them (or, for many daemons, send a signal telling them to close and re-open any open files).

share|improve this answer
    
thanks, good info, something (more) I understand today.But to find a hidden used space, I try to run the fuser command to see all the files that are being used by a process but unlinked - I could not find any. Do you have any command or direction that leads me to find it? This is the command I use fuser -v -a / –  RakeshS Jun 30 '11 at 11:03
    
I had to reboot it to get rid of the hidden space but I couldn't find a proper solution to remove those hidden files. –  RakeshS Jul 5 '11 at 8:01
add comment

On my side, I simply restarted syslogd to get the disk space back. I had 3GB missing ! My server was up and running for 250 days old.

share|improve this answer
add comment

There have been times when if the disk gets full, it can then be confused till reboot/remount that the disk is still full, even when you've deleted a load of files.

share|improve this answer
    
I can not reboot since it's a production site. But I am looking for a solution which can help me find these hidden space and bring it back. –  RakeshS Jun 30 '11 at 9:01
    
It maybe production but sometimes a reboot is the only answer. Thats the unfortunate problem of it being your root disk and all your OS is on it. Which is why many advocate the use of tmp, var, home etc being on other disks, as they can then be remounted. It seems to be more common that the OS doesnt realise the space is available to the root disk. –  BugFinder Jul 2 '11 at 10:50
add comment

There is a way to cleanup space without restarting application. Here are the details:

  1. Let's say you have process foo running and creating a 2 GB file named abc.log. Now say this abc.log is deleted by someone else.

  2. Get foo's pid (let's say 123). So /proc/123/fd will show list of file descriptors opened by foo. One with abc.log will show as deleted. Let's say fd of abs.log is 111. If you run less /proc/123/fd/111, it will still show you all that 2 GB of data.

  3. Run echo " " > /proc/123/fd/111. This will overwrite the contents with an empty string. After this command if you try df it will show an additional 2 GB recovered by cleaning abc.log.

That's it. I tried this on CentOS and it works.

share|improve this answer
    
This is a useful thing to know. But it's not an answer to the question. –  Isaac Rabinovitch Dec 3 '12 at 22:09
    
sorry my ans was more targetted towards cleaning up disk space if you know the process id who deleted the file. For this specific case, you have to go through all the files in /proc/[0-9]*/fd, grep the deleted ones, and follow the logic mentioned above. –  Kaustubh Sathe Dec 5 '12 at 22:34
    
If you are really curious which file was causing the the disk space leak, clean on deleted file at a time, check df output every time and log this information. –  Kaustubh Sathe Dec 5 '12 at 22:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.