Linux temporarily thinks disk is full

Question

I’m working on a Linux server with CentOS 6.5 and a NAS NFS on a QDR Infiniband network. I am running a bash script that basically creates a directory, makes symlinks inside of it, and cats together one small file into each directory. It does this for a few hundred directories.

I noticed in the output log that one of the symlinks and the subsequent cat failed to run, claiming that the disk was full. It was quite clearly not. Running that same script for a few thousand directories, I began getting a very large number of these messages. I checked and the disk looked to be full, so I immediately killed my script, but then after a few minutes, the disk returned to normal.

Here are the sequential df commands that I saw, the first while the script was running, the second just after killing it, and the third some seconds later /home3(a NAS) is the one I’m working on:

[swfl 07:40:56 JPM]$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_misisss6-lv_root  135G   25G  104G  19% /
tmpfs                             12G     0   12G   0% /dev/shm
/dev/sda1                        485M   69M  392M  15% /boot
misisss-nasib3:/home              26T   26T  1.0M 100% /home3
misisss-nas1:/shared              77G  437M   73G   1% /shared
misisss-nasib2:/home              15T   15T   95G 100% /home2
You have new mail in /var/spool/mail/swfl

[swfl 07:41:39 JPM]$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_misisss6-lv_root  135G   25G  104G  19% /
tmpfs                             12G     0   12G   0% /dev/shm
/dev/sda1                        485M   69M  392M  15% /boot
misisss-nasib3:/home              26T   26T  1.0M 100% /home3
misisss-nas1:/shared              77G  437M   73G   1% /shared
misisss-nasib2:/home              15T   15T   94G 100% /home2

[swfl 07:41:58 JPM]$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_misisss6-lv_root  135G   25G  104G  19% /
tmpfs                             12G     0   12G   0% /dev/shm
/dev/sda1                        485M   69M  392M  15% /boot
misisss-nasib3:/home              26T   21T  4.2T  84% /home3
misisss-nas1:/shared              77G  437M   73G   1% /shared
misisss-nasib2:/home              15T   15T   93G 100% /home2

At the time, there was relatively little CPU usage on most cores and low to moderate disk usage. I don’t have monitoring software running, so I can’t give IOps figures or anything of the sort, but I’ve done work similar to this but at much higher intensity without issue.

In short, it’d be very difficult to believe I was overwhelming any part of the system with the work being done. Breadcrumbs as to where to search for issues?

UPDATE 1 Running watch 'df -h; df -i' to keep track of inodes and disk usage, I can see disk space dropping precipitously (things are OK for ~5 seconds, then several TB disappear over 10-20 seconds), until I start getting the errors, but in odes isn’t dropping nearly as much.

I can see in odes does have fairly high utilization (30-70%), though. I have ~16 billion inodes and am creating ~40000 files/directories. After I kill the process, the disk space will start climbing slowly (a few GB) for 10-20 seconds, then will jump back up a few TB to what it was originally.

Answer 1

By noticing that the disk space freed up on a 5-minute cycle, we were able to identify the problem. It's a behavior that may be unique to the file system we're using, the XFS file system.

XFS allows you to specify a pre-allocated file size. We mounted the file-system with an allocsize=1G, given that this file system was built with large files in mind and we wanted to avoid fragmentation. You can also specify an update frequency for the file system to then go back and revise utilization from the pre-allocated values. The default value of 5 minutes was why we saw this cyclical behavior. Some related info on this behavior can be found here.

So, when I created a file, then performed a cat to it, this second action on the file was enough to trigger the system to pre-allocate 1 GB to that file. Looping over several thousand such files at a very high speed thus resulted in all the disk space appearing to be exhausted before the storage unit could adjust these allocations.

We have removed that mount option to allow the file system to go with the dynamic pre-allocation, which is smarter about smaller files and file system available capacity.