16

Yesterday, I have deleted 71 GB of files on my home/media server.

  • Free space before: 117 GB
  • Free space after: 126 GB

So, instead of having 71 GB of additional free space, I only had 9 GB. I have double-checked that no files were open and I really have deleted 71 GB and free space really increased only 9 GB.

I also tried to sync, but no effect.

This is not the first time it happens. Indeed, I have seen this behaviour since years every now and then. First on ext3, now on ext4.

When this happens, I can reclaim the free space by unmounting and then remounting the filesystem. In these cases, unmounting takes up to 2 minutes instead of nearly no time.

Nowadays, I can not easily unmount and remount the filesystem, because it is permanently busy from my video recorder software, the owncloud server for my family, and a few more services I did not have so far. And I don't want to get up at 3 o'clock in the night just to unmount and remount.

No, the at utility won't do because one of the services performs non resumable long running tasks and thus needs a manual state check to find a good moment when it can be shut down, i.e. a task was just finished.

But this morning, I noticed that the space has been freed overnight. Looks to me as if there was some kind of cleanup, and this may be the same thing that takes the additional time when unmounting.

So far, I only noticed this behaviour when deleting a large amount of data. In the other hand, I am not sure if it happens regularly and the difference is just too small to notice.

The filesystem was created with 0% reserved for root (mkfs -m 0). According to fsck -f (I'm doing this always between unmount and remount), the filesystem is not corrupted, and according to S.M.A.R.T. diagnostics extended test, the hardware is also OK.

tune2fs 1.42 (29-Nov-2011)  
Filesystem volume name:   bigdata  
Last mounted on:          /bigdata  
Filesystem UUID:          6aebd17a-e064-41dc-9c68-c9a3acbe4f66  
Filesystem magic number:  0xEF53  
Filesystem revision #:    1 (dynamic)  
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize  
Filesystem flags:         signed_directory_hash   
Default mount options:    user_xattr acl  
Filesystem state:         clean  
Errors behavior:          Continue  
Filesystem OS type:       Linux  
Inode count:              121413632  
Block count:              485645568  
Reserved block count:     0  
Free blocks:              29081276  
Free inodes:              121382378  
First block:              0  
Block size:               4096  
Fragment size:            4096  
Reserved GDT blocks:      908  
Blocks per group:         32768  
Fragments per group:      32768  
Inodes per group:         8192  
Inode blocks per group:   512  
Flex block group size:    16  
Filesystem created:       Tue Dec 25 23:42:35 2012  
Last mount time:          Fri Jan  3 17:37:36 2014  
Last write time:          Fri Jan  3 17:37:36 2014  
Mount count:              37  
Maximum mount count:      -1  
Last checked:             Thu Apr 18 17:03:40 2013  
Check interval:           0 (<none>)  
Lifetime writes:          14 TB  
Reserved blocks uid:      0 (user root)  
Reserved blocks gid:      0 (group root)  
First inode:              11  
Inode size:           256  
Required extra isize:     28  
Desired extra isize:      28  
Journal inode:            8  
Default directory hash:   half_md4  
Directory Hash Seed:      be2b977e-5127-4843-9123-fe33b6d7b573  
Journal backup:           inode blocks  

So here are my 2 questions:

  1. What is happening here? Why is space freed at umount or with a delay instead of immediately at delete?
  2. Is there something I can do to trigger freeing the space right now without unmounting and remounting?
15
  • This smells a lot to me like the file(s) are still open by something. How are you confirming that none of the files are open after deletion? lsof | grep -i deleted is usually the best way to check this.
    – Garrett
    Jan 14, 2014 at 1:22
  • 2
    Normally you can not umount a filesystem when files are open. So, if umount succeeds, there are no open files. Except yesterday, I always did the umount, re-mount cycle when I noticed this.
    – Markus N.
    Jan 14, 2014 at 1:37
  • This is indeed the case. What are your ext4 mount options?
    – Garrett
    Jan 14, 2014 at 3:13
  • noatime,defaults
    – Markus N.
    Jan 14, 2014 at 18:31
  • Do you have some sort of backup system running that might be keeping copies of the files, or alternatively hardlinks to them?
    – derobert
    Jan 15, 2014 at 22:18

1 Answer 1

10

There are two factors that may be interacting.

  • Unlike Windows you can delete files that are open. If you delete a movie that is being streamed, it will be removed from the directory, but will still exist as a file until the streaming program closes it. Once the streaming software closes it, the space will be released. The fuser -m command can be used to find the process id of any processes with open files. Some programs may not immmediately close files when they are done with them.

  • These are both journaled file systems. Changes are written to a journal and then committed. It can take a while for the changes to commit. The operating system will often cache disk changes and only commit changes to disk periodically. Running the sync command should flush any pending changes to disk. Mounting the disk with the sync option will improve the speed to disk, but works the disk harder.

5
  • 1
    Ye, I know I can delete files that are open. I know about the relations between directory entries and inodes. But I am sure the files were not open,otherwise I would not be able to umount the filesystem... But your second item appears interesting. Can it really take more than 30 minutes for deleted files to commit ? 30 minutes is the time between deleting the files and going to bed the day before yesterday, so I can not tell how long it really took.
    – Markus N.
    Jan 14, 2014 at 18:35
  • 1
    @MarkusN. Some operating systems cache a lot of file system data. If the space isn't needed, they may write sufficient transaction log data to ensure the space is freed, but take their time freeing the space. If you unmount a USB key after writing a lot of data, it can take a long time for the data to flush before you can remove it. It is a trade off between securely writing to disk and not delaying other actions.
    – BillThor
    Jan 15, 2014 at 14:04
  • Thanks for the edit. But as you see in my original question, I already tried sync with no effect.
    – Markus N.
    Jan 16, 2014 at 10:04
  • 1
    @MarkusN. I don't think sync is as effective as it once was. It likely just ensures the journal data is written, not necessarily that all changes are committed. Unmount/mount will force the journal to be applied. I don't know the scheduling priority for deletions, but I would expect it to be relatively low. Exploring the code may answer more of your questions.
    – BillThor
    Jan 17, 2014 at 0:16
  • Sounds reasonable to me.
    – Markus N.
    Jan 17, 2014 at 11:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .