Once again I'm cleaning up my desktop because I'm running out of HDD space and everytime I call du -sh ~, it takes a really long time to calculate the current size of the directory.

From my perspective, I can't see a reason why wouldn't ext4 filesystem cache the information about the directory size and update it whenever a file is created/deleted. The information would then be exposed to the system, and eventually to the user. Of course it would need to propagate the directory size data up the directory tree, but I can't imagine a situation in which on a normal Linux desktop it would cause any problems.

Where did I get this wrong?

  • The kernel does cache some data in its file subsystem. Individual filesystems (like Ext4) might do some more. But the metadata on your disk probably won't fit into available RAM.... Nov 5 '12 at 14:12
  • It could as well cache it on HDD.
    – d33tah
    Nov 5 '12 at 14:14
  • No, the cache inside the HDD is dealt with the firmware inside the HDD, and some SATA commands like TRIM. The kernel can only send SATA commands to the hard disk drive. Nov 5 '12 at 14:19
  • And you could always improve Ext4 (which is free software) to suit your own needs, or even implement a new file system. Take advantage that GNU/Linux is free software. Nov 5 '12 at 14:25
  • 1
    I actually meant cache as real data on HDD. Just like directory's name is stored there, I don't see why its size wouldn't. @BasileStarynkevitch, the reason I didn't do it is because I have absolutely no idea about FS development. I'm just trying to understand.
    – d33tah
    Nov 5 '12 at 14:25

A simple cache wouldn't work. A cache is about checking if you already have the answer and only reprocess if you don't. But in this case, a single missing entry would make others useless. So it would have to keep all directory sizes updated all the time.

Also don't underestimate the possible impact of your proposal. Back when journalling filesystems were new, there was a lot of opposition because updating the journal was too expensive. Also most filesystems allow options like noatime, nodiratime and relatime that reduce these kinds of medatata updating. Note that all these (journals and time updatings) are bound in time, they all take a specific number of block accesses (and are usually 'hidden' by advanced IO scheduling), but updating the size of every directory up the path means an unknown amount of accesses.

Finally, in POSIX filesystems, there's no real 'containing directory'. A file entry on a directory points to an inode (the disk structure that holds the file information), but there's no reference from the inode back to the directory. This allows the 'hard link' feature, where more than one entry (usually in different directories) points to the same inode. Even if you kept a list of directories that point to the inode, you're multiplying the (already big) number of updates. Worse, now you have to keep track if you've already updated each directory, since at some point up the chain you'll get a shared ancestor, which shouldn't count twice the updated. Or should it? maybe you'll have to keep two sizes on each directory, one that counts all 'real' files, and other that counts each time it appears....

It doesn't seem so useful after all.

  • You touched a very interesting subject there - the shared ancestors, which I understand as symlinks. Never really thought of that.
    – d33tah
    Nov 5 '12 at 14:28
  • 5
    The shared ancestors are not symlinks (which are made using the symlink(2) syscall) but hard links made with the link(2) syscall Nov 5 '12 at 14:37

There's also another reason. du doesn't take into account if part of the subsystem is on another volume that is mounted in the directory you're measuring.

Thus, even if usage were evaluated per volume, du wouldn't exploy it. On the other hand, df makes use of such information....

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.