Why doesn't Ext4 cache directory size?

Question

Once again I'm cleaning up my desktop because I'm running out of HDD space and everytime I call 'du -sh ~', it takes a really long time to calculate the current size of the directory. From my perspective, I can't see a reason why wouldn't Ext4 filesystem cache the information about the directory size and update it whenever a file is created/deleted. The information would then be exposed to the system, and eventually to the user. Of course it would need to propagate the directory size data up the directory tree, but I can't imagine a situation in which on a normal Linux desktop it would cause any problems. Where did I get this wrong?

Answer 1

A simple cache wouldn't work. A cache is about checking if you already have the answer and only reprocess if you don't. But in this case, a single missing entry would make others useless. So it would have to keep all directory sizes updated all the time.

Also don't underestimate the possible impact of your proposal. Back when journalling filesystems were new, there was a lot of opposition because updating the journal was too expensive. Also most filesystems allow options like noatime, nodiratime and relatime that reduce these kinds of medatata updating. Note that all these (journals and time updatings) are bound in time, they all take a specific number of block accesses (and are usually 'hidden' by advanced IO scheduling), but updating the size of every directory up the path means an unknown amount of accesses.

Finally, in POSIX filesystems, there's no real 'containing directory'. A file entry on a directory points to an inode (the disk structure that holds the file information), but there's no reference from the inode back to the directory. This allows the 'hard link' feature, where more than one entry (usually in different directories) points to the same inode. Even if you kept a list of directories that point to the inode, you're multiplying the (already big) number of updates. Worse, now you have to keep track if you've already updated each directory, since at some point up the chain you'll get a shared ancestor, which shouldn't count twice the updated. Or should it? maybe you'll have to keep two sizes on each directory, one that counts all 'real' files, and other that counts each time it appears....

It doesn't seem so useful after all.

Answer 2

There's also another reason. du doesn't take into account if part of the subsystem is on another volume that is mounted in the directory you're measuring.

Thus, even if usage were evaluated per volume, du wouldn't exploy it. On the other hand, df makes use of such information....