Use rmdir to delete lots of empty folders within a folder

Question

I have a folder which should at any time have less than 500k folders. These folders are created dynamically, and when the files (images) inside them are deleted by some script, the folder itself should also be deleted. Due to a bug, the deletion of the empty folders was not done, and we now ended up with millions of empty folders...

So at the moment I have about 500k folders with files in them and besides them millions of empty folders.

Because there are so many dirs in there even a simple ls | wc -l takes hours so can't even find the exact number.

I could run a find command: find . -maxdepth 1 -type d -empty; and then pipe it with a rmdir; but again, the find takes ages (or even use the -delete option that find offers)

I then got the idea to use rmdir --ignore-fail-on-non-empty /path/to/huge/folder/*; but I am not sure on its efficiency. Basically this command only deletes the empty directories in my folder, which means the ones which have files should be safe and not deleted. Then the '--ignore-fail-on-non-empty' option ignores the hundreds of thousands or warnings for the dirs that are actually not empty.

I have quickly created about 100k empty folders on my local, then 'touched' a few files in some of them, and then used the rmdir command. While it took 2 minutes to create 100k empty folders, it took 2 seconds(!) to delete the empty ones. So my hopes are high, but I'm not sure how it will handle about 10 million folders... (I guess I could use ionice to stop the system from locking down in case it would actually take ages)

If anyone has any ideas/advice, or faced something like this before, can you please point me to the best way of approaching my issue. Or even to tell me which way is better to go: find or rmdir...

EDIT: I forgot to mention the folder structure before. The folders within the main folder that have files in them, have no more folders in them. The other folders are empty. So all those millions of folders cam either be empty, or have files in them, but no more folders. So basically there is only one level of folders in the tree structure.

Answer 1

Without actually knowing how many directories you have, it's difficult to predict how which approach will behave, but I can tell you this:

find . -maxdepth 1 -type d -empty -delete

is a very bad idea. find is excruciatingly slow when it comes to deleting things. The times I've benchmarked it, the -delete switch was 40 to 50 times slower than the optimal approach.

Likewise,

rmdir --ignore-fail-on-non-empty /path/to/huge/folder/*

should be suboptimal. The shell will expand the glob to all directory entries (files and folders) and, before even passing them to rmdir, sort them alphabetically, which will take time and is completely unnecessary.

The combined approach should be the best:

find . -maxdepth 1 -type d -empty -exec rmdir {} +

Note that the plus sign at the end avoids calling rmdir for every single directory. Instead, find gathers as many directory names as it can fit in 128 KiB (because of the kernel limit for command line arguments) and executes rmdir for all of them at once.

Combining the above with ionice couldn't hurt. If there are "only" a few million directories, I doubt it will be necessary. But since you don't know the exact number, better safe than sorry.