1

I have a folder which should at any time have less than 500k folders. These folders are created dynamically, and when the files (images) inside them are deleted by some script, the folder itself should also be deleted. Due to a bug, the deletion of the empty folders was not done, and we now ended up with millions of empty folders...

So at the moment I have about 500k folders with files in them and besides them millions of empty folders.

Because there are so many dirs in there even a simple ls | wc -l takes hours so can't even find the exact number.

I could run a find command: find . -maxdepth 1 -type d -empty; and then pipe it with a rmdir; but again, the find takes ages (or even use the -delete option that find offers)

I then got the idea to use rmdir --ignore-fail-on-non-empty /path/to/huge/folder/*; but I am not sure on its efficiency. Basically this command only deletes the empty directories in my folder, which means the ones which have files should be safe and not deleted. Then the '--ignore-fail-on-non-empty' option ignores the hundreds of thousands or warnings for the dirs that are actually not empty.

I have quickly created about 100k empty folders on my local, then 'touched' a few files in some of them, and then used the rmdir command. While it took 2 minutes to create 100k empty folders, it took 2 seconds(!) to delete the empty ones. So my hopes are high, but I'm not sure how it will handle about 10 million folders... (I guess I could use ionice to stop the system from locking down in case it would actually take ages)

If anyone has any ideas/advice, or faced something like this before, can you please point me to the best way of approaching my issue. Or even to tell me which way is better to go: find or rmdir...

EDIT: I forgot to mention the folder structure before. The folders within the main folder that have files in them, have no more folders in them. The other folders are empty. So all those millions of folders cam either be empty, or have files in them, but no more folders. So basically there is only one level of folders in the tree structure.

  • How many directories deep does this go? – UtahJarhead Jan 3 '14 at 17:46
  • Related to UtahJarhead's question, if the directory structure ends up being too deep, and even if it doesn't but you'd feel more comfortable splitting the job into pieces, you could start your rmdir command half, or two-thirds the way into your directory tree. – panhandel Jan 3 '14 at 18:45
  • thanks for the answers. I updated the question with the folder structure. – Titi Jan 3 '14 at 21:03
0

Without actually knowing how many directories you have, it's difficult to predict how which approach will behave, but I can tell you this:

find . -maxdepth 1 -type d -empty -delete

is a very bad idea. find is excruciatingly slow when it comes to deleting things. The times I've benchmarked it, the -delete switch was 40 to 50 times slower than the optimal approach.

Likewise,

rmdir --ignore-fail-on-non-empty /path/to/huge/folder/*

should be suboptimal. The shell will expand the glob to all directory entries (files and folders) and, before even passing them to rmdir, sort them alphabetically, which will take time and is completely unnecessary.

The combined approach should be the best:

find . -maxdepth 1 -type d -empty -exec rmdir {} +

Note that the plus sign at the end avoids calling rmdir for every single directory. Instead, find gathers as many directory names as it can fit in 128 KiB (because of the kernel limit for command line arguments) and executes rmdir for all of them at once.

Combining the above with ionice couldn't hurt. If there are "only" a few million directories, I doubt it will be necessary. But since you don't know the exact number, better safe than sorry.

  • Thanks for the useful command. I am currently running it and it is deleting on average about 10k empty folders every 10 minutes. Fortunately, I miscalculated the number of empty dirs, and there were only just above 1 million folders in that main folder (not 10 million). Now I just have to play the waiting game... – Titi Jan 6 '14 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.