When running a find command on a very large directory I discover that find's use of memory spikes much more than I think it would. I've found it very hard to search for solutions due to the overloaded use of the word "find" ;) This is the command:

find /var/dir -size +500k -iname "pattern_*" -mtime +90 -type f -exec rm {} \; -print

Does find keep the entire file list for a directory in memory as it processes it? It seems like it could just examine one entry and move on. The directory I'm dealing with has ~2 million files in it and I'm starting to run into memory limit issues now.

Are they any ways to use find that'll reduce its memory usage? Any alternatives out there that behave better? I just need to delete files in this directory that meet a couple critera and then print them. (My last resort solution is I'll just write my own program to perform what I need by streaming in the results of ls...)

  • I don't have evidence, but it seems highly unlikely that find would waste memory like that. Are you sure find is the real cause of the memory spike? Also, I doubt it will make a difference, but instead of -exec, it's simpler to use -delete, that is, find ... -delete -print. Lastly, instead of parsing the result of ls which is usually not recommended, this could be done fairly easily with a Perl one-liner. – janos Feb 2 '17 at 8:31
  • Let me give using -delete a try! I could go the perl oneliner route, but will have to make sure I process the file list in a stream. – Chris Feb 3 '17 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.