Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I had a dir fill up with millions of gif images. Too many for rm command.

I have been trying the find command like this:

find . -name "*.gif" -print0 | xargs -0 rm

Problem is, it bogs down my machine really bad, and causes time outs for customers since it's a server.

Is there any way that is quicker to delete all these files...without locking up the machine?

share|improve this question

migrated from stackoverflow.com Nov 24 '13 at 20:39

This question came from our site for professional and enthusiast programmers.

    
I"m at about 6 gb/hr deletion rate using the "nice find" command below. Probably will take 48 hrs straight to get rid of all the files. The reason this happened was b/c a scour script failed. I had surpassed the "event horizon" with rm command, then it ran away. –  Corepuncher Nov 23 '13 at 18:12
3  
Would removing the whole dir not be substantially quicker? Just take out the "good" files before nuking the remaining ones... –  tucuxi Nov 23 '13 at 18:42
    
Well, every file is bad right now, because it was moved to /dir_old , and I remade the /dir. But won't rmdir run into same limitation as rm * ? –  Corepuncher Nov 23 '13 at 19:43
    
@Corepuncher: I would expect that removing the entire directory (as with rm -rf would be faster. It's worth a try. –  Jason R Nov 23 '13 at 19:44
    
I'm currently running "rm -rf" on the dir. It's been running for over 20 min now...no change in disk size yet. But also it didn't automatically return "arguement list too long" yet either. Only problem is, it's really hammering my machine and making other things slow/fail. Not sure how long to let it go. –  Corepuncher Nov 23 '13 at 20:01
show 2 more comments

5 Answers

up vote 35 down vote accepted

Quicker is not necessarily what you want. You may want to actually run slower, so the deletion chews up fewer resources while it's running.

Use nice(1) to lower the priority of a command.

nice find . -name "*.gif" -delete

For I/O-bound processes nice(1) might not be sufficient. The Linux scheduler does take I/O into account, not just CPU, but you may want finer control over I/O priority.

ionice -c 2 -n 7 find . -name "*.gif" -delete

If that doesn't do it, you could also add a sleep to really slow it down.

find . -name "*.gif" -exec sleep 0.01 \; -delete
share|improve this answer
2  
wow... millions of files with a sleep of .1 s ... needs a day for 864000 files. –  glglgl Nov 23 '13 at 16:38
6  
@glglgl All right, smart ass. I changed the timeout. :-P –  John Kugelman Nov 23 '13 at 16:39
26  
The sleep may be a good choice, but nice won't do, since the task here is IO bound, not CPU bound; you may try ionice instead. Notice that if the sleep is too small it will be useless. –  Matteo Italia Nov 23 '13 at 16:40
3  
@glglgl: the point is exactly that if you don't want to cause service disruption on the server you have to go slowly, the time where this code sleeps is there to let the server do actually useful work with the disk . –  Matteo Italia Nov 23 '13 at 16:44
    
I must admit your solution with the sleep is probably the best, better than mine with split and xargs. +1 –  Tomas Nov 23 '13 at 17:15
show 4 more comments

Since you're running Linux and this task is probably I/O-bound, I advise to give your command idle I/O scheduler priority using ionice(1):

ionice -c3 find . -name '*.gif' -delete

Comparing to your original command, I guess this may even spare some more CPU cycles by avoiding the pipe to xargs.

share|improve this answer
    
@Braiam What do you mean? This is not a find ... -exec where that would make sense. –  user2719058 Nov 23 '13 at 21:59
    
Oh, yeah, sorry. My bad. You sure that is efficient, tho? –  Braiam Nov 23 '13 at 22:03
1  
Well, the find(1) documentation claims so. :) And it should be obvious that letting find itself remove files is more efficient than forking a rm command for this. –  user2719058 Nov 23 '13 at 22:15
add comment

No.

There is no quicker way, appart from soft-format of the disk. The files are given to rm at once (up to the limit of the command line, it could be also set to the xargs) which is much better than calling rm on each file. So no, there is definitely no faster way.

Using nice (or renice on a running process) helps only partially, because that is for scheduling the CPU resource, not disk! And the CPU usage will be very low. This is a linux weakness - if one process "eats up" the disk (i.e. works a lot with it), the whole machine gets stuck. Modified kernel for real time usage could be a solution.

What I would do on the server is to manually let other processes do their job - include pauses to keep the server "breathe":

find . -name "*.gif" > files
split -l 100 files files.
for F in files.* do
    cat $F | xargs rm
    sleep 5 
done

This will wait 5 seconds after every 100 files. It will take much longer but your customers shouldn't notice any delays.

share|improve this answer
    
"The files are given to rm at once (up to the limit of the command line"—so when the shell is ordered to rm *, it expands * into the line with all of the filenames and pass it to rm? That's incredibly stupid. Why would shell expand wildcards? –  Joker_vD Nov 23 '13 at 23:15
    
:-D @Joker_vD, are you joking, as your name suggests? :-) –  Tomas Nov 23 '13 at 23:28
1  
@Joker_vD: Compatibility with a Unix decision from 1970 or so. Windows doesn't do it. There, programs can pass wildcards to FindNextFile/FindNextFile, so they get the results one at a time. –  MSalters Nov 23 '13 at 23:30
    
@Tomas Not in this case. Honestly, I can see 2 problems with such design immediately: first, command line isn't rubber; second, the program can't tell if it was called with * or /* and give a doubt to such decision of the user. –  Joker_vD Nov 24 '13 at 15:20
1  
@Joker_vD There are a lot of good things about the shell doing wildcard expansion. It's different from Windows, but don't jump to the conclusion that it's incredibly stupid merely because it's different from what you're used to. If you want to know more, I encourage you to Google it or post a question on the relevant Stack Exchange site. It's a huge derail for this comment area. –  John Kugelman Nov 24 '13 at 15:27
show 1 more comment

Have you tried:

find . -name "*.gif" -exec rm {} +

The + sign at the end will cause find to include more files for the single rm command to be executed. Check this question for more details.

share|improve this answer
    
It execute much faster than -print0 | xargs solution because rm process is not invoked for every file but for large set of them and therefore it is causing lower load. –  Bartosz Firyn Nov 23 '13 at 16:42
    
@JohnKugelman You are correct, but it's a GNU extension that isn't always available with the native find command. –  CodeGnome Nov 23 '13 at 17:04
    
OK, interesting, but this is quite new thing (as well as -delete) which doesn't always have to be there.. –  Tomas Nov 23 '13 at 17:09
    
However this certainly brings nothing better compared to the OP's solution. –  Tomas Nov 23 '13 at 17:11
add comment

If the number of files that are to be deleted vastly outnumbers the files which are left behind, it may not be the most efficient approach to walk the tree of files to be deleted and do all those filesystem updates. (It analogous to doing doing clumsy reference-counted memory management, visiting every object in a large tree to drop its reference, instead of making everything unwanted into garbage in one step, and then sweeping through what is reachable to clean up.)

That is to say, clone the parts of the tree that are to be kept to another volume. Re-create a fresh, blank filesystem on the original volume. Copy the retained files back to their original paths. This is vaguely similar to copying garbage collection.

There will be some downtime, but it could be better than continuous bad performance and service disruption.

It may be impractical in your system and situation, but it's easy to imagine obvious cases where this is the way to go.

For instance, suppose you wanted to delete all files in a filesystem. What would be the point of recursing and deleting one by one? Just unmount it and do a "mkfs" over top of the partition to make a blank filesystem.

Or suppose you wanted to delete all files except for half a dozen important ones? Get the half a dozen out of there and ... "mkfs" over top.

Eventually there is some break-even point when there are enough files that have to stay, that it becomes cheaper to do the recursive deletion, taking into account other costs like any downtime.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.