Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 4 down vote favorite
1

I have a directory that contains millions of sub-directory and trillions of files. And now I have to clear it. Saying trillion, I'm not talking about file size, but the number of files.

I've tried deleting it with del/s, and using windows explorer. Neither can complete the task. I've deleting some of the sub-directories one by one, and that toke me days. The problem i met was that every time, no matter using del or explorer, I can see in the task manager that the explorer instance consume sky high memory, and gradually pushes my system to crash.

There are still some hundred million files to be deleted. Is there any possibility to achieve with one (or just a few) commands / actions?

[EDITED]

I've tried doing it with Cygwin rm -fr, and yielded the same result. Summarized as:

  1. No matter using Windows Explorer, DEL from command prompt, or Cygwin rm command, the System memory gradually drops to zero, and the box will eventually crash.

  2. If at any point, before the system fails, the process is closed (by CTRL+C or what else), the box will continue to work as normal. However, all used memory will NOT be freed. Say, I've stop the process while system memory reaches 91%, Task Manager tells: 4G RAM in total, Cache is 329M, and Available 335MB. Then the memory usage will stay around this level until I reboot the machine. If I stop the explorer instance in Task Manager, the screen will go blank with HDD light all time on, and never came back. Normally, when I stop the explorer instance in Task Manager, I can re-invoke it by pressing Win+E, or it were restarted automatically.

Well, really nice memory management!

[EDIT AGAIN] It seems that some of the used memory did got freed after a long while, but not all. Some of the Cached & Available memory did come back in Task Manager. I haven't waited any longer, not sure what will happen then.

share|improve this question
So your main problem is the fact that directories and subdirectories aren't being deleted? – Sandeep Bansal Apr 24 '12 at 10:01
@Jackey Cheung: which version of windows you are using? – Siva Charan Apr 24 '12 at 10:10
The version I'm using is Windows 7 64-bits. The files/directories that are processed got deleted. The problem is that it can't process so many files in a run, and eventually stuck/crashed. – Jackey Cheung Apr 24 '12 at 10:27
You could write a batch script that recoursively deletes files, not starting from the top level but on e.g. the fifth level of the folder structure. That would split the job into many separate and sequential 'rm's – bytesum Apr 24 '12 at 11:27
3  
I have to know, how the hell did you get a trillion files, really... – Moab Apr 25 '12 at 1:35
show 7 more comments

6 Answers

up vote 3 down vote

To delete all folders will take a long time, and there is not a whole lot you can do about it. What you can do is save your data, and format your drive. It is not optimal, but it will work (and quickly).

Another option is perhaps to use some linux distro on a live CD that can read from an NTFS partition. I know from personal experience that rm -rf folderName can run for at least 2 days without crashing a system with 2GB of RAM. It will take a while, but at least it will finish.

share|improve this answer
hm, Linux. I'm thinking about the Cygwin. Though it is suppose to use underlining Windows functions, just wondering if it will make any difference in the case. I shall try it out. – Jackey Cheung Apr 24 '12 at 11:10
up vote 3 down vote

Erm.. I don't want to know how you created so many.

What's happening is Explorer is trying to enumerate every single file, and store the information in memory, before it starts deleting. And there's obviously way too many.

Have you tried the command rmdir /s? As long as it actually deletes the files as they are found rather than waiting on every single one to be enumerated, it may work.

How many levels of subdirectories are there? If there's only one, or some other low number, then a quick batch file that manually recurses through might work.

Any method will take a while, though.

share|improve this answer
Aside from soandos' suggestion of reformatting, of course. That would be fast, but if this is your system drive you will have to reinstall Windows. – Bob Apr 24 '12 at 10:04
I am pretty sure that enumeration has to take place, just so the program knows what to delete next. rmdir cannot delete files as they are found, as it starts from the top, and has to traverse somehow. The only question is how much excess info it stores. – soandos Apr 24 '12 at 10:50
@soandos Explorer counts every file. I was thinking some method that implements a DFS style of enumeration: going as far as possible down a branch, deleting when it hits a file, before coming back up. In other words, recursion, which is what rm -rf does. That works best with relatively shallow directory structures. I'm not sure if rmdir /s does this. It should. – Bob Apr 24 '12 at 11:00
I don't think rmdir will help in the case, since the system has finished deleting files yet. I think there shouldn't be much problem to rmdir all the directories, it'll take long time, but I don't think it'll crash. The problem is on deleting files. As I've mentioned, I've tried using del (command prompt) and explorer. Both have crashed after about working half day or so. And in both cases, the explorer instance in task manager shows insanely high memory consumption. – Jackey Cheung Apr 24 '12 at 11:17
@JackeyCheung rmdir /?: /s Removes all directories and files in the specified directory in addition to the directory itself. Used to remove a directory tree. In other words, the /s flag removes files as well. How did you use del? And yea, it might be better to just use rm -rf as soandos suggested. – Bob Apr 24 '12 at 11:24
show 2 more comments
up vote 1 down vote

Shift+Delete skips the Recycle Bin, and might significantly speed up things.

If that doesn't work (extreme cases), try Fast Folder Eraser and / or Mass Directory Eraser

share|improve this answer
up vote 0 down vote

Per this answer on StackOverflow use a combination of del and rmdir:

del /f/s/q foldername > nul
rmdir /s/q foldername
share|improve this answer
Yeah, I've read that before. Actually it doesn't help in my case. Since the del command crashes definitely. – Jackey Cheung Apr 25 '12 at 1:02
up vote 0 down vote

It's probably your antivirus/antimalware consuming all the memory and then crashing the system.

Windows itself doesn't have a problem deleting huge numbers of files, although it certainly is slower than a similar operation on most non-Microsoft filesystems.

share|improve this answer
Nice point! Sure worth a look at. – Jackey Cheung Apr 25 '12 at 3:34
I've turned off antivirus, and the memory still got eaten as before. – Jackey Cheung Apr 25 '12 at 5:46
Turning off antivirus doesn't help freeing memory after process is stopped either. – Jackey Cheung Apr 25 '12 at 7:11
@JackeyCheung: Which antivirus program is it? Some do not actually turn off completely... – Ben Voigt Apr 25 '12 at 13:31
up vote 0 down vote

Since deleting the files all at once uses too much memory, you need a way to delete them one at a time, but with the process automated. This sort of thing is a lot easier to do in a Unix-style shell, so let's use Cygwin. The following command generates a list of ordinary files, transforms that list into a sequence of rm commands, then feeds the resulting script to a shell.

 find dir \! -type d | sed 's/^/rm /' | sh

The script is being executed even as it is being generated, and there are no loops, so the shell does not (hopefully) have to create any big temp files. It will certainly take a while, since the script is millions of lines long. You might have to tweak the rm command (perhaps I should have used -f? but you understand your files better than me) to get it to work.

Now you have nothing left but directories. Here's where things get dicy. Maybe you've deleted enough files so that you can do rm -rf without running out of memory (and it will probably be faster than another script). If not, we can adapt this Stackoverflow answer:

 find dir | perl -lne 'print tr:/::, " $_"' | sort -n | cut -d' ' -f2 | sed 's/^/rmdir /' | sh

Again, tweaking may be necessary, this time with sort, to avoid creating huge temp files.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.