77

I work with tens of thousands 20 kB JPEG files in batches which add up to gigabytes. When I try to delete an old batch, Windows 10 has a very low delete rate which sometimes even drops to zero for some reason:

Enter image description here

Is this just a fact of life for the hardware I'm on, or is there some tips and tricks I could try to speed up this deletion process?

  • 11
    What type of storage device are the files on an HDD or SSD? If we are talking about a HDD what is the cache size? – Ramhound Sep 22 at 14:26
  • 7
    A number of options are explored at superuser.com/questions/416071/… for speedily deleting a folder with many files on Windows. – Gary Sep 22 at 15:01
  • 2
    @Kagaratsch what is the source of your batch...if you could get each batch as an ISO file then you could use it in a virtual drive and on finishing you would only have to eject and delete a single ISO file. – Elmo Sep 23 at 5:30
  • 2
    Maybe you can just store all your files in a disk image which you mount to a drive letter when you need them and delete it, when you don't need the files anymore – Benj Sep 23 at 13:06
  • 3
    @Kagaratsch: is it an external hard drive, with "optimize for safe removal" or whatever Windows calls it? That would probably make Windows use a lot of I/O barriers to make sure metadata is on disk before sending another write. Or maybe disable write cache entirely. – Peter Cordes Sep 23 at 19:04

12 Answers 12

87

From the image it looks like you are deleting the files through Explorer, which is the slowest method possible.

What you can do to improve:

  • Delete the files using Shift+Del so the deleted files are not moved to the Recycle Bin (no recovery possible)

  • Issue the delete from inside a Command prompt using a command similar to (use del /? to see all parameters):

    del /f /q *.*           (del in current folder, add `/s` to traverse sub-folders)
    del /f /q /s folder
    
  • 3
    Comments are not for extended discussion; this conversation has been moved to chat. – DavidPostill Sep 24 at 17:16
  • 1
    About the del commands, if I am not mistaken, redirecting the output to NUL should also speed them up a little. – Matteo Tassinari Sep 25 at 12:46
29

I don't know why Explorer is so slow, but you might consider putting each batch of images on a separate filesystem volume and quick-formatting the volume to delete them. Windows 10 has native support for mounting files as disk volumes. The speed of reformatting is independent of what's on the volume since it ignores the existing file system and just writes a new one.

Tens of thousands of 20K files sounds like not too many gigabytes. If the total size of all live batches is smaller than your physical RAM, you could also put them on RAM disks (using, e.g., ImDisk), still formatting or unmounting the volume to delete them. Even ordinary recursive deletion on the RAM disk might be much faster depending on what's causing the slowdown. This is obviously not an option if you need the files to survive a reboot.

One disadvantage of reformatting to delete is that you need to be an elevated administrator. Another is that you can accidentally format the wrong volume, but really any approach to mass deletion of files carries the risk of deleting the wrong ones. If you give the volumes descriptive labels like TempJpeg1 then formatting them is probably as mistake-proof as deleting a huge directory from the command line.

  • 2
    Can Windows transparently view JPEGs inside a ZIP file? That's much less work to set up, and still makes deletion of a whole batch from main filesystem trivially fast. It's a different set of tradeoffs between admin complexity to set up vs. transparent to all programs using it. And fixed-size vs. elastic container size. Also faster add/remove of files from the batch for a separate filesystem, probably. (Getting JPEGs into a ZIP could maybe be done on the fly as you copy them from somewhere else, or overnight so the next day you have batches that you can sort through.) – Peter Cordes Sep 23 at 19:14
  • 3
    @PeterCordes WinArchiver and Pismo File Mount can mount ZIP as a read-only drive. But VHD can be set to expandable (only using the actual space it needs), adding the batch into mounted VHD should take less overhead than repeatedly modifying a zip file to add them. – Martheen Cahya Paulo Sep 24 at 6:38
  • 4
    In association with the other answer about antivirus, the directory containing these virtual disk files should also be excluded from antivirus, or the OP may experience a double-penalty of the data inside the virtual disk being scanned, as well as the outer file container also being repeatedly scanned. – Dale Mahalko Sep 25 at 1:31
24

Disable any antivirus/antimalware real-time protection you have, at least for the duration of this operation. This includes Windows Defender. Also any anti-ransomware protection (as offered by some backup programs).

Real-time protection typically has hooks on file operations, and are historically notorious for drastically slowing down any operation involving large numbers of files.

If you feel the need to, you can re-enable them after the deletion is complete.

  • 16
    In case anybody else is wondering, I've observed in profiling that the AV stupidly scans files that are being deleted. – Joshua Sep 23 at 18:17
  • 3
    @Joshua - I don't think it's entirely stupid, but it's certainly annoying. It's always possible that malware which installed itself successfully is deleting the files it used to get there, to make it harder to detect/analyze. That said, that's definitely an edge case, and I doubt it's worth the extra overhead. – Bobson Sep 23 at 21:37
  • 6
    @Bobson: It would have already gotten scanned 3ms earlier when the file got saved or 2ms earlier when the file got executed. – Joshua Sep 23 at 22:27
  • 2
    AV software completely lacks any reasonable logic for when scans are needed. Should only be the first open for read/exec after the file was changed. – R.. Sep 25 at 0:16
8

Create a partition per batch, let's say 10GB or so, when done with given batch quick format the partition. All files gone in matter of seconds.

  • 2
    How would the OP go about doing this? Please see How to Answer or tour to improve your answer. – Burgi Sep 23 at 16:05
7

Running this as a command will be a lot faster. Launch cmd and run

RD frames /S /Q

That will remove the folder and all its files and subdirectories in it, without spamming the screen with each file name it is deleting.

RD = remove directory
/S = subdirectories included
/Q = quiet mode
6

I think that the speed resides in the Microsoft Windows command-line preprocessor and this is why we all love it so much.

Used with care, it can even avoid historical faults like the rm -Rf / that causes the removal of the entire root folder in the Unix family of operating systems from where it originates.

Now, what I would suggest would be just to get rid of your frames\ folder that appears in your question. For that, the operating system command is RMDIR or RD as it appears in the picture pasted below.

The RMDIR operating system command

Your operating system command would be RMDIR /S /Q frames\ issued in the Microsoft Windows command-line preprocessor started in the folder that is the parent folder of the frames\ folder.

The only issue with this solution is that it deletes the contents of the entire frames\ folder and not only your 47,189 *.JPG files.

So we have a few ideas in this respect:

  1. The first situation would be when your frames\ folder contains ONLY your 47,189 *.JPG files in which situation you are free to apply this solution and afterward to recreate your frames\ folder using the MKDIR or MD operating system command.

  2. The second situation would be when your frames\ folder contains also A FEW files among your 47,189 *.JPG files in which situation you could first move them somewhere else using a file-management software application and then you could move them back into the frames\ folder after its recreation as described at point (1.).

  3. The third situation would be when your frames\ folder contains also A LOT - let us say millions - of files among your 47,189 *.JPG files. In this situation it is better for you to use other alternative solutions as described by other posters on this web-page.

UPDATE: In case either of you are wondering what has happened with the DELTREE operating system command, then you are all invited to check this link out.

4

Dealing with roaming profiles, I've seen this quite a bit. In my experience, sending any sort of delete command is not as fast as using Robocopy to /MIR an empty folder to the destination. I don't have any raw data or actual metrics that this is faster than any of the already mentioned options, but it is considerably faster than using the Explorer shell.

Just make a new empty directory, or reference a current empty directory, then run the following command:

robocopy x:\path\to\emptyfolder x:\path\to\delete /MIR /R:1 /W:1 /MT:128 /LOG:x:\path\to\logfile.log

/MIR :: MIRror a directory tree (equivalent to /E plus /PURGE). 
/R:1 :: number of Retries on failed copies: default 1 million.
/W:1 :: Wait time between retries: default is 30 seconds. 
/MT:128 :: Do multi-threaded copies with n threads (default 8).
               n must be at least 1 and not greater than 128.
               This option is incompatible with the /IPG and /EFSRAW options.
               Redirect output using /LOG option for better performance.

You'll want to LOG when using MT, there is definitely a measurable delay when displaying the output on screen.

By using /MT:128 as an option, it will max out the amount of files it can 'delete' at one time.

Also note that if this is your first time with Robocopy, you should do a dry run with the /L flag:

/L :: List only - don't copy, timestamp or delete any files.

Then check the logs and ensure the actions match what you want.

References:

  • 2
    This is BY FAR the fastest way to do it in Windows. Works also great while deleting huge folder-trees on network drives and it can handle those very deep folder-trees where the path gets too long for explorer. – Tonny Sep 23 at 19:51
3

To add to the answers provided, the reason that the deletion operations take so long is due the type of hard drive you're using. In one of your comments you state that you're using a mechanical hard drive, and the amount of IOPS that they can perform is pretty small compared to SSDs. The Wikipedia has a list of the rough amount of IOPS that mechanical and solid state drives can perform on their site.

What's basically happening is that the OS is telling the hard drive to erase the metadata of the file in the partition table, which removes it from view, (it's still there, but the OS no longer knows how to access it). It's basically a very small write to the hard drive, (it's writing 0s to the it), so it's a quick IOP, which can do a lot more than what's listed on the Wiki since the Wiki is listing random and sustained writes.

The thing to keep in mind now is that Explorer tries to play nice with everything else running on the system while it is deleting, which can be slow. The del command on the other hand doesn't try to be nice at all, and deletes as quickly as possible.

Upgrading to an SSD would help with your work since their IOPS are normally measured in the thousands, and mechanical drives are measured in the hundreds. You could also try using a flash drive to see if that helps; however your results would be highly dependent on the drive you use. If you have the RAM, (and don't mind the risk of loosing the images), you could create a RAM drive and use that. I have personally use ImDisk Virtual Disk Driver, (no affiliation), with a great amount of success and it's free.

2

I had this exact problem with some automated testing that created and deleted thousands of files for each test run.

Here is what I did to get good performance.

  1. Always delete the entire folder and avoid the recycling bin. If you are doing it manually, then use the command line.

  2. Can you convert some of your memory to a RAM drive? I bought extra RAM and it worked brilliantly because the files were supposed to be deleted as part of my testing. (A better solution is dedicated battery backed RAM drives, but they are very expensive.)

  3. Writing to 1st and 2nd generation SSD drives gets slower over time and they eventually become unusable. Running a low level "Trim command" fixes this, although not all OS/Hardware combinations support Trim. Eventually I needed to buy new SSDs because mine had been rewritten too many times. (Obviously my testing was was killing my cheap SSD drives. Took me a while to figure that out.)

  4. If you are using a HDD, defragment the drive regularly.

Good luck. It took me many months and a few SSDs, followed by extra RAM, to find a strategy that worked for me, so I feel your pain..

2

Deleting many directory entries will take longer than deleting a single directory of the net size. When deleting many files, it takes many commands instead of a single when a single file is deleted.

Now, when one big file is deleted, the filemapper is told to unallocate the memory of that big file, but the 1s and 0s of the file are not reset. The mapper is just told it is free to write new stuff over the old 1s and 0s. Basically, it is one command, that takes time to clear up all of the memory allocations of the fragmented file. When you delete a million small files, you are sending a million commands to the mapper, that has to do each one at a time.

So the solution is to use software to wipe a drive of deleted files and the 1s and 0s left behind. It will take one single command to wipe the space. Use the software only if you think the files are not needed to be recovered later.

  • Nope, to wipe (in the sense of overwriting all data), you'll need (at least) one internal command per block of the disk (which is typically between 512 bytes and 4096 bytes in size). A Quick Format would be way faster here (which just overwrites the root file allocation table of the disk/partition with an empty one, leaving all blocks of the disk in unallocated state). But anyway, this approach is impractical if you just want to delete a folder instead of a whole disk/partition. – orithena Sep 23 at 8:17
  • that has to do each one at a time The filesystem doesn't have to update the contents on disk after every delete; it can leave the metadata dirty in memory for a few seconds and only update contents on disk after hundreds or thousands of deletes. (It will have to read the disk if it wasn't already hot in cache to find out which blocks a file owned). – Peter Cordes Sep 23 at 19:27
2

Besides the other great answers I wanted to share a few tricks with you:

  • delete all files in a folder: del *.*
  • shortcut: same as above, less typing: del.. Note that this is undocumented, I came across this shortcut by mistake.

Now make your life easier:

  • delete multiple files without having to answer the "are you sure" question: echo Y | del.
1

If you're deleting many small files the disk has to spend a lot of seeking, especially since you're using a spinning disk drive and not an SSD. Deleting a big file is like running a marathon all at once, but deleting small files is like running a marathon in 100 meter chunks spread out across the city. You're main activity is seeking the location in the disk. If this is a regular activity you could make a partition for holding these files temporarily and then wipe the partition in a second when you're done with them.

  • Even your marathon analogy doesn't really work, because the distance isn't the same. For the most part, the work needed to delete a file depends on the number of fragments, not the total size. So not only are you running the marathon all in one go, but it's just a 10m run rather than 420 000 chunks of running 1m each, while also spending most of the time in transit (especially if the files have been added to the folder at different times). – Luaan Sep 24 at 6:47
  • That's true. I was just trying to make a simple analogy that explained the travel time. – Jay Speidell Sep 24 at 17:57

protected by Community Sep 24 at 4:23

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.