104

I work with tens of thousands 20 kB JPEG files in batches which add up to gigabytes. When I try to delete an old batch, Windows 10 has a very low delete rate which sometimes even drops to zero for some reason:

Enter image description here

Is this just a fact of life for the hardware I'm on, or is there some tips and tricks I could try to speed up this deletion process?

Improve this question
13
  • 12
    What type of storage device are the files on an HDD or SSD? If we are talking about a HDD what is the cache size?
    – Ramhound
    Sep 22, 2019 at 14:26
  • 7
    A number of options are explored at superuser.com/questions/416071/… for speedily deleting a folder with many files on Windows.
    – Gary
    Sep 22, 2019 at 15:01
  • 2
    @Kagaratsch what is the source of your batch...if you could get each batch as an ISO file then you could use it in a virtual drive and on finishing you would only have to eject and delete a single ISO file.
    – Elmo
    Sep 23, 2019 at 5:30
  • 2
    Maybe you can just store all your files in a disk image which you mount to a drive letter when you need them and delete it, when you don't need the files anymore
    – Benj
    Sep 23, 2019 at 13:06
  • 3
    @Kagaratsch: is it an external hard drive, with "optimize for safe removal" or whatever Windows calls it? That would probably make Windows use a lot of I/O barriers to make sure metadata is on disk before sending another write. Or maybe disable write cache entirely.
    – Peter Cordes
    Sep 23, 2019 at 19:04

11 Answers 11

Reset to default
101

From the image it looks like you are deleting the files through Explorer, which is the slowest method possible.

What you can do to improve:

  • Delete the files using Shift+Del so the deleted files are not moved to the Recycle Bin (no recovery possible)

  • Issue the delete from inside a Command prompt using a command similar to (use del /? to see all parameters):

      del /f /q *.*           (del in current folder, add `/s` to traverse sub-folders)
  del /f /q /s folder
Improve this answer
2
  • 3
    Comments are not for extended discussion; this conversation has been moved to chat.
    – DavidPostill ♦
    Sep 24, 2019 at 17:16
  • 2
    About the del commands, if I am not mistaken, redirecting the output to NUL should also speed them up a little.
    – Matteo Tassinari
    Sep 25, 2019 at 12:46
31

I don't know why Explorer is so slow, but you might consider putting each batch of images on a separate filesystem volume and quick-formatting the volume to delete them. Windows 10 has native support for mounting files as disk volumes. The speed of reformatting is independent of what's on the volume since it ignores the existing file system and just writes a new one.

Tens of thousands of 20K files sounds like not too many gigabytes. If the total size of all live batches is smaller than your physical RAM, you could also put them on RAM disks (using, e.g., ImDisk), still formatting or unmounting the volume to delete them. Even ordinary recursive deletion on the RAM disk might be much faster depending on what's causing the slowdown. This is obviously not an option if you need the files to survive a reboot.

One disadvantage of reformatting to delete is that you need to be an elevated administrator. Another is that you can accidentally format the wrong volume, but really any approach to mass deletion of files carries the risk of deleting the wrong ones. If you give the volumes descriptive labels like TempJpeg1 then formatting them is probably as mistake-proof as deleting a huge directory from the command line.

Improve this answer
3
  • 2
    Can Windows transparently view JPEGs inside a ZIP file? That's much less work to set up, and still makes deletion of a whole batch from main filesystem trivially fast. It's a different set of tradeoffs between admin complexity to set up vs. transparent to all programs using it. And fixed-size vs. elastic container size. Also faster add/remove of files from the batch for a separate filesystem, probably. (Getting JPEGs into a ZIP could maybe be done on the fly as you copy them from somewhere else, or overnight so the next day you have batches that you can sort through.)
    – Peter Cordes
    Sep 23, 2019 at 19:14
  • 3
    @PeterCordes WinArchiver and Pismo File Mount can mount ZIP as a read-only drive. But VHD can be set to expandable (only using the actual space it needs), adding the batch into mounted VHD should take less overhead than repeatedly modifying a zip file to add them.
    – Martheen
    Sep 24, 2019 at 6:38
  • 4
    In association with the other answer about antivirus, the directory containing these virtual disk files should also be excluded from antivirus, or the OP may experience a double-penalty of the data inside the virtual disk being scanned, as well as the outer file container also being repeatedly scanned.
    – Dale Mahalko
    Sep 25, 2019 at 1:31
29

Disable any antivirus/antimalware real-time protection you have, at least for the duration of this operation. This includes Windows Defender. Also any anti-ransomware protection (as offered by some backup programs).

Real-time protection typically has hooks on file operations, and are historically notorious for drastically slowing down any operation involving large numbers of files.

If you feel the need to, you can re-enable them after the deletion is complete.

Improve this answer
4
  • 17
    In case anybody else is wondering, I've observed in profiling that the AV stupidly scans files that are being deleted.
    – Joshua
    Sep 23, 2019 at 18:17
  • 3
    @Joshua - I don't think it's entirely stupid, but it's certainly annoying. It's always possible that malware which installed itself successfully is deleting the files it used to get there, to make it harder to detect/analyze. That said, that's definitely an edge case, and I doubt it's worth the extra overhead.
    – Bobson
    Sep 23, 2019 at 21:37
  • 6
    @Bobson: It would have already gotten scanned 3ms earlier when the file got saved or 2ms earlier when the file got executed.
    – Joshua
    Sep 23, 2019 at 22:27
  • 2
    AV software completely lacks any reasonable logic for when scans are needed. Should only be the first open for read/exec after the file was changed.
    – R.. GitHub STOP HELPING ICE
    Sep 25, 2019 at 0:16
16

Running this as a command will be a lot faster. Launch cmd and run

RD frames /S /Q

That will remove the folder and all its files and subdirectories in it, without spamming the screen with each file name it is deleting.

RD = remove directory
/S = subdirectories included
/Q = quiet mode
Improve this answer
1
  • This was shockingly fast (about 6 seconds) compared to what Explorer estimated remained (6 minutes) to delete a node_modules folder.
    – Corrodias
    Sep 22 at 17:10
8

Create a partition per batch, let's say 10GB or so, when done with given batch quick format the partition. All files gone in matter of seconds.

Improve this answer
1
  • 2
    How would the OP go about doing this? Please see How to Answer or tour to improve your answer.
    – Burgi
    Sep 23, 2019 at 16:05
8

Dealing with roaming profiles, I've seen this quite a bit. In my experience, sending any sort of delete command is not as fast as using Robocopy to /MIR an empty folder to the destination. I don't have any raw data or actual metrics that this is faster than any of the already mentioned options, but it is considerably faster than using the Explorer shell.

Just make a new empty directory, or reference a current empty directory, then run the following command: 

robocopy x:\path\to\emptyfolder x:\path\to\delete /MIR /R:1 /W:1 /MT:128 /LOG:x:\path\to\logfile.log

/MIR :: MIRror a directory tree (equivalent to /E plus /PURGE). 
/R:1 :: number of Retries on failed copies: default 1 million.
/W:1 :: Wait time between retries: default is 30 seconds. 
/MT:128 :: Do multi-threaded copies with n threads (default 8).
               n must be at least 1 and not greater than 128.
               This option is incompatible with the /IPG and /EFSRAW options.
               Redirect output using /LOG option for better performance.

You'll want to LOG when using MT, there is definitely a measurable delay when displaying the output on screen.

By using /MT:128 as an option, it will max out the amount of files it can 'delete' at one time.

Also note that if this is your first time with Robocopy, you should do a dry run with the /L flag: 

/L :: List only - don't copy, timestamp or delete any files.

Then check the logs and ensure the actions match what you want.

References:

Improve this answer
1
  • 2
    This is BY FAR the fastest way to do it in Windows. Works also great while deleting huge folder-trees on network drives and it can handle those very deep folder-trees where the path gets too long for explorer.
    – Tonny
    Sep 23, 2019 at 19:51
4

To add to the answers provided, the reason that the deletion operations take so long is due the type of hard drive you're using. In one of your comments you state that you're using a mechanical hard drive, and the amount of IOPS that they can perform is pretty small compared to SSDs. The Wikipedia has a list of the rough amount of IOPS that mechanical and solid state drives can perform on their site.

What's basically happening is that the OS is telling the hard drive to erase the metadata of the file in the partition table, which removes it from view, (it's still there, but the OS no longer knows how to access it). It's basically a very small write to the hard drive, (it's writing 0s to the it), so it's a quick IOP, which can do a lot more than what's listed on the Wiki since the Wiki is listing random and sustained writes.

The thing to keep in mind now is that Explorer tries to play nice with everything else running on the system while it is deleting, which can be slow. The del command on the other hand doesn't try to be nice at all, and deletes as quickly as possible.

Upgrading to an SSD would help with your work since their IOPS are normally measured in the thousands, and mechanical drives are measured in the hundreds. You could also try using a flash drive to see if that helps; however your results would be highly dependent on the drive you use. If you have the RAM, (and don't mind the risk of loosing the images), you could create a RAM drive and use that. I have personally use ImDisk Virtual Disk Driver, (no affiliation), with a great amount of success and it's free.

Improve this answer
3

I had this exact problem with some automated testing that created and deleted thousands of files for each test run.

Here is what I did to get good performance.

  1. Always delete the entire folder and avoid the recycling bin. If you are doing it manually, then use the command line.

  2. Can you convert some of your memory to a RAM drive? I bought extra RAM and it worked brilliantly because the files were supposed to be deleted as part of my testing. (A better solution is dedicated battery backed RAM drives, but they are very expensive.)

  3. Writing to 1st and 2nd generation SSD drives gets slower over time and they eventually become unusable. Running a low level "Trim command" fixes this, although not all OS/Hardware combinations support Trim. Eventually I needed to buy new SSDs because mine had been rewritten too many times. (Obviously my testing was was killing my cheap SSD drives. Took me a while to figure that out.)

  4. If you are using a HDD, defragment the drive regularly.

Good luck. It took me many months and a few SSDs, followed by extra RAM, to find a strategy that worked for me, so I feel your pain..

Improve this answer
2

Besides the other great answers I wanted to share a few tricks with you:

  • delete all files in a folder: del *.*
  • shortcut: same as above, less typing: del..

Now make your life easier:

  • delete multiple files without having to answer the "are you sure" question: echo Y | del.
Improve this answer
3
  • 1
    "del." is not really undocumented. The dot stands for the current directory (likewise, ".." stands for the parent directory. You can see both by typing in "dir". The missing space between command and parameter was historically allowed in MS-DOS and also works for other built-in commands, e.g., "cd.."
    – Douba
    Dec 26, 2020 at 22:57
  • 1
    @Douba, you're right. I don't know, though, why "delete the current directory", which del . essentially is, works for deleting all files.
    – Michał Leon
    Dec 28, 2020 at 14:41
  • Hmm, does it really? I think it only deletes all files in the current directory, but not subdirectories. Even if you specify the /s switch (was not orient in older DOS versions), it only traverses subdirectories to delete files, but leaves the folders intact.
    – Douba
    Dec 28, 2020 at 18:22
1

If you're deleting many small files the disk has to spend a lot of seeking, especially since you're using a spinning disk drive and not an SSD. Deleting a big file is like running a marathon all at once, but deleting small files is like running a marathon in 100 meter chunks spread out across the city. You're main activity is seeking the location in the disk. If this is a regular activity you could make a partition for holding these files temporarily and then wipe the partition in a second when you're done with them.

Improve this answer
2
  • Even your marathon analogy doesn't really work, because the distance isn't the same. For the most part, the work needed to delete a file depends on the number of fragments, not the total size. So not only are you running the marathon all in one go, but it's just a 10m run rather than 420 000 chunks of running 1m each, while also spending most of the time in transit (especially if the files have been added to the folder at different times).
    – Luaan
    Sep 24, 2019 at 6:47
  • That's true. I was just trying to make a simple analogy that explained the travel time.
    – Jay Speidell
    Sep 24, 2019 at 17:57
1

Deleting many directory entries will take longer than deleting a single directory of the net size. When deleting many files, it takes many commands instead of a single when a single file is deleted.

Now, when one big file is deleted, the filemapper is told to unallocate the memory of that big file, but the 1s and 0s of the file are not reset. The mapper is just told it is free to write new stuff over the old 1s and 0s. Basically, it is one command, that takes time to clear up all of the memory allocations of the fragmented file. When you delete a million small files, you are sending a million commands to the mapper, that has to do each one at a time.

So the solution is to use software to wipe a drive of deleted files and the 1s and 0s left behind. It will take one single command to wipe the space. Use the software only if you think the files are not needed to be recovered later.

Improve this answer
2
  • 1
    Nope, to wipe (in the sense of overwriting all data), you'll need (at least) one internal command per block of the disk (which is typically between 512 bytes and 4096 bytes in size). A Quick Format would be way faster here (which just overwrites the root file allocation table of the disk/partition with an empty one, leaving all blocks of the disk in unallocated state). But anyway, this approach is impractical if you just want to delete a folder instead of a whole disk/partition.
    – orithena
    Sep 23, 2019 at 8:17
  • that has to do each one at a time The filesystem doesn't have to update the contents on disk after every delete; it can leave the metadata dirty in memory for a few seconds and only update contents on disk after hundreds or thousands of deletes. (It will have to read the disk if it wasn't already hot in cache to find out which blocks a file owned).
    – Peter Cordes
    Sep 23, 2019 at 19:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .