Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to create scheduled task for copying a directory tree containing a very large number of small files from one server to another. Copying the tree directly takes a couple of days, so I was thinking it might be faster to zip the directory, copy the zip file, and unzip on the destination server.

I tried 7Zip's command-line utility, 7z.exe. It choked on an "unable to allocate memory" error. So now I'm looking for an alternative.

Does anyone know of a zip-utility that:

  1. Runs on all modern flavors of Windows
  2. Is free
  3. Can be controlled from the command-line
  4. Doesn't need to be installed (simply copying the EXE, and perhaps an associated DLL, the way that 7z.exe can be.)
  5. Doesn't choke on very large directory trees


share|improve this question

closed as off-topic by Michael Kjörling, Tog, Kevin Panko, Excellll, random Aug 14 '14 at 16:10

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – Michael Kjörling, Tog, Kevin Panko, Excellll, random
If this question can be reworded to fit the rules in the help center, please edit the question.

Wow, 7zip's usually pretty darn good - maybe a word in the support forum? although I think there can be issues on allocating massive amounts of heap space on 32bit operating systems. – Linker3000 Jan 25 '11 at 20:14
What was the Commandline you user for how much of data? – akira Jan 25 '11 at 20:22
7z.exe a -tzip dirname – Jeff Dege Jan 25 '11 at 20:38
up vote 1 down vote accepted

Have you considered using something like Robocopy with the /MT option? I would guess having a parallel copy is going to beat out compressing, transferring and decompressing. I would start with the default of 8 threads and monitor the disk queue length on both ends to make sure you are saturating, but not thrashing the disk to death. Robocopy can also only copy the files that actually are required to be transferred.

Usage :: ROBOCOPY source destination [file [file]...] [options]

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
           n must be at least 1 and not greater than 128.

If that doesn't work for some reason, I have had good success using the zip.exe executable from the UnxUtils package or for more extreme zip needs, the KZIP.exe executable. Kzip can be quite a bit slower at compressing at the extreme compression settings, but I haven't seen a zip archiver that can consistently beat it for space efficiency with a standard archive. Of course, it will run as a stand-alone executable and is easily command line scriptable. I can't say I have ever used them on extremely large directories.

You should also be aware that a "standard" ZIP archive can be at most 4GB in size, and contain at most 65535 files. If your structure exceeds those limits you will need to use newer versions of the zip standard (ZIP64).

share|improve this answer

Info-ZIP is the original free ZIP program.

I haven't tried it on very large directories. How big is the directory (number of files, total size, etc.)?

share|improve this answer
Tens of millions of 2-5k files, in a tree totaling 10-20GB. – Jeff Dege Jan 25 '11 at 20:40
>>>>> Ouch! <<<<< – Linker3000 Jan 25 '11 at 21:57

I may suggest PeaZip as possible solution. It makes very easy to create command scripts to batch execute compression tasks without the burden and limitations of GUI, and it is a frontend of Robocopy for file / directory copy operations.

share|improve this answer




They require some support dlls but they are most likely already in place in most Windows installs.

Worth trying.

Edit: How about DeltaCopy - a Windows-packaged version of rsync, although it may still choke on the amount of files:

share|improve this answer
+1 for rsync. Copying only the changes to maintain an up to date copy of the entire directory tree on your other server makes much more sense - in terms of both time and bandwidth. I think most programs will likely choke with that many files to be honest. – Kez Jan 25 '11 at 22:38

Not the answer you're looking for? Browse other questions tagged .