Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm trying to create scheduled task for copying a directory tree containing a very large number of small files from one server to another. Copying the tree directly takes a couple of days, so I was thinking it might be faster to zip the directory, copy the zip file, and unzip on the destination server.

I tried 7Zip's command-line utility, 7z.exe. It choked on an "unable to allocate memory" error. So now I'm looking for an alternative.

Does anyone know of a zip-utility that:

  1. Runs on all modern flavors of Windows
  2. Is free
  3. Can be controlled from the command-line
  4. Doesn't need to be installed (simply copying the EXE, and perhaps an associated DLL, the way that 7z.exe can be.)
  5. Doesn't choke on very large directory trees

???

share|improve this question
    
Wow, 7zip's usually pretty darn good - maybe a word in the support forum? 7-zip.org/support.html although I think there can be issues on allocating massive amounts of heap space on 32bit operating systems. –  Linker3000 Jan 25 '11 at 20:14
    
What was the Commandline you user for how much of data? –  akira Jan 25 '11 at 20:22
    
7z.exe a -tzip output.zip dirname –  Jeff Dege Jan 25 '11 at 20:38
add comment

3 Answers

up vote 1 down vote accepted

Have you considered using something like Robocopy with the /MT option? I would guess having a parallel copy is going to beat out compressing, transferring and decompressing. I would start with the default of 8 threads and monitor the disk queue length on both ends to make sure you are saturating, but not thrashing the disk to death. Robocopy can also only copy the files that actually are required to be transferred.

Usage :: ROBOCOPY source destination [file [file]...] [options]

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
           n must be at least 1 and not greater than 128.

If that doesn't work for some reason, I have had good success using the zip.exe executable from the UnxUtils package or for more extreme zip needs, the KZIP.exe executable. Kzip can be quite a bit slower at compressing at the extreme compression settings, but I haven't seen a zip archiver that can consistently beat it for space efficiency with a standard archive. Of course, it will run as a stand-alone executable and is easily command line scriptable. I can't say I have ever used them on extremely large directories.

You should also be aware that a "standard" ZIP archive can be at most 4GB in size, and contain at most 65535 files. If your structure exceeds those limits you will need to use newer versions of the zip standard (ZIP64).

share|improve this answer
add comment

Info-ZIP is the original free ZIP program.

I haven't tried it on very large directories. How big is the directory (number of files, total size, etc.)?

share|improve this answer
    
Tens of millions of 2-5k files, in a tree totaling 10-20GB. –  Jeff Dege Jan 25 '11 at 20:40
    
>>>>> Ouch! <<<<< –  Linker3000 Jan 25 '11 at 21:57
add comment

BZIP2: http://gnuwin32.sourceforge.net/packages/bzip2.htm

GZIP: http://gnuwin32.sourceforge.net/packages/gzip.htm

Tar: http://gnuwin32.sourceforge.net/packages/gtar.htm

They require some support dlls but they are most likely already in place in most Windows installs.

Worth trying.

Edit: How about DeltaCopy - a Windows-packaged version of rsync, although it may still choke on the amount of files: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp

share|improve this answer
    
+1 for rsync. Copying only the changes to maintain an up to date copy of the entire directory tree on your other server makes much more sense - in terms of both time and bandwidth. I think most programs will likely choke with that many files to be honest. –  Kez Jan 25 '11 at 22:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.