3

I'm trying to create scheduled task for copying a directory tree containing a very large number of small files from one server to another. Copying the tree directly takes a couple of days, so I was thinking it might be faster to zip the directory, copy the zip file, and unzip on the destination server.

I tried 7Zip's command-line utility, 7z.exe. It choked on an "unable to allocate memory" error. So now I'm looking for an alternative.

Does anyone know of a zip-utility that:

  1. Runs on all modern flavors of Windows
  2. Is free
  3. Can be controlled from the command-line
  4. Doesn't need to be installed (simply copying the EXE, and perhaps an associated DLL, the way that 7z.exe can be.)
  5. Doesn't choke on very large directory trees

???

Improve this question
3
  • 1
    Wow, 7zip's usually pretty darn good - maybe a word in the support forum? 7-zip.org/support.html although I think there can be issues on allocating massive amounts of heap space on 32bit operating systems.
    – Linker3000
    Jan 25, 2011 at 20:14
  • What was the Commandline you user for how much of data?
    – akira
    Jan 25, 2011 at 20:22
  • 7z.exe a -tzip output.zip dirname
    – Jeff Dege
    Jan 25, 2011 at 20:38

4 Answers 4

Reset to default
2

Have you considered using something like Robocopy with the /MT option? I would guess having a parallel copy is going to beat out compressing, transferring and decompressing. I would start with the default of 8 threads and monitor the disk queue length on both ends to make sure you are saturating, but not thrashing the disk to death. Robocopy can also only copy the files that actually are required to be transferred.

Usage :: ROBOCOPY source destination [file [file]...] [options]

/MT[:n] :: Do multi-threaded copies with n threads (default 8).
           n must be at least 1 and not greater than 128.

If that doesn't work for some reason, I have had good success using the zip.exe executable from the UnxUtils package or for more extreme zip needs, the KZIP.exe executable. Kzip can be quite a bit slower at compressing at the extreme compression settings, but I haven't seen a zip archiver that can consistently beat it for space efficiency with a standard archive. Of course, it will run as a stand-alone executable and is easily command line scriptable. I can't say I have ever used them on extremely large directories.

You should also be aware that a "standard" ZIP archive can be at most 4GB in size, and contain at most 65535 files. If your structure exceeds those limits you will need to use newer versions of the zip standard (ZIP64).

Improve this answer
1

Info-ZIP is the original free ZIP program.

I haven't tried it on very large directories. How big is the directory (number of files, total size, etc.)?

Improve this answer
2
  • Tens of millions of 2-5k files, in a tree totaling 10-20GB.
    – Jeff Dege
    Jan 25, 2011 at 20:40
  • 1
    >>>>> Ouch! <<<<<
    – Linker3000
    Jan 25, 2011 at 21:57
1

I may suggest PeaZip as possible solution. It makes very easy to create command scripts to batch execute compression tasks without the burden and limitations of GUI, and it is a frontend of Robocopy for file / directory copy operations.

Improve this answer
0

BZIP2: http://gnuwin32.sourceforge.net/packages/bzip2.htm

GZIP: http://gnuwin32.sourceforge.net/packages/gzip.htm

Tar: http://gnuwin32.sourceforge.net/packages/gtar.htm

They require some support dlls but they are most likely already in place in most Windows installs.

Worth trying.

Edit: How about DeltaCopy - a Windows-packaged version of rsync, although it may still choke on the amount of files: http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp

Improve this answer
1
  • +1 for rsync. Copying only the changes to maintain an up to date copy of the entire directory tree on your other server makes much more sense - in terms of both time and bandwidth. I think most programs will likely choke with that many files to be honest.
    – Kez
    Jan 25, 2011 at 22:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.