I need to compress a directory with around 350,000 fairly small files that amount to about 100GB total. I am using OSX and am currently using the standard "Compress" tool that converts this directory into a .zip file. Is there a faster way to do this?
For directories I'd use a
a simple way to go is,
tar cfj archive.tar.bz2 dir-to-be-archived/
This works great if you don't intend to fetch small sets of files out of the archive
I prefer to call such archives
The max-compression pipe looks like this,
tar cf - dir-to-be-archived/ | bzip2 -9 - > archive.tar.bz2 # ^pipe tarball from here to zip-in^ into the archive file.
Note: the '
If you have a fast network and the archive is going to be placed on a different machine,
tar cf - dir/ | ssh user@server "bzip2 -9 - > /target-path/archive.tar.bz2" # ^ pipe tarball over network to zip ^ and archive on remote machine.
This guy did some research on that. It appears that .zip will compress larger files faster. However, it yields one of the largest compression sizes. It also looks like he was using Windows utilities, but I'm betting OSX's utility is almost as optimized.
Here is an excellent website where numerous compression utilities have been benchmarked for speed over many files. There are many other tests on that site you could look at to determine the best utility for you.
Much of the speed has to do with the program you use. I've used 7zip's utility for Windows, and I find that to be very fast. However, compressing many files takes a long time no matter what so I would just let it go overnight. Or you could just tar the whole thing and not compress it...Personally I hate unzipping large archives so I would be careful if that's what you want to do.