I tried to use .rar and I get a great rate of compression. However it take a lot of time to show the folder structure when I open the .rar file (tried different programs), it looks it is needed to read every file before displaying the folder structure. (I need to compress like 1 million files)

What compression format should I use so I can open the archive and extract some file on it fast when needed?

  • What operating system are you using? Do you need compression or just a container? – pyther Dec 6 '14 at 15:15
  • Needed compression and ability to access files in Mac OS X and Windows – NeDark Dec 6 '14 at 15:17

There will be trade off between speed/compression ratio.

You may want to view the blog post, Linux Compression Comparison (GZIP vs BZIP2 vs LZMA vs ZIP vs Compress). The author's results reflect my general experiences with the various compression algorithms. Pay specific attention to the decompression graph.

The ZIP format provides the best platform compatibility (both Windows and OS X support zip natively). However, its compression ratio will likely be worse than RAR.

The GZIP format is a very popular choice in the Linux world. It provides very quick compression while average compression rates. Mac OS X supports gzip archives natively, however with Windows you need an application such as 7zip.

The BZIP2 and LZMA formats tend to take a much longer time to compress and decompress files, however they tend to yield much better compression ratios. I think OS X supports both of these formats natively, again with Windows, you will need to use something like 7zip.

The type of files being compressed makes a huge difference in compression time, decompression, and compression ratio and differs for each format.

Ultimately you will need to try various formats and see which meets your needs tho best.

  • I dont care about descompression/compression speed nor rate very much, i only want to open the file fast. With that amount of files, if I need only one I have to open the compressed file, wait 5-10min till opened and then wait a few seconds to extract the file I need. – NeDark Dec 6 '14 at 16:42
  • @NeDark check 7Zip and there the 7z format. The browser seems to be very fast, although I didn't test it with millions of files. – boboes Dec 6 '14 at 16:48

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.