2

I have a lot of data that I don't have to 'get at right away', and I was wondering if there was any compression software I could use that could compress it down to a smaller size than I'm currently getting with 7zip. Compressing a gig of data takes ~10 minutes (optimizing for file size), and much less to decompress. Assuming I could spend hours on compression+extraction, would there be any way to get the compressed files even smaller?

So far, I've yet to see any GPU assisted compression programs, has anyone seen anything floating around?

4
  • You're aware that 7-zip has an "Ultra" mode?
    – LawrenceC
    Mar 28, 2011 at 2:19
  • Yes, I'm using that now.
    – Scott
    Mar 28, 2011 at 2:20
  • @ultrasawblade, Better paq8l in "normal" mode.
    – Vi.
    May 10, 2011 at 18:51
  • Why bother? Storage is cheap, and compression can make your data more vulnerable to corruption. Apr 12, 2014 at 19:12

5 Answers 5

3

https://en.wikipedia.org/wiki/Data_compression

This actually depends on what you are compressing, and if you need lossless compression or not.

Check out:

maximumcompression.com

It's a website with benchmarks for different data compression programs, testing them at various conditions and rating for various parameters (compress/decompress speed/ratio).

1
  • You might want to remove the index.html from the URI. It's returning a 404. Apr 12, 2014 at 19:03
2

The best answer to this question is almost certainly the PAQ family of algorithms designed by Matt Mahoney. He used it to win several data compression contents, and when I last checked it still held some many of its records. It takes a very long time to compress and can require huge amounts of RAM, but I've seen it significantly shrink even files that were already highly compressed (like existing archives).

You can get the latest version, and all major previous versions (some with different areas of focus, like ease of use or parallel operation), of the command-line utility here: http://mattmahoney.net/dc/zpaq.html

There are also other implementations that you might prefer. If you want a graphical interface, there is an "additional formats" plugin for PeaZip that includes PAQ: http://peazip.sourceforge.net/peazip-add-ons.html

lrzip is a command-line utility that is optimized for huge (GB-range) files. It uses a long-range redundancy reduction algorithm followed by a second algorithm seleced by the user, which can include a version of the PAQ algorithm (or various faster algorithms, if you want them). It also has other interesting features, including the ability to use virtual RAM to compress files larger than your system's physical RAM at the expense of slowing down drastically.

In addition to using a powerful compression algorithm, you can reduce the size of your final archive by pre-processing the files with other tools. You can convert some files from inherently bloated formats to leaner ones or strip metadata from them. Using a utility like dar (though most people here probably prefer the older tar) you can combine large numbers of files to make compression more efficient, and control the amount of filesystem metadata (like Unix permissions) that programs like 7zip unconditionally add to their archives. Any of these techniques assume that you are willing to commit more personal time in addition to machine resources.

EDIT: I have found that tar and dar do not allow control over metadata, nor does dar supercede tar's feature set as its mission implies. However they both support sparse files and dar might make compression more efficient if the option to write file headers only in the index is used (normally dar writes tar-style headers in addition to the index).

Regarding GPU compression, I haven't seen that either. It might not be viable if compression is more reliant on RAM than number of cores, like Litecoin.

All programs listed here are free, open-source, and (with the possible exception of lrzip) available for all major operating systems.

1
  • I'm surprised I got an upvote for this. I didn't check how old the question was before I answered it, and when I noticed the asker hasn't been active for about 3 years I felt quite silly.
    – Wutaz
    Apr 13, 2014 at 15:07
1

Jeff Atwood, our fearless leader, has a lot to say about file compression.

For what it's worth, 7-Zip's Ultra setting for 7z compression is probably the best you'll get when compressing several different file types.

1

One program that can sometimes improve compression ratio: cromfs

It creates compressed read-only filesystem that can have greater compression ratio than usual archive in some cases (although maximum number of blocks seems to be limited - compression is very memory-hungry).

0

You may try 7zip with the following ultra settings:

7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on big_file.mysql.7z big_file.mysql

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .