Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a lot of data that I don't have to 'get at right away', and I was wondering if there was any compression software I could use that could compress it down to a smaller size than I'm currently getting with 7zip. Compressing a gig of data takes ~10 minutes (optimizing for file size), and much less to decompress. Assuming I could spend hours on compression+extraction, would there be any way to get the compressed files even smaller?

So far, I've yet to see any GPU assisted compression programs, has anyone seen anything floating around?

share|improve this question
    
You're aware that 7-zip has an "Ultra" mode? –  ultrasawblade Mar 28 '11 at 2:19
    
Yes, I'm using that now. –  Scott Mar 28 '11 at 2:20
    
@ultrasawblade, Better paq8l in "normal" mode. –  Vi. May 10 '11 at 18:51
    
Why bother? Storage is cheap, and compression can make your data more vulnerable to corruption. –  Roland Smith Apr 12 at 19:12
add comment

5 Answers 5

http://en.wikipedia.org/wiki/Data_compression

This actually depends on what you are compressing, and if you need lossless compression or not.

Check out:

http://www.maximumcompression.com/

It's a website with benchmarks for different data compression programs, testing them at various conditions and rating for various parameters (compress/decompress speed/ratio).

share|improve this answer
    
You might want to remove the index.html from the URI. It's returning a 404. –  Roland Smith Apr 12 at 19:03
add comment

Jeff Atwood, our fearless leader, has a lot to say about file compression.

For what it's worth, 7-Zip's Ultra setting for 7z compression is probably the best you'll get when compressing several different file types.

share|improve this answer
add comment

One program that can sometimes improve compression ratio: cromfs

It creates compressed read-only filesystem that can have greater compression ratio than usual archive in some cases (although maximum number of blocks seems to be limited - compression is very memory-hungry).

share|improve this answer
add comment

The best answer to this question is almost certainly the PAQ family of algorithms designed by Matt Mahoney. He used it to win several data compression contents, and when I last checked it still held some many of its records. It takes a very long time to compress and can require huge amounts of RAM, but I've seen it significantly shrink even files that were already highly compressed (like existing archives).

You can get the latest version, and all major previous versions (some with different areas of focus, like ease of use or parallel operation), of the command-line utility here: http://mattmahoney.net/dc/zpaq.html

There are also other implementations that you might prefer. If you want a graphical interface, there is an "additional formats" plugin for PeaZip that includes PAQ: http://peazip.sourceforge.net/peazip-add-ons.html

lrzip is a command-line utility that is optimized for huge (GB-range) files. It uses a long-range redundancy reduction algorithm followed by a second algorithm seleced by the user, which can include a version of the PAQ algorithm (or various faster algorithms, if you want them). It also has other interesting features, including the ability to use virtual RAM to compress files larger than your system's physical RAM at the expense of slowing down drastically.

In addition to using a powerful compression algorithm, you can reduce the size of your final archive by pre-processing the files with other tools. You can convert some files from inherently bloated formats to leaner ones or strip metadata from them. Using a utility like dar (though most people here probably prefer the older tar) you can combine large numbers of files to make compression more efficient, and control the amount of filesystem metadata (like Unix permissions) that programs like 7zip unconditionally add to their archives. Any of these techniques assume that you are willing to commit more personal time in addition to machine resources.

EDIT: I have found that tar and dar do not allow control over metadata, nor does dar supercede tar's feature set as its mission implies. However they both support sparse files and dar might make compression more efficient if the option to write file headers only in the index is used (normally dar writes tar-style headers in addition to the index).

Regarding GPU compression, I haven't seen that either. It might not be viable if compression is more reliant on RAM than number of cores, like Litecoin.

All programs listed here are free, open-source, and (with the possible exception of lrzip) available for all major operating systems.

share|improve this answer
    
I'm surprised I got an upvote for this. I didn't check how old the question was before I answered it, and when I noticed the asker hasn't been active for about 3 years I felt quite silly. –  Wutaz Apr 13 at 15:07
add comment

You may try 7zip with the following ultra settings:

7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on big_file.mysql.7z big_file.mysql
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.