Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have seen some highly compressed files around, such as 700MB of data compressed to around 30-50MB.

But how do you get such compressed files? I have tried using software like WinRAR and 7Zip but have never achieved such high compression.

What are the techniques/software that allow you to compress files so well?

(P.S. I'm using Windows XP)

share|improve this question
    
Nice idea ... but where do you get such files from anyways? –  Jarvis Aug 22 '09 at 11:42
3  
I've seen 7zip compress server log files (mainly text) down to about 1% of their original size. –  Umber Ferrule Oct 20 '09 at 13:07
1  
Open Notepad. Type 1 Billion times "A". Save, then compress. WOW! Create an app that writes 1 Billion (true) random numbers to a file. Compress that. HUH? –  igrimpe Dec 28 '12 at 8:51
add comment

9 Answers

up vote 8 down vote accepted

If time taken to compress the data is not an issue, then you can optimize compressed size by using several different tools together.

Compress the data several times using different tools like 7zip, winrar (for zip) and bjwflate.

Next, run deflopt on each archive to reduce each archive a little more.

Finally, run zipmix on the collection of archives. Since different zip tools are better on different files, zipmix picks the best compressed version of each file from each of the archives and produces an output which is smaller than any that any of the zip tools could have produced individually.

You should note however that this is not guaranteed to work any kind of magic on your files. Certain types of data simply do not compress very well, like JPEGs and MP3s. These files are, to put it simply, already zipped internally.

share|improve this answer
3  
JPEGs and MP3s aren't zipped. They are compressed but not zipped. –  KovBal Jul 21 '09 at 18:27
    
@davr: You're absolutely right. I suggest removing both your and my comments. –  Georg Schölly Feb 27 '11 at 22:55
    
This is incredibly poor advice, trying to use multiple compression algorithms sequentially is a very bad idea. Each algorithm creates a compressed file + overhead, so by using multiple you're actually adding data to the data you're trying to compress - it's like trying to dig a hole in the sand, the deeper you go the more sand pours in on you. You're far better off using a single good algorithm at maximum compression settings. –  Tacroy May 5 '12 at 0:02
    
I think you misunderstand.. the same data is not being recompressed repeatedly. Rather you are simply choosing the best single algorithm on a per file basis rather than per archive. –  izb May 5 '12 at 6:54
3  
Compress the data several times is pretty misleading. –  ta.speot.is Mar 11 '13 at 4:24
add comment

This depends entirely on the data being compressed.

Text compresses very well, binary formats not so well and compressed data (mp3, jpg, mpeg) not at all.

Here is a good Compression Comparison Table from wikipedia.

share|improve this answer
    
I am aware that compression depends upon the type of data, but are there any specific techniques that help you compress files further? –  rzlines Jul 17 '09 at 10:35
3  
Once you have compressed something it's usually impossible to get it measurably smaller. You just have to select the appropriate compression method for your data. –  Nifle Jul 17 '09 at 10:39
    
Text can easily be compressed up to 90%. –  Georg Schölly Jun 13 '10 at 7:32
add comment

Squeezechart.com contains comparisons of various compression rates. Although, as stated by Nifle's answer - you're unlikely to get such high compression rates for binary formats.

share|improve this answer
add comment

Most compression tools have settings to allow you to achieve a higher compression rate at a compromise of slower compression/decompression times and more RAM usage.

For 7-Zip, search for "Add to Archive Dialog Box" in the built-in help for more detail.

share|improve this answer
add comment

This question is long dead, however previous answers are wrong by an order of magnitude!

The best compression algorithm that I have personal experience with is paq8o10t

Hint: the command to compress files_or_folders would be like:

paq8o10t -5 archive files_or_folders

A slightly better compression algorithm, and winner of the Hutter Prize, is decomp8 (see link on prize page). However, there is no compressor program that you can actually use.

Update: For really large files lrzip can achieve compression ratios that are simply comical.

An example from README.benchmarks:


Let's take six kernel trees one version apart as a tarball, linux-2.6.31 to linux-2.6.36. These will show lots of redundant information, but hundreds of megabytes apart, which lrzip will be very good at compressing. For simplicity, only 7z will be compared since that's by far the best general purpose compressor at the moment:

These are benchmarks performed on a 2.53Ghz dual core Intel Core2 with 4GB ram using lrzip v0.5.1. Note that it was running with a 32 bit userspace so only 2GB addressing was posible. However the benchmark was run with the -U option allowing the whole file to be treated as one large compression window.

Tarball of 6 consecutive kernel trees.

Compression    Size                 Percentage      Compress    Decompress
None           2373713920           100             [n/a]       [n/a]
7z             344088002            14.5            17m26s      1m22s
lrzip          104874109            4.4             11m37s      56s
lrzip -l       223130711            9.4             05m21s      1m01s
lrzip -U       73356070             3.1             08m53s      43s
lrzip -Ul      158851141            6.7             04m31s      35s
lrzip -Uz      62614573             2.6             24m42s      25m30s
share|improve this answer
    
It's optimized to provide maximum compression ratio, but is enormously slower than near-contenders. –  Eric J. Mar 15 '13 at 12:57
    
@Eric J. yes, but the question didn't specify speed of compression/decompression ;) –  Alexander Riccio Jan 8 at 7:33
add comment

Your best bet here seems to be trial and error. Try all your available compression techniques on each file and pick the best to put on your website. Luckily computers do this sort of thing pretty fast and don't get bored. You could write a simple script to automate the process so it would be "relatively painless". I'd probably write the script in python but you might have a better choice available.

Just don't expect miracles - 700 mb down to 30 mb just doesn't happen that often. Log files as mentioned above - yes. "Your average file" - no way.

Hotei

share|improve this answer
add comment

Just visit http://www.maximumcompression.com/

Best compression list http://www.maximumcompression.com/data/summary_mf2.php It lists complete compression benchmark, Best tool till now is FreeArc.

share|improve this answer
add comment

Nanozip seems to have highest compression together with FreeArc. But it is not in final version yet. There is how good compression Nanozip achieves. It has very high compression and it does not takes too much time see http://www.maximumcompression.com/data/summary_mf.php#data, but FreeArc is faster.

share|improve this answer
add comment

You may try 7zip with the following ultra settings:

7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on big_file.mysql.7z big_file.mysql
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.