Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I want to be able to compress a file losslessly, and if the original file is identical to another user's file, I want both of our compressed files to match, even if the original file dates are different.

I want to use a maximum of 1GB of RAM while compressing. I'm leaning towards an asymmetric algorithm because the files I have are fairly large, and they take at least an hour to compress with LZMA1 "ultra" in 7-zip on a P4 machine with 1GB RAM and nothing else running. I think 7-zip and FreeARC can be used for my purposes. I've tried to find the commands I should be using, but I'm not having much luck.

edit: 100% identical files should be produced, even if the dates of creation are different. This should be possible through --nodates in Freearc, and with ???? in 7-zip. I'm looking for an equivalent command for 7-zip, and a way to standardize compression across multiple computers.

share|improve this question
1  
How certain are you that LZMA is deterministic? –  Ignacio Vazquez-Abrams Nov 20 '10 at 16:31
2  
Expand on the not having much luck part. –  fideli Nov 20 '10 at 16:31
2  
Where's the question? Any compression program should be lossless unless it's specifically made to be lossy. –  Dennis Williamson Nov 20 '10 at 16:33

4 Answers 4

Create a couple of identical files:

$ echo hello > file1.test
$ echo hello > file2.test

gzip them...

$ gzip file1.test
$ gzip file2.test

observe timestamp field as the only difference:

$ hexdump file1.test.gz

0000000 8b1f 0808 TIME STMP 0300 6966 656c 2e31
0000010 6574 7473 cb00 cd48 c9c9 02e7 2000 3a30
0000020 0636 0000 0000                         

For more info on the timestamp, see the RFC

Now, you can either take an MD5 that starts after byte 8, zero these four bytes in your files and lose their timestamps, or extract the CRC16 from those gzips (also see the RFC for info on how to extract this)

Or, you could save without the timestamp:

$ echo test > file1.test
$ echo test > file2.test
$ gzip -n file1.test
$ gzip -n file2.test
$ md5sum file1.test.gz
cfe4ddf1c4c3891b4ff4a1269b42db82  file1.test.gz
$ md5sum file2.test.gz
cfe4ddf1c4c3891b4ff4a1269b42db82  file2.test.gz
share|improve this answer
    
And, if no other ports of gzip are available for Windows, then Cygwin provides it. –  Arjan Nov 20 '10 at 17:18

Not a direct answer to your question but it might be of help anyway.

Long time ago (a different millenium) I had the same problem. We wanted to know if compressed files where the same without decompressing them and comparing them.

Our solution was to get a md5sum of the file before compressing it, then we compressed the file and renamed it to md5sum.zip (.zip or .tar.gz or .rar or .whatever). That way we knew that if two files had the same name (without suffix) they where identical.

share|improve this answer
1  
Nice, though of course we knew should read we assumed ;-) –  Arjan Nov 20 '10 at 16:56
    
Also nice as it's totally independent of the compression rate: compressed files created by different programs or using different settings may still be recognised as being the same. –  Arjan Nov 20 '10 at 21:50

Hack into the sources of 7-zip. Where it reads the file date, just insert the code to change date to 01.01.1997, or anything other - fixed for all files. Compile with different name and use it.

share|improve this answer

pristine-tar has a hacked-up version of gzip that always produces the same results (and another for bzip2). Pick an algorithm variant and a timestamp and you're good to go.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.