I accidentally deleted a handful of gzipped files from a folder. Thankfully, I had uncompressed them in a different location, and am in the process of restoring them. I had the md5 checksums for the old (now deleted) files, but the checksums for the newly compressed files don't match. Crap.

But... I have another folder that contains similar gzipped files from the same source, and I when I gunzip and then immediately gzip one of those files, the checksum is again different, leading me to suspect that the originator of the files used different parameters for gzip (if there's an alternative explanation, I'd love to hear it).

Is there any way to identify the gzip parameters used so that I can verify that my manipulation hasn't messed up the contents of the files?


The standard Unix file utility gives you some basic info about a .gz file, e.g.:

$ file foo.gz
foo.gz: gzip compressed data, was "foo", from Unix, last modified: Tue Aug  1 14:19:21 2017, max compression

As you can see, the header stores the original filename, the OS on which the compression was performed, modification time, and compression level. Note that the original filename might be different if you did something like gzip -c tempfile > foo.gz, in which case the original filename would be tempfile. Or it might not even exist if gzip didn't get an original filename because it read from a stream (e.g., tar czf foo.tar.gz somedir).

So you probably want to get an idea of what factors might be different first. I don't know how important all this really is to you, but you could look at RFC 1952, which gives the file format. You could try different settings, and even hex-edit some of the fields to match the originator's if needed (e.g., different OS).

  • I didn't realize md5 looked at the header as well, though I suppose that makes sense. These files do have a different provenance (when they were decompressed, I concatenated them with a python script and then separated them). So in principal I'd have to reproduce the header to get the checksums to line up? – kevbonham Aug 2 '17 at 11:51
  • 1
    A checksum on a file generally treats it as just a stream of bytes, with no regard to its internal format/structure. So you'd need to produce a new file that is bit-for-bit identical with the original to get a matching checksum. – jjlin Aug 2 '17 at 18:48

All these utilities include some meta information that can change with each run so even with identical files you get slightly different ZIPs (and so a different MD5). To compare the contents you have to unzip them.

If you lookup GZIP in Wikipedia, you learn that a GZip file starts with a 10-byte header, containing a magic number (1f 8b), a version number and a timestamp. In other words, each run is guaranteed to give a different file.

  • This actually isn't necessarily true. If you gzip a file (on disk), checksum it, gunzip it, gzip it again, and checksum it again, you'll get matching checksums (or at least, I do). The timestamp is of the original file. If the original "file" wasn't on disk, then it'll be the current time. – jjlin Aug 1 '17 at 22:04
  • @jjin Correct. But if you do the very common tar -czf some.tar.gz bunch of files then the timestamp is the current time (modification of the on-the-fly tar). – xenoid Aug 1 '17 at 23:33
  • Yes, that's true for the case you describe. I'm just pointing out it's not true for all gzipped files. From my reading of the question, there is nothing that strongly suggests that OP is working with .tar.gz files. It actually looks more like he's working with individual .gz files. – jjlin Aug 2 '17 at 0:14
  • So, I suppose an updated question - is there any way for me to replicate the checksum that was initially done in order to compare, it is that just not how md5 checksum works? Recall I have another set of unadulterated files that I could potentially mimic. – kevbonham Aug 2 '17 at 11:45
  • Comparing the MD5 hashes of GZIP files is fraught with peril (and false negatives). But you can gunzip them and compare the MD5 hashes of the uncompressed files and these should match. – xenoid Aug 2 '17 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.