I'm trying to analyze heap dumps from a server which are fairly large files (10-15 GB). I create those files on the server and want to analyze them on my machine, so for downloading them, I tried compressing them with both bzip and gzip. Both programs consistently produce corrupted files that they can't decompress anymore.

I'm using ext3 with a block size of 4 KiB, so the file size limit should be 2 TiB and therefore irrelevant in my case. I'm using gzip 1.3.12 and bzip 1.0.5 on a Ubuntu Jaunty, 64-bit server edition, in a mostly vanilla state (only added some packages, nothing fancy).

There is a RAID-1 running, but it reports no synchronization problems or delays.

The dumps are created with jmap.

Is there any particular type of data that makes those programs choke?

Is the size a problem?

What could I try to find out more or circumvent the problem?

  • Are you doing your test decompressions on the same machine that you do the compression on? – Paul Nov 10 '11 at 10:52
  • Does gzip and/or bzip2 exit successfully ? (gzip file; echo $?) At which point is the file corrupted, before or after the transfer ? (That is, if you try to gunzip/bunzip2 the file where it was created, is it corrupted already at that stage ?) – Kjetil Jørgensen Nov 10 '11 at 12:00
  • Also, which versions of gzip and bzip2 are you using ? (--version) Apparently, older gzip versions had problems decompressing files larger than 4Gb. gzip.org/#faq10 – Kjetil Jørgensen Nov 10 '11 at 12:22
  • Have you run out of space for intermediate files? – pjc50 Nov 10 '11 at 12:40
  • @Paul - Yes, I am, with the -t flag. – Hanno Fietz Nov 11 '11 at 10:14

If this is more than one file, try putting them in a tar archive first

tar czvf dumps.tar.gz file1 file2

or for bzip compression

tar cjvf dumps.tar.bz2 file1 file2

I've never had any problems with either method on numerous systems and filesystems.

Will also work for 1 file of course!

  • Seems to work. Weird. Do you have any idea why this might be? – Hanno Fietz Nov 11 '11 at 13:37
  • I can only guess that passing it through tar resolves some sort of timing issue. I have no end of trouble with zipping on various NFS systems at work, and yet tar works every time. In my case the filesystems are over half the planet away, but I've had issues with gzip at home too, and that's all local. I just tar things up by default now. – Steve Nov 15 '11 at 19:24

gzip versions 1.2.4 and older has problems decompressing files larger than 4Gb (see: http://www.gzip.org/#faq10)

According to bzip2's changelog it seems to also have had some trouble with larger files prior to verision 1.0.0

  • 1
    This is useful information. My versions are higher, though. (Will edit them into the original question) – Hanno Fietz Nov 11 '11 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.