3

I'm trying to analyze heap dumps from a server which are fairly large files (10-15 GB). I create those files on the server and want to analyze them on my machine, so for downloading them, I tried compressing them with both bzip and gzip. Both programs consistently produce corrupted files that they can't decompress anymore.

I'm using ext3 with a block size of 4 KiB, so the file size limit should be 2 TiB and therefore irrelevant in my case. I'm using gzip 1.3.12 and bzip 1.0.5 on a Ubuntu Jaunty, 64-bit server edition, in a mostly vanilla state (only added some packages, nothing fancy).

There is a RAID-1 running, but it reports no synchronization problems or delays.

The dumps are created with jmap.

Is there any particular type of data that makes those programs choke?

Is the size a problem?

What could I try to find out more or circumvent the problem?

8
  • Are you doing your test decompressions on the same machine that you do the compression on?
    – Paul
    Nov 10, 2011 at 10:52
  • Does gzip and/or bzip2 exit successfully ? (gzip file; echo $?) At which point is the file corrupted, before or after the transfer ? (That is, if you try to gunzip/bunzip2 the file where it was created, is it corrupted already at that stage ?) Nov 10, 2011 at 12:00
  • Also, which versions of gzip and bzip2 are you using ? (--version) Apparently, older gzip versions had problems decompressing files larger than 4Gb. gzip.org/#faq10 Nov 10, 2011 at 12:22
  • Have you run out of space for intermediate files?
    – pjc50
    Nov 10, 2011 at 12:40
  • @Paul - Yes, I am, with the -t flag. Nov 11, 2011 at 10:14

2 Answers 2

4

If this is more than one file, try putting them in a tar archive first

tar czvf dumps.tar.gz file1 file2

or for bzip compression

tar cjvf dumps.tar.bz2 file1 file2

I've never had any problems with either method on numerous systems and filesystems.

Will also work for 1 file of course!

2
  • Seems to work. Weird. Do you have any idea why this might be? Nov 11, 2011 at 13:37
  • I can only guess that passing it through tar resolves some sort of timing issue. I have no end of trouble with zipping on various NFS systems at work, and yet tar works every time. In my case the filesystems are over half the planet away, but I've had issues with gzip at home too, and that's all local. I just tar things up by default now.
    – Steve
    Nov 15, 2011 at 19:24
3

gzip versions 1.2.4 and older has problems decompressing files larger than 4Gb (see: http://www.gzip.org/#faq10)

According to bzip2's changelog it seems to also have had some trouble with larger files prior to verision 1.0.0

1
  • 1
    This is useful information. My versions are higher, though. (Will edit them into the original question) Nov 11, 2011 at 13:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .