up vote 3 down vote favorite
share [g+] share [fb]

I have a bunch of gzip files that I have to convert to bzip2 every now and then. Currently, I'm using a shell script that simply 'gunzip's each file and then 'bzip2's it. Though this works, it takes a lot of time to complete.

Is it possible to make this process more efficient? I'm ready to take a dive and look into gunzip and bzip2's source codes if necessary, but I just want to be sure of the payoff. Is there any hope of improving the efficiency of the process?

link|improve this question
100% accept rate
feedback

3 Answers

up vote 7 down vote accepted

Rather than gunzip in one step and bzip2 in another, I wonder if it would perhaps be more efficient to use pipes. Something like gunzip --to-stdout foo.gz | bzip2 > foo.bz2

I'm thinking with two or more CPUs, this would definitely be faster. But perhaps even with only a single core. I shamefully admit to not having tried this out, though.

link|improve this answer
1  
+1 for piping, disk I/O is something you want to avoid. As for compression, unless I'm mistaking, bzip2 isn't parallell. You'd have to use something like pbzip2 to compress in parallell: compression.ca/pbzip2 – gustafc Aug 17 '09 at 7:01
... and unfortunately, there doesn't seem to be any parallell gzip decompression utility available. – gustafc Aug 17 '09 at 7:07
@gustafc: Thanks for the link to pbzip2, that was very helpful... @OP: I shied away from piping bcos I want to be able to deal with corrupt gz files, etc., without losing them in the pipe... – sundar Aug 18 '09 at 5:32
2  
@gustafc: Even if bzip2 and gzip don't work in parallel internally, by using a pipe you can have them work in parallel, because a pipe implicitly starts two processes, which will run in parallel. So at least decompression and compression will run in parallel. – sleske Apr 17 '11 at 18:49
feedback
up vote 2 down vote

What you're currently doing is your best bet. There is no conversion tool available, and attempting to bzip2 an already gzipped file is not really an option, as it frequently has undesired effects. Since the algorithm is different, converting would involve retrieving the original data regardless. Unless of course gzipping was a step in the bzip2 process, in which it isn't unfortunately.

link|improve this answer
Don't the algorithms have any overlapping steps such that I could skip one step in gzip decompression and the same in bzip compression also? – sundar Aug 19 '09 at 6:54
@sundar I wouldn't think so. gzip uses Leimpel-Ziv 77, while bzip2 uses Burrows-Wheeler. Different algorithms, I'm afraid. – new123456 Jul 3 '11 at 14:21
feedback
up vote 1 down vote

If you have more than a few, check out the LJ article with a nice shell script.

http://linuxgazette.net/123/bechtel.html

7zip gets better compression, and is multi threaded.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.