Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a bunch of gzip files that I have to convert to bzip2 every now and then. Currently, I'm using a shell script that simply 'gunzip's each file and then 'bzip2's it. Though this works, it takes a lot of time to complete.

Is it possible to make this process more efficient? I'm ready to take a dive and look into gunzip and bzip2's source codes if necessary, but I just want to be sure of the payoff. Is there any hope of improving the efficiency of the process?

share|improve this question

6 Answers 6

up vote 9 down vote accepted

Rather than gunzip in one step and bzip2 in another, I wonder if it would perhaps be more efficient to use pipes. Something like gunzip --to-stdout foo.gz | bzip2 > foo.bz2

I'm thinking with two or more CPUs, this would definitely be faster. But perhaps even with only a single core. I shamefully admit to not having tried this out, though.

share|improve this answer
2  
+1 for piping, disk I/O is something you want to avoid. As for compression, unless I'm mistaking, bzip2 isn't parallell. You'd have to use something like pbzip2 to compress in parallell: compression.ca/pbzip2 –  gustafc Aug 17 '09 at 7:01
    
... and unfortunately, there doesn't seem to be any parallell gzip decompression utility available. –  gustafc Aug 17 '09 at 7:07
    
@gustafc: Thanks for the link to pbzip2, that was very helpful... @OP: I shied away from piping bcos I want to be able to deal with corrupt gz files, etc., without losing them in the pipe... –  sundar Aug 18 '09 at 5:32
4  
@gustafc: Even if bzip2 and gzip don't work in parallel internally, by using a pipe you can have them work in parallel, because a pipe implicitly starts two processes, which will run in parallel. So at least decompression and compression will run in parallel. –  sleske Apr 17 '11 at 18:49

What you're currently doing is your best bet. There is no conversion tool available, and attempting to bzip2 an already gzipped file is not really an option, as it frequently has undesired effects. Since the algorithm is different, converting would involve retrieving the original data regardless. Unless of course gzipping was a step in the bzip2 process, in which it isn't unfortunately.

share|improve this answer
    
Don't the algorithms have any overlapping steps such that I could skip one step in gzip decompression and the same in bzip compression also? –  sundar Aug 19 '09 at 6:54
1  
@sundar I wouldn't think so. gzip uses Leimpel-Ziv 77, while bzip2 uses Burrows-Wheeler. Different algorithms, I'm afraid. –  new123456 Jul 3 '11 at 14:21

GNU parallel (http://www.gnu.org/software/parallel) might be an option if you have multiple cores (or even multiple machines):

ls *.gz | parallel "gunzip -c {} | bzip2 > {.}.bz2"

Read the tutorial / man page for details and options.

share|improve this answer

Occasionally, I need to do the same thing with log files. I start with the smallest *.gz files first (ls -rS), gunzip and then and bzip2 them individually. I do not know if it is possible to direct the gunzip output directly to the bzip2 input. The bzip2 command is so much slower at compressing than gunzip is at decompression that it may consume the memory and swap space on the host.

Improvements or suggestions are welcome. Here is my one liner:

for i in $(ls -rS *.gz | sed 's/\.gz//'); do gunzip ${i}.gz; bzip2 -9 ${i}; done
share|improve this answer
    
Thanks for the input, the point about the difference in speed between the two processes and its implication is an important one. –  sundar Dec 15 '12 at 16:46

If you have more than a few, check out the LJ article with a nice shell script.

http://linuxgazette.net/123/bechtel.html

7zip gets better compression, and is multi threaded.

share|improve this answer

Just had to do this a few minutes ago:

find . -name "*.gz" | perl -pi -e 's/\.gz$//g;' | xargs -n1 ./rezip

Where rezip would be defined as:

#!/bin/bash
gunzip -v $1.gz && bzip2 -9v $1

Optionally, you can also make it multi-threaded by using a -P option with xargs, but be careful with that one. (Start low!)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.