Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is it possible to gunzip multiple files and concatenate them into one big file, but do it in parallel given a multicore machine? For example, right now I do:

gunzip -c file1.gz > final
gunzip -c file2.gz >> final
gunzip -c file3.gz >> final
gunzip -c file4.gz >> final

Can I do the same so that the gunzip processing of the different files is done in different CPUs in the multicore machine and they are concatenated all into the same final file?

share|improve this question
    
Are you sure that it is CPU and not disk speed that is limiting? –  Daniel Andersson Oct 3 '12 at 14:45
add comment

2 Answers 2

It is a little shorter to do that using GNU Parallel:

parallel gunzip -c ::: file*.gz > final

but essentially it also writes to temporary files.

Watch the introvideos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

share|improve this answer
add comment

you have to use temporary files for this:

gunzip -c file1.gz > final &
one=$!
gunzip -c file2.gz > final2 &
two=$!
gunzip -c file3.gz > final3 &
three=$!
gunzip -c file4.gz > final4 &
four=$!

wait $one && wait $two && wait $three && wait $four
cat final2 >> final
cat final3 >> final
cat final4 >> final

to decompress the parts of a bigger file into the one final file you would have to know the decompressed size of the parts. only then you could create a biiiig empty file and write the output of the decompression to the right position in the big file (with dd for example). since you don't know the decompressed size (without decompressing the parts first) this won't work.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.