Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I got a several of big files using command

mysql> select * into outfile ...

for example, dump_news.sql, dump_weibo.sql, dump_report.sql ...

The total size is about 7.6G, I use command

tar -zcvf dump.tar.gz dump_*

on linux, to compress the above files, however it takes about 4 hours. That's to long. I tested the tool of bzip2, it takes a longer time, but the size of the compressed file is smaller. So i want to know if there is a tool for me to finish the compressing more quickly.

share|improve this question
1  
Parallel gzip: zlib.net/pigz Parallel bzip2: compression.ca/pbzip2 –  Piskvor May 3 '12 at 4:27
    
do a gzip , should be faster –  Satya May 3 '12 at 4:28
add comment

migrated from stackoverflow.com May 13 '12 at 4:16

This question came from our site for professional and enthusiast programmers.

4 Answers

up vote 2 down vote accepted

A few months ago I had to compress my entire home directory, so I made a bash script that ran tar in multiple processes and kept track of the number of processes so that there were always 6 going at once (I have 4 cores, so the 6 threads was so that it could compress while waiting for the iowait since I was putting it over usb onto my hard drive).

My script is here: https://gist.github.com/3989700

If change the output folder to your desired output destination, put all the sql files in the same folder as the script, give it executable permissions, and run it from the terminal, it should compress your files.

You may also want to change the number of threads it runs to suit your own processor.

share|improve this answer
add comment

4 hours to compress ~8G is way too high. Explore @Piskvor suggestion.

Also instead of saving sql output to a file & compressing, can query & compress via pipe.

mysql -e "query" other_options | gzip > OUT_FILE

Also if you interested check gzip vs bzip vs lzma

share|improve this answer
add comment

First off, you can accept a little less compression for much higher speed. You can pipe the tar output to gzip, and give gzip a compression level parameter of -1 to -3 and see how that works for you.

Second, as @Piskvor noted, you can use multiple processors and cores with pigz, which can be used as a drop-in replacement for gzip.

If you want speed, bzip2 (including parallel bzip2) would be moving in the wrong direction.

share|improve this answer
add comment

lzop is faster in terms of cpu than gzip ( but gets less compression ), though you won't be able to get too much faster than that due to disk IO limitations, unless you have a solid state drive or raid array.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.