I got a several of big files using command

mysql> select * into outfile ...

for example, dump_news.sql, dump_weibo.sql, dump_report.sql ...

The total size is about 7.6G, I use command

tar -zcvf dump.tar.gz dump_*

on linux, to compress the above files, however it takes about 4 hours. That's to long. I tested the tool of bzip2, it takes a longer time, but the size of the compressed file is smaller. So i want to know if there is a tool for me to finish the compressing more quickly.

migrated from stackoverflow.com May 13 '12 at 4:16

This question came from our site for professional and enthusiast programmers.


A few months ago I had to compress my entire home directory, so I made a bash script that ran tar in multiple processes and kept track of the number of processes so that there were always 6 going at once (I have 4 cores, so the 6 threads was so that it could compress while waiting for the iowait since I was putting it over usb onto my hard drive).

My script is here: https://gist.github.com/3989700

If change the output folder to your desired output destination, put all the sql files in the same folder as the script, give it executable permissions, and run it from the terminal, it should compress your files.

You may also want to change the number of threads it runs to suit your own processor.


4 hours to compress ~8G is way too high. Explore @Piskvor suggestion.

Also instead of saving sql output to a file & compressing, can query & compress via pipe.

mysql -e "query" other_options | gzip > OUT_FILE

Also if you interested check gzip vs bzip vs lzma


First off, you can accept a little less compression for much higher speed. You can pipe the tar output to gzip, and give gzip a compression level parameter of -1 to -3 and see how that works for you.

Second, as @Piskvor noted, you can use multiple processors and cores with pigz, which can be used as a drop-in replacement for gzip.

If you want speed, bzip2 (including parallel bzip2) would be moving in the wrong direction.


lzop is faster in terms of cpu than gzip ( but gets less compression ), though you won't be able to get too much faster than that due to disk IO limitations, unless you have a solid state drive or raid array.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy