Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I got a several of big files using command

mysql> select * into outfile ...

for example, dump_news.sql, dump_weibo.sql, dump_report.sql ...

The total size is about 7.6G, I use command

tar -zcvf dump.tar.gz dump_*

on linux, to compress the above files, however it takes about 4 hours. That's to long. I tested the tool of bzip2, it takes a longer time, but the size of the compressed file is smaller. So i want to know if there is a tool for me to finish the compressing more quickly.

share|improve this question

migrated from May 13 '12 at 4:16

This question came from our site for professional and enthusiast programmers.

Parallel gzip: Parallel bzip2: – Piskvor May 3 '12 at 4:27
do a gzip , should be faster – Satya May 3 '12 at 4:28
up vote 2 down vote accepted

A few months ago I had to compress my entire home directory, so I made a bash script that ran tar in multiple processes and kept track of the number of processes so that there were always 6 going at once (I have 4 cores, so the 6 threads was so that it could compress while waiting for the iowait since I was putting it over usb onto my hard drive).

My script is here:

If change the output folder to your desired output destination, put all the sql files in the same folder as the script, give it executable permissions, and run it from the terminal, it should compress your files.

You may also want to change the number of threads it runs to suit your own processor.

share|improve this answer

4 hours to compress ~8G is way too high. Explore @Piskvor suggestion.

Also instead of saving sql output to a file & compressing, can query & compress via pipe.

mysql -e "query" other_options | gzip > OUT_FILE

Also if you interested check gzip vs bzip vs lzma

share|improve this answer

First off, you can accept a little less compression for much higher speed. You can pipe the tar output to gzip, and give gzip a compression level parameter of -1 to -3 and see how that works for you.

Second, as @Piskvor noted, you can use multiple processors and cores with pigz, which can be used as a drop-in replacement for gzip.

If you want speed, bzip2 (including parallel bzip2) would be moving in the wrong direction.

share|improve this answer

lzop is faster in terms of cpu than gzip ( but gets less compression ), though you won't be able to get too much faster than that due to disk IO limitations, unless you have a solid state drive or raid array.

share|improve this answer

You must log in to answer this question.