Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm currently writing an advanced starting script for my minecraft server. Everything works out as I want it too. But the problem is the backups. I created a function for it. It works also great and does exactly what i want it to do. But since the server has to be down during the backup and the backup itself takes ages (I mean hours) I need to speed the backup up!

My current backup works like this:

tar -cjvf $BACKUP_PATH/$filename $MINECRAFT_PATH

As I said this takes ages!

So I had the idea to just make a tar file without compressing then start the server and then compress the archive.

tar -cvf $BACKUP_PATH/$filename $MINECRAFT_PATH
bzip2 -9 $BACKUP_PATH/$filename
rm $BACKUP_PATH/$filename

This should work faster. But then i had an idea. And this is the point where i need you.

First I wanted to modify the actual backupping in a way that shows how much is already backed up and an estimation how long it will approximately take to finish. Therefor the output of tar would bes supressed, the command would be run in the background and a (while) loop (that waits for the command to finish) would calculate the percentage calculated and also calculate an estimate of the remaining time.

To to this I would first make a call to du to get the size of all files being compressed. This would be saved in a variable. the i would get the current timestamp. Also saved in a variable. At this point the loop would start. Then in the loop the percentage and the remaining time would be calculated (simple...) and this would be printed in two lines that always are updated. And this is my major problem. I would be very happy if someone could help me with the loop!

Some basic code:

tar -cf $BACKUP_PATH/$filename $MINECRAFT_PATH &

backupsize=$(du -bs $MINECRAFT_PATH)
starttimestamp=$(date +%s)

# Here is the loop (how do I stay here while the tar command is still running?
# And how do i update the two lines with the newly created numbers?


bzip2 -9 $BACKUP_PATH/$filename
rm $BACKUP_PATH/$filename

If you are in the helpy mood you can help me getting this kind of thing also to work for the bzip command. If it's not possible I will just leave it! (bzip accepts the same flags as zip)

share|improve this question
Predicting things is a big hassle. And I IMHO don't get why you need to know the time it takes to compress the .tar archive. – Doktoro Reichard Aug 24 '13 at 16:14
So, in order to make things faster, you wan to add operations that take time (du) ? That does not make much sense. In any case, you can't really predict how long it will take since the compression speed depends directly on the data being compressed and can be very different even for files of the same size. – terdon Aug 24 '13 at 16:17
I don't need it. I just think this is a nice feature to know how long operations approximately will take. And as I said, I could easily live without it. – BrainStone Aug 24 '13 at 16:17
This being said, not compressing the archive first will indeed save you the time, the drawback being that while your server is up the compression will also be up, so users might experience lag. Have you compared uncompressed-compressed sizes to make sure it is worth the cost? If the compression ratio is at 90%, forget it, in my opinion. – Doktoro Reichard Aug 24 '13 at 16:17
Do all files in the $MINECRAFT_PATH folder always change? If not try the following: 0) Stop server. 1) Use rsync to copy all data to another place on the server (no compression, local disk to local disk). Rsync will skip files which did not change. 2) restart server 3) Store the backup offsite. E.g. copy and optionally compress the data, then scp. Or also rsync it over a slow (LAN/WAN) link. – Hennes Aug 24 '13 at 16:54
up vote 3 down vote accepted

You could use rsync to create a backup copy of your $MINECRAFT_PATH. This will only copy those parts of the files that have been changed since the last run, so it should be relatively fast. (except for the first run ever, because then it has to copy everything) Once the rsync is done, you can restart the server. Then you use tar to make a tarball of the backup copy of $MINECRAFT_PATH.

Having a backup copy right on the machine is also nice in case something goes wrong.

Regarding compression, I would recommend using standard gzip compression instead of bzip2. When I did a test compressing OS dumps, gzip file sizes were around 40% of the uncompressed file. Compression speed was 11.5-17.7 MB/s. When I used bzip2, the compressed file sizes were 32-38% of the uncompressed file, but the compression speed was down to 5.7-7.3 MB/s. If you have a multi-core machine, you might try pbzip2, which uses the multiple processors to speed things up. But it will probably still be slower than gzip. Also, using an external decompression program complicates restoring backups.

share|improve this answer
We seem to have more or less the same idea. (Though mine is in a comment and you went much deeper into the compression part). +1. – Hennes Aug 24 '13 at 16:57
@RolandSmith What flags do i have to use if I want to delete files that no longer exist? Meaning what flags do I have to set in order to make the temp directory an exact copy of the main directory without copying and deleting unecessary files? – BrainStone Aug 24 '13 at 20:50
Would -r (-rh --progress in my case) work? – BrainStone Aug 24 '13 at 21:28
@BrainStone Just use -av --delete. – Roland Smith Aug 24 '13 at 21:48
@RolandSmith Is --delete really the right thing/flag? I mean: delete extraneous files from destination dirs I do not want to delete anything from the destination! – BrainStone Aug 24 '13 at 22:06
  1. manipulate exported BZIP2 variable. by default bzip2 uses -3 or -4 compression ratio. by setting export BZIP2 variable you can tell to bzip2 started from tar (tar -j) about compression ratio. see to page
  2. try different ratios for speed optimize. no compression is not best speed because you have hardware speed limitation. the best compression is slow becauses utilize CPU. thats why -3 or -4 ratio is default, but you should make experiment. after backup you can repack archive:

    bzip2 -d archive_after_repack.bz2 but better (and on time slower) compress ratio you use using xz archiver.

  3. using rsync is not the best idea because you must have second full data copy.

  4. try mix differential and full backups. you can use option --newer=DATE-OR-FILE (man tar)
  5. use tar with --checkpoint or --checkpoint-action , this help you for getting progress. you can calculate this and get estimations.
  6. if it is possible use file system snapshot. you can combine this using lvm with ext3,ext4,jfs or xfs. If you can you may use zfs support but this is not officially supported. Snapshot making is very fast. You musn't stop services. After snapshot you have frozen compact file system whose you can backup as long as it's needed. after backup you remove snapshot.
share|improve this answer
How would I do that exporting? (BZIP2 variable) – BrainStone Sep 8 '13 at 14:18
very simple. first export BZIP2=-6 for example, and second as usual tar cvjf somefile.tar.bz2 /src/files . tar runs bzip2 following -j tar option and pass prepared BZIP2 variable. Runned bzip2 program read BZIP2 variable the same way as given to it in the command line parameters. Of course you can set export BZIP2=-9v for display compress ratio. See man bzip2 :) – Znik Sep 10 '13 at 7:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .