Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to script a very simple backup strategy. Here is the general idea.

Daily - Backup the entire filesystem using rsync, overwriting the previous day's backup.

Weekly - Once a week copy the daily backup to a separate folder to keep around for a week, overwriting the previous week's backup.

Monthly - on the first of the month copy the daily backup to a montly backup folder to keep around for a month, overwriting last month's backup.

Here is the conundrum: Every day I do the weekly backup, the weekly and daily backups will be the same, so I won't have a few day old backup.

If this day falls on the first of the month all the backups will be the same, diminishing the whole point of having multiple backups.

I am limited on space and three backups is all I have room for. I am backing up VMs and websites so I don't need long term, but I do want backups that go back a while in case an error goes unnoticed for a few days.

Anyone have some ideas to rework this strategy? So I don't have periods where all the backups are the same.

share|improve this question
    
Can you use differental or incremental backups instead of full backups? –  Scott Chamberlain Jul 22 '13 at 14:04
1  
A problem I have seen with rsync is that it cannot copy small changes in a VM image. Instead, since the img is always different, the entire file gets copied (large .img files). I think if rdiff kept the rollbacks around, I would run out of space pretty quick. –  dan08 Jul 22 '13 at 14:10
1  
Buy more hard drive space then. There is not much that can be done. I have used rdiff a lot myself, but not on VM's I do know that it chunks the changes and those chunk sizes are configurable. It will be a trade off of data size vs signature size. –  Scott Chamberlain Jul 22 '13 at 14:16
2  
If you want to copy only differences inside a single file, you'll have to use a tool that knows how to do that, which rsync doesn't. (And that would cause heavy I/O.) Have you looked at rsnapshot? It wraps around rsync, uses hardlinks to de-duplicate data at the file level while providing the ability to restore a full backup from any backup target location, and can be configured for multiple rolling backup schedules. I added a script to clean out old backups if the backup disk starts filling up, and so far it's worked nicely (although I only have one backup schedule, for daily backups). –  Michael Kjörling Jul 22 '13 at 14:16
2  
Just make the weekly happen on a Thursday and daily backups happen everyday. This allows you to have say backups created on Thursday and Friday likewise monthly updates should happen on the first Monday or Tuesday –  Ramhound Jul 22 '13 at 15:07
show 4 more comments

3 Answers

up vote 2 down vote accepted

I would write a script that checks if a backup is more than 1,7 or 30 days old and acts accordingly. You have not said so but I assume you are using Linux (I added the tag to your question) and you are backing up to a remote server. The first step will be to write a little script that runs your rsync command and also creates a file on the remote server when the backup is finished. This will be used both to tell whether a backup is currently running and to check the backup's age (I assume you are keeping the original timestamps when you backup files, so you can't get the date from the files themselves):

Rsync script (this assumes you have password-less access to the remote server):

#!/usr/bin/env bash
ssh user@remote rm /path/to/daily/backup/backup_finished.txt
rsync /path/to/source/ user@remote:/path/to/daily/backup/
ssh user@remote touch /path/to/daily/backup/backup_finished.txt

On the local machine, set up a cron job that does daily backups:

@daily rsync_script.sh

On the remote machine, you need to run the script I give below every few hours:

@hourly check_backup.sh

The check_backup.sh script:

#!/usr/bin/env bash

daily=/path/to/daily;
weekly=/path/to/weekly;
monthly=/path/to/monthly;

## The dates will be measured in seconds since the UNIX epoch, 
## so we need to translate weeks and months (31 days) to seconds.
week=$((60*60*24*7));
month=$((60*60*24*31));  

## Make sure no backup is currently running
if [ ! -e $daily/backup_finished.txt ]; then 
 echo "A backup seems to be running, exiting." && exit;
fi

## Get the necessary dates
weekly_backup_date=$(stat -c %Y $weekly/backup_finished.txt)
monthly_backup_date=$(stat -c %Y $monthly/backup_finished.txt)
now=$(date +%s)
monthly_backup_age=$((now - monthly_backup_date))
weekly_backup_age=$((now - weekly_backup_date))

## Check the age of the daily backup and copy it accordingly
 if [[ "$monthly_backup_age" -gt "$month" ]]; then

    ## Copy unless the current $daily is identical to $weekly
    diff $daily $weekly > /dev/null ||
    ## Delete the previous backup and copy the new one over
    rm -rf $monthly && cp -rp $daily $monthly
fi
## Copy the weekly backup if it is older than a week but only
## if it is not identical to $monthly. The -r flag makes cp 
## recursive and the -p flag makes it preserve dates and permissions. 
if [[ "$weekly_backup_age" -gt "$week" ]]; then
    ## Copy unless the current $daily is identical to $monthly
    diff $daily $monthly > /dev/null ||
    rm -rf $weekly && cp -rp $daily $weekly
fi

So, this script (check_backup.sh) will be run every hour on your backup server. Since it does nothing unless the backup is old enough, it's no problem to have it run so often. Now, every time a daily backup is older than 31 days, it will be copied to the monthly directory and the contents of monthly will be deleted. Similarly for weekly when the backup is more than 7 days old.

I am using diff to compare the backups. This means that we will copy daily to weekly if the current weekly is more than a week old but only if the backup that will be copied (the current daily) is not the same as the existing weekly and similarly for monthly. For example, if the script has just run and it has seen that the monthly backup is the same as the current weekly one, it will not overwrite the existing monthly. However, one week later when the weekly will have changed, then it will copy the monthly one.

The net result of this is that at any time you should have a minimum of two different backups and usually you will have three. The worst case scenario is that something fails and you don't have a week old backup, just a month old one or, vice versa, you don't have a month old one but you do have last week's.

share|improve this answer
add comment

This is more of a long comment, adding to what others have already pointed out.

First, use hardlinks and incremental backups with rsync to greatly reduce the amount of actual disk space used: each extra backup will only take up the size of the files that differ. If you are backing up large VM images, then I'd suggest to not backup the image files, but actually their filesystem contents (as @Michael already commented). A tool like rsnapshot should work fine, although (from experience) it is easy enough to roll a script of your own.

Then remove old backups, keeping older ones at increasingly longer intervals. I once wrote a program precisely to allow to configure this, it can be found here (called bu-rmselect).

share|improve this answer
add comment

I am the author of the command line program timegaps, which allows you to filter periodically created backups and reject (list, delete, or move) older ones in a way so that the "time gaps" between accepted backups become larger with increasing age of backups.

Consider the following situation: all *.tar.gz files in the current working directory happen to be daily snapshots of something. The task now is to accept one snapshot for each of the last 20 days, one for each for the last 8 weeks, and one for each of the last 12 months, and to move all others to the directory notneededanymore. Using timegaps, this is a simple task:

$ mkdir notneededanymore
$ timegaps --move notneededanymore days20,weeks8,months12 *.tar.gz
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.