0

I am using the ideas laid out in this article to create incremental versioned backups of my data. I basically sync the data to a current folder in my backup destination and then create a date folder with hard links to the current folder. I end up with this:

$ ls
...
2019-01-01_10-00-01
2019-01-02_10-00-01
...
2019-02-15_10-00-01
...
current

It works great. If I ever need to do a full restore from a specific date, I can just restore everything from that date's folder.

But if you're looking for previous versions of a specific file, you have to go through each of the date folder's to find what you want. I want to create another folder that has a running total of all files, each time they were changed. A combined view if you will.

I came up with this, and it works, but I am wondering if there is a more elegant, standard way to do this.

#!/bin/bash

NOW=$(/bin/date +\%Y-\%m-\%d_\%H-\%M-\%S)

# the data that needs to be backed up
SOURCES=("/path/to/source 1" "/path/to/source 2")

# where it is going
DESTINATION="/path/to/backup"

# make sure the destination exists
mkdir -p "$DESTINATION"

# make sure there is a place to put the current data
mkdir -p "$DESTINATION/current"

# make sure there is a place to put the "combined" data
mkdir -p "$DESTINATION/combined"

# sync the data
rsync -v -a --delete "${SOURCES[@]}" "$DESTINATION/current"

# check if files were backed up
# any file with only one link is either new, and needs to have a hard link version
# or it wasn't fully backed up previously and needs a hard link version
if [[ $(find "$DESTINATION/current" -type f -links 1 | wc -l) -ne 0 ]] ; then
    # make a date folder backup using hard links
    cp -al "$DESTINATION/current" "$DESTINATION/$NOW"

    # make a combined view
    #  - find all files with 2 links
    #    - one link is to the file in the $DESTINATION/current
    #    - the other link is to the file in $DESTINATION/$NOW
    # - there should never be any files with only 1 hard link since the previous command
    #   is sure to have created a second link
    # - any files with more than 2 links were, hopefully, already covered during a previous iteration
    cd "$DESTINATION/current" && find * -type f -links 2 -print0 | while IFS= read -r -d $'\0' filePath
    do
        fileName="$(basename "$filePath")"
        fileFolder="$(dirname "$filePath")"

        # where the file will live in the combined folder
        # need to mirror the folder structure
        destinationFolder="$DESTINATION/combined/$fileFolder"
        mkdir -p "$destinationFolder"

        # make a hard link to it
        cp -al "$filePath" "$destinationFolder/$fileName.$NOW"
    done
fi

The code does work. After a few iterations, this is what it creates:

Files in the current folder (this is a "live" copy of the source data):

backup/current/source 1/001
backup/current/source 1/002
backup/current/source 1/003
backup/current/source 1/file 100
backup/current/source 1/folder/004
backup/current/source 2/006

Files in the date specific folders (note files from the first backup have files that aren't in the second because they were deleted):

backup/2019-01-15_23-08-02/source 1/001
backup/2019-01-15_23-08-02/source 1/002
backup/2019-01-15_23-08-02/source 1/003
backup/2019-01-15_23-08-02/source 1/file 100
backup/2019-01-15_23-08-02/source 1/folder/004
backup/2019-01-15_23-08-02/source 1/folder/005
backup/2019-01-15_23-08-02/source 2/006
backup/2019-01-15_23-08-02/source 2/007

backup/2019-01-15_23-09-00/source 1/001
backup/2019-01-15_23-09-00/source 1/002
backup/2019-01-15_23-09-00/source 1/003
backup/2019-01-15_23-09-00/source 1/file 100
backup/2019-01-15_23-09-00/source 1/folder/004
backup/2019-01-15_23-09-00/source 2/006

And these are the files in the combined view:

backup/combined/source 1/001.2019-01-15_23-08-02
backup/combined/source 1/002.2019-01-15_23-08-02
backup/combined/source 1/003.2019-01-15_23-08-02
backup/combined/source 1/003.2019-01-15_23-09-00
backup/combined/source 1/file 100.2019-01-15_23-08-02
backup/combined/source 1/folder/004.2019-01-15_23-08-02
backup/combined/source 1/folder/004.2019-01-15_23-09-00
backup/combined/source 1/folder/005.2019-01-15_23-08-02
backup/combined/source 2/006.2019-01-15_23-08-02
backup/combined/source 2/006.2019-01-15_23-09-00
backup/combined/source 2/007.2019-01-15_23-08-02

This way, if I need to find a previous version of source 1/folder/004, I just need to go to it's matching folder in backup/combined/ (backup/combined/source 1/folder) and all the 004 files are there, with a date/time stamp appended.

Is there a better, more elegant way to do this?

  • Might just want to use backup software – Xen2050 Jan 16 at 6:24
  • None do what I want. And most are unnecessarily bloated. Why use them when a small script works. The code I have works and does what I want but I thought there might be a more apt way to do it -- like some Linux command that I might not know about. – IMTheNachoMan Jan 17 at 1:03
  • Even duplicity? – Xen2050 Jan 17 at 1:42
  • No. In fact, I think each archive of duplicity only has the changed files so there is no way to view a point in time archive of all the files and you can't browse all files in a combined folder structure to quickly find a previous version of a specific file. – IMTheNachoMan Jan 17 at 2:26
  • Actually, duplicity has a --time option to "Specify the time from which to restore or list files." I think files are normally compressed, so browsing them immediately may not be possible, but they would take up less space (and only the changes from large files are saved, instead of a whole file copy), and extracting only a few compressed & encrypted files from a specific date should be really easy – Xen2050 Jan 17 at 2:35

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.