0

Let's have following task:

  • backup large amount (e.g. 500GB) of small files (few kB up to 1MB) on linux
  • the backup storage is mostly read-only
  • the storage is fast enough to access particular files in regular directory/file view, ideally through built-in or plugged-in function in regular file managers (like mc, TotalCommander (via samba) or so)
  • the storage should be ideally just single file (able to be effectively moved to nas or so)
  • no compression is needed
  • adding file(s) might be costly operation (even the initial storage initialization)

I tried plain old tar, but "opening" the index for 500G seems to be endless - so I'd probably need to extract it as a whole. Is there for example any way how to dd part of the filesystem into an image and then mount it?

Any thoughts?

3 Answers 3

1

You can indeed create a file and treat it like a block device. You might need to manually mount it though.

  1. The first step is to create the "block device" – you can use dd to do this (e.g. dd if=/dev/zero of=/path/to/file.name bs=100M count=6000) or other tool (fallocate, truncate).
  2. You then format the device using something like mkfs.ext4 /path/to/file.name.
  3. Next mount it – mkdir /mntpoint; moint /path/to/file.name /mntpoint.
  4. Copy files into /mntpoint using your preferred tool – for example rsnapshot, rsync or plain old cp.
  5. Unmount when you are done – making sure you are not in the /mntpoint directory, umount /mntpoint.
0

I do in such cases when one need quick access to backuped content - incremental backup using rsync with --link-dest= option.
It runs pretty fast, don't effect previous backup copies that you can have as many as you want and backup don't take a lot of space since backup copies are hard links to existent files. In this case access to backup is instant and actual file transfer takes a seconds even on huge amount of files since rsync copied only new files.

#!/bin/sh

srcDir='/importunt/data'  # Use full path
bkpDir='/backups'         # Use full path

cd "${bkpDir}"

previousDir="$(ls -td -- */ | head -n 1 | awk -F'/' '{print $1}')"   # Get most newest directory
currentDir="$(date '+%Y-%m-%dT%H;%M;%S')"

[ -n "${previousDir}" ] && {
  rsync_opts="-aPvz --safe-links --link-dest=${bkpDir}/${previousDir} --exclude=*.mp3"
} || {
  rsync_opts="-aPvz --safe-links --exclude=*.mp3"
}

mkdir -m 770 "${currentDir}"
rsync  ${rsync_opts}  "${srcDir}" "${bkpDir}"/"${currentDir}"

Basically such solution create exact snapshot in time, so restoration of files are pretty easy.

Don't be scary if you using du on /backups directory when it show increasing size on each update, if you will use df you will find that actual space isn't reduced. That is how hard links counted on Linux and FreeBSD, so no worry. To be make sure I don't lied you can check inode on some file in incremental backup with ls -i file. You will find that the same file in all directories has the same inode which is means rsync duplicate only filenames with hard links but all of them pointing to the same content.

One more advantage of this method is that you can delete oldest backup directories in any order, - latest, intermediate or the oldest ones.

Script above is simplified example. If content in incremental backup supposed to be edited then you shouldn't use mechanism of ls -t to detect newest previous directory in backup but instead save ${currentDir} to some file and restore to ${previousDir} on subsequent call.

Since rsync supports transfer over ssh you can move incremental backup to remote machine with the same efficiency, the only changes will be synchronized.

0

tar doesn't store a concise index at one point in the file (like Zip) - instead it declares each entity with the entity's data, hence the "seems to be endless" - you need to read the whole file to get a list of every entity inside it.

If you want to have easy access to the index, you could just capture the output of tar -cv, and store that alongside the archive.

tar -cv -f ./test.tar ./to_backup/ \
    > index.txt

Alternatively if you need extra information, you could use tar -cT ${FILE_LIST}, which accepts a list of files from ${FILE_LIST}. This way, you could use find to gather filenames, record details of each file into your "index" and produce the filename to stdout for tar to archive.

find ./to_backup/ -type f \
    | tee index.txt \
    | tar -cT /dev/stdin \
    > ./test.tar

Without compression, it is very easy to add files to a tar archive (tar stands for "Tape Archive"... shuffling data on tape is painful). A file format that has a concise index is going to be more difficult to add files to at a later date, though it is usually possible.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .