What to use for long term data archival?

Question

I like to archive everything. Currently a small box of CDs holds most of my work and downloads from the past 15 years. Unfortunately, some of the CDs are starting to suffer bit rot, resulting in corrupted and unreadable files.

So I'm looking for the best system for this. I expect to put all the files on my hard drive, and rsync a copy onto an external drive, using <something> to manage it as a repository. Here are my top requirements:

1. Simple recovery from bit rot and sector rot. It should be easy to verify the integrity of the entire repository, or identify the extent of data corruption. As long as data corruption is below a certain level, it should be easy to make a pristine clone out of a corrupted repository.
2. Robust recovery from bit rot and sector rot. I should be allowed to choose sufficient levels of redundant storage to recover from arbitrary levels of data corruption.
3. Open source. Since I'd like to rely on it for my lifetime, I don't want something that may disappear due to commercial whims. Also I want something that runs on Linux. Unlike this question I am willing to maintain and migrate as needed.
4. Ability to store a lot. At present, I'd like to store a few hundred thousand files taking a few dozen gigs, but I'd prefer a tool that can expand beyond that.
5. Bonus if it preserves unix timestamps and permissions. Bonus if it's easy to access the stored files without a separate extraction step. And bonus if it's a system that other people also use and understand.

Here are some suboptimal solutions I've considered. Feel free to explain to me why one of these is actually optimal, in case I didn't understand its true capability:

• Git: Easily verifies the integrity of the repository. But recovering from corruption seems very complicated and not robust. Git also seems to have trouble with large repos.
• Par2: Provides robust recovery from bit rot and sector rot, but it has a 32,768-file limit.
• Tar: Archives things, but doesn't seem to provide any recovery from bit rot that I can tell. Also, you have to untar the whole tarball to access the contents.

I suppose I could make a git repository of everything, make an extra tarball of it, and make par2 recovery files for the tarball. But it's also suboptimal to use a jury-rigged solution like this.

Answer 1

IMHO the best archiving method is (now) an second computer with FreeBSD + ZFS. For example 3xHDD inside, 2x1TB with raid-Z ZFS filesystem, and one small for the freebsd and temp files.

While it is not as cheap as CD burning, have some advantages:

• immediately online all your files - without need search for slow CDs

• the space will be never issue. (zfs is a 128bit filesystem, and you can address every atom in the universe with it...) - so simple add another hdd and with one command you run towards. (if you have hot swappable hdds you can do this in live, running system)

• zfs can detect if the files got corrupted (for example bad block on the hdd), every block is checksumed.

• drive failure? - no problem - replace and go... (hm. this sound as "Wash & Go") :)

• zilion other very cool features - search google for "ZFS advantages" (especially read about the copy-on-write and snapshots and clones)

• and at the last - you can forget fsck - will never need it.

If you mount it over a gigabit LAN, your NAS (in some situations) can be faster as your local HDD. ;)

You will must do only one thing - every X years replace HDD what get broken.

IMHO the ZFS is today the only solution when you don't need make external backups. (Because you can roll back deleted files too.)

Answer 2

No storage is forever, thus my suggestion is to use "common" backup utilities, like rsync, discs or whatever, and refresh them in intervals (lets say once a year).