I like to archive everything. Currently a small box of CDs holds most of my work and downloads from the past 15 years. Unfortunately, some of the CDs are starting to suffer bit rot, resulting in corrupted and unreadable files.
So I'm looking for the best system for this. I expect to put all the files on my hard drive, and rsync a copy onto an external drive, using <something> to manage it as a repository. Here are my top requirements:
- Simple recovery from bit rot and sector rot. It should be easy to verify the integrity of the entire repository, or identify the extent of data corruption. As long as data corruption is below a certain level, it should be easy to make a pristine clone out of a corrupted repository.
- Robust recovery from bit rot and sector rot. I should be allowed to choose sufficient levels of redundant storage to recover from arbitrary levels of data corruption.
- Open source. Since I'd like to rely on it for my lifetime, I don't want something that may disappear due to commercial whims. Also I want something that runs on Linux. Unlike this question I am willing to maintain and migrate as needed.
- Ability to store a lot. At present, I'd like to store a few hundred thousand files taking a few dozen gigs, but I'd prefer a tool that can expand beyond that.
- Bonus if it preserves unix timestamps and permissions. Bonus if it's easy to access the stored files without a separate extraction step. And bonus if it's a system that other people also use and understand.
Here are some suboptimal solutions I've considered. Feel free to explain to me why one of these is actually optimal, in case I didn't understand its true capability:
- Git: Easily verifies the integrity of the repository. But recovering from corruption seems very complicated and not robust. Git also seems to have trouble with large repos.
- Par2: Provides robust recovery from bit rot and sector rot, but it has a 32,768-file limit.
- Tar: Archives things, but doesn't seem to provide any recovery from bit rot that I can tell. Also, you have to untar the whole tarball to access the contents.
I suppose I could make a git repository of everything, make an extra tarball of it, and make par2 recovery files for the tarball. But it's also suboptimal to use a jury-rigged solution like this.