Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I like to archive everything. Currently a small box of CDs holds most of my work and downloads from the past 15 years. Unfortunately, some of the CDs are starting to suffer bit rot, resulting in corrupted and unreadable files.

So I'm looking for the best system for this. I expect to put all the files on my hard drive, and rsync a copy onto an external drive, using <something> to manage it as a repository. Here are my top requirements:

  1. Simple recovery from bit rot and sector rot. It should be easy to verify the integrity of the entire repository, or identify the extent of data corruption. As long as data corruption is below a certain level, it should be easy to make a pristine clone out of a corrupted repository.
  2. Robust recovery from bit rot and sector rot. I should be allowed to choose sufficient levels of redundant storage to recover from arbitrary levels of data corruption.
  3. Open source. Since I'd like to rely on it for my lifetime, I don't want something that may disappear due to commercial whims. Also I want something that runs on Linux. Unlike this question I am willing to maintain and migrate as needed.
  4. Ability to store a lot. At present, I'd like to store a few hundred thousand files taking a few dozen gigs, but I'd prefer a tool that can expand beyond that.
  5. Bonus if it preserves unix timestamps and permissions. Bonus if it's easy to access the stored files without a separate extraction step. And bonus if it's a system that other people also use and understand.

Here are some suboptimal solutions I've considered. Feel free to explain to me why one of these is actually optimal, in case I didn't understand its true capability:

  • Git: Easily verifies the integrity of the repository. But recovering from corruption seems very complicated and not robust. Git also seems to have trouble with large repos.
  • Par2: Provides robust recovery from bit rot and sector rot, but it has a 32,768-file limit.
  • Tar: Archives things, but doesn't seem to provide any recovery from bit rot that I can tell. Also, you have to untar the whole tarball to access the contents.

I suppose I could make a git repository of everything, make an extra tarball of it, and make par2 recovery files for the tarball. But it's also suboptimal to use a jury-rigged solution like this.

share|improve this question

migrated from stackoverflow.com May 31 '11 at 14:51

This question came from our site for professional and enthusiast programmers.

2 Answers 2

IMHO the best archiving method is (now) an second computer with FreeBSD + ZFS. For example 3xHDD inside, 2x1TB with raid-Z ZFS filesystem, and one small for the freebsd and temp files.

While it is not as cheap as CD burning, have some advantages:

  • immediately online all your files - without need search for slow CDs

  • the space will be never issue. (zfs is a 128bit filesystem, and you can address every atom in the universe with it...) - so simple add another hdd and with one command you run towards. (if you have hot swappable hdds you can do this in live, running system)

  • zfs can detect if the files got corrupted (for example bad block on the hdd), every block is checksumed.

  • drive failure? - no problem - replace and go... (hm. this sound as "Wash & Go") :)

  • zilion other very cool features - search google for "ZFS advantages" (especially read about the copy-on-write and snapshots and clones)

  • and at the last - you can forget fsck - will never need it.

If you mount it over a gigabit LAN, your NAS (in some situations) can be faster as your local HDD. ;)

You will must do only one thing - every X years replace HDD what get broken.

IMHO the ZFS is today the only solution when you don't need make external backups. (Because you can roll back deleted files too.)

share|improve this answer
    
Thanks for the idea! It sounds like a good solution, although I wasn't expecting to buy another computer for this, since I already have an external drive. I'll read more about ZFS to see if it's compelling enough to do it. –  krubo Jun 1 '11 at 0:24

No storage is forever, thus my suggestion is to use "common" backup utilities, like rsync, discs or whatever, and refresh them in intervals (lets say once a year).

share|improve this answer
    
Well, yes, but how specifically? For example, if I periodically rsync from A to B, how should I prevent a bit-rotted file in A clobbering a good file in B? –  krubo Jun 1 '11 at 0:15
1  
Create a hashlist (for example md5) for your files. If you verify one backup against this list, you even know which file is broken. –  KingCrunch Jun 1 '11 at 7:34
    
Hey, that really could work. Kind of like what git does under the hood for a single commit, but without using git. Let me think through this. Okay, I'll make 2-3 identical copies of my repository using rsync, and use md5sum $(find -type f) > ../hashlist.dat to make a master hashlist of the repository, then every year I'll use md5sum -c ../hashlist.dat to verify the integrity of each copy. As long as I don't lose all copies at once, I can keep making more copies of whichever existing copy has verified integrity. Let me try it and report back how it goes. –  krubo Jun 2 '11 at 1:58

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.