Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I like to archive everything. Currently a small box of CDs holds most of my work and downloads from the past 15 years. Unfortunately, some of the CDs are starting to suffer bit rot, resulting in corrupted and unreadable files.

So I'm looking for the best system for this. I expect to put all the files on my hard drive, and rsync a copy onto an external drive, using <something> to manage it as a repository. Here are my top requirements:

  1. Simple recovery from bit rot and sector rot. It should be easy to verify the integrity of the entire repository, or identify the extent of data corruption. As long as data corruption is below a certain level, it should be easy to make a pristine clone out of a corrupted repository.
  2. Robust recovery from bit rot and sector rot. I should be allowed to choose sufficient levels of redundant storage to recover from arbitrary levels of data corruption.
  3. Open source. Since I'd like to rely on it for my lifetime, I don't want something that may disappear due to commercial whims. Also I want something that runs on Linux. Unlike this question I am willing to maintain and migrate as needed.
  4. Ability to store a lot. At present, I'd like to store a few hundred thousand files taking a few dozen gigs, but I'd prefer a tool that can expand beyond that.
  5. Bonus if it preserves unix timestamps and permissions. Bonus if it's easy to access the stored files without a separate extraction step. And bonus if it's a system that other people also use and understand.

Here are some suboptimal solutions I've considered. Feel free to explain to me why one of these is actually optimal, in case I didn't understand its true capability:

  • Git: Easily verifies the integrity of the repository. But recovering from corruption seems very complicated and not robust. Git also seems to have trouble with large repos.
  • Par2: Provides robust recovery from bit rot and sector rot, but it has a 32,768-file limit.
  • Tar: Archives things, but doesn't seem to provide any recovery from bit rot that I can tell. Also, you have to untar the whole tarball to access the contents.

I suppose I could make a git repository of everything, make an extra tarball of it, and make par2 recovery files for the tarball. But it's also suboptimal to use a jury-rigged solution like this.

share|improve this question

migrated from stackoverflow.com May 31 '11 at 14:51

This question came from our site for professional and enthusiast programmers.

IMHO the best archiving method is (now) an second computer with FreeBSD + ZFS. For example 3xHDD inside, 2x1TB with raid-Z ZFS filesystem, and one small for the freebsd and temp files.

While it is not as cheap as CD burning, have some advantages:

  • immediately online all your files - without need search for slow CDs

  • the space will be never issue. (zfs is a 128bit filesystem, and you can address every atom in the universe with it...) - so simple add another hdd and with one command you run towards. (if you have hot swappable hdds you can do this in live, running system)

  • zfs can detect if the files got corrupted (for example bad block on the hdd), every block is checksumed.

  • drive failure? - no problem - replace and go... (hm. this sound as "Wash & Go") :)

  • zilion other very cool features - search google for "ZFS advantages" (especially read about the copy-on-write and snapshots and clones)

  • and at the last - you can forget fsck - will never need it.

If you mount it over a gigabit LAN, your NAS (in some situations) can be faster as your local HDD. ;)

You will must do only one thing - every X years replace HDD what get broken.

IMHO the ZFS is today the only solution when you don't need make external backups. (Because you can roll back deleted files too.)

share|improve this answer
    
Thanks for the idea! It sounds like a good solution, although I wasn't expecting to buy another computer for this, since I already have an external drive. I'll read more about ZFS to see if it's compelling enough to do it. – krubo Jun 1 '11 at 0:24

No storage is forever, thus my suggestion is to use "common" backup utilities, like rsync, discs or whatever, and refresh them in intervals (lets say once a year).

share|improve this answer
    
Well, yes, but how specifically? For example, if I periodically rsync from A to B, how should I prevent a bit-rotted file in A clobbering a good file in B? – krubo Jun 1 '11 at 0:15
1  
Create a hashlist (for example md5) for your files. If you verify one backup against this list, you even know which file is broken. – KingCrunch Jun 1 '11 at 7:34
    
Hey, that really could work. Kind of like what git does under the hood for a single commit, but without using git. Let me think through this. Okay, I'll make 2-3 identical copies of my repository using rsync, and use md5sum $(find -type f) > ../hashlist.dat to make a master hashlist of the repository, then every year I'll use md5sum -c ../hashlist.dat to verify the integrity of each copy. As long as I don't lose all copies at once, I can keep making more copies of whichever existing copy has verified integrity. Let me try it and report back how it goes. – krubo Jun 2 '11 at 1:58

Well, there are a few solutions (duh).

Depending on the quantity of the data, if you are serious about storing large amounts of data, a solution such as tapes and tape drives could be worth the investment. Bear in mind that the tape drives alone are at least a thousand dollars in most cases, so this is only if you are serious about keeping your work. It seems you only want to store a few gigabytes, so it's probably not the route you should go.

Another money-solution is simply buying archival-grade disks. They'll obviously expire, but not nearly as fast as standard CD/DVD-Rs.

Depending on how sensitive this data is, you could also upload it to a cloud-storage service, such as DropBox or even Google Drive. If you're looking a more DIY way to do this, you could run something such as OwnCloud and have that on a VPS somewhere. Granted, this requires you to depend on other people for longer-term data storage, which you stated you weren't too keen on. You could also just try running off of your own hardware.

Another solution is just to constantly "reburn" the data. This is literally as simple as moving the data about. For instance, from one drive to another then back to the original. You could easily set up a job to do this for you. If you want to get fancy, you could constantly monitor the hashes of these files (or just compare them to another location they might exist).

Although, as someone above me mentioned, no data is eternal (as you've obviously seen). The person who mentioned using ZFS has quite an idea as well, so it might be worth looking into that. There's also more ways to do this, but this is all I came up with off the top of my head that wasn't just "make backups!"

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .