Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Because most storage media degenerate over time (CD, DVD, harddisk, tape), it is very important to test your backups. Is there a way to monitor the condition of your backups?

I am looking for both hardware and software solutions, as well as strategies you can use. For example:

  • Hardware solution: using the S.M.A.R.T. features of harddisks; ...
  • Software solution: using a backup tool that calculates MD5 checksums; using a tool for checking the Reed-Solomon ECC on CDs; ...
  • Sample strategies: keeping multiple copies of your backup and doing a file comparison every month; scanning every now and then the MD5 checksums calculated by your backup tool; ...
share|improve this question

13 Answers 13

The only way you can be sure of your backups is to restore them. Checksums may verify the content of your medium but they won't tell you if a restore is possible: what if you're not backing up everything you need to?

The complexity of this depends on whether you're backing up for bare-metal recovery (which has its own issues) or just carrying out data backups.

For data backups, one option is to build a virtual machine and periodically fire it up and test a restore. This is less than valid for bare-metal because the VM will undoubtedly require a completely different set of e.g. drivers, etc. I guess (depending on your OS and tool-set), if you were really keen, you could probably script the process of carrying out the restore and do it for every backup.

share|improve this answer
    
I think you're missing the context of the original question. Verifying the checksums does confirm that the data on the medium hasn't been lost as it involves reading it all. What you're describing tests your backup strategy which is crucial, but for medium tests checksums are just fine. –  Draemon Nov 12 '10 at 23:20
    
Fair enough, but I stand by my statement. The only way you can be sure you can restore is to restore. –  serialhobbyist Dec 10 '10 at 10:28

You should be verifying your backups before storage to begin with. A lot of attention is brought to how hard it is to do backups, but no one ever thinks to check them before safely bringing them off site.

I generally only wait for 1 bad write to a CD-R to toss it, for how cheap they are, and focus on S.M.A.R.T. results for HDD media to tell me when to replace them.

Best of luck to you.

share|improve this answer
3  
SMART results may not tell you as much as you'd like. Google labs did some testing - labs.google.com/papers/disk_failures.pdf and found that they got lots of failures with no SMART warning at all. –  Michael Kohne Sep 1 '09 at 13:28
    
Good paper. Thanks for the input –  bobby Sep 1 '09 at 19:23
    
However, SMART warning means it's time to act quickly! –  harrymc Sep 15 '09 at 11:25

With today's large amounts of data and relatively small optical disc sizes, backup to removable HDD's seems best to me. Also, HDD's are easier to maintain both physically and in terms of filling data onto them.

I regularly run GRC's SpinRite on my disks. SpinRite tests the written data, and optionally refreshes or even recovers it.

share|improve this answer
1  
That tool won't help at all to detect silent data corruption. Checksums are the way to go, not some tool that claims your drive is OK. –  user23307 Jan 12 '10 at 12:52
1  
That tool both bypasses S.M.A.R.T. and scrubs down your disk moving data on damaged sectors away from them and repairing as much data as it can. I've heard story after story about how it recovered lost photos, critical systems, etc. from hard drive failure. It works good as a last chance tool but also great as maintenance. (Sorry to sound like a crusader i just don't like to hear a tool i know works well maligned) +1 for spinrite –  RCIX Jan 12 '10 at 13:41
2  
It's great that you like the tool, but it will not detect or fix data corruption. –  user23307 Jan 14 '10 at 12:25
    
It's also well known that the author makes false claims about the software. –  Draemon Nov 12 '10 at 23:21

testing your backups is an interesting point. however, it is time consuming and questionable in my opinion. because if a medium is damaged, your backup may be a total and partial loss, thus testing is rather moot.

i prefer multiple backups. as for optical media (not my preferred choice), if you create a new copy every, say, 5 years you should be on the safe side, if you have two copies (kept in different locations to reduce the risk of data loss in case the house burns down :), the chance that at least one has survived the aging process is very high. of course the risk is decreasing reciprocally to the number of backup sets.

share|improve this answer
    
Testing can show failures in your backup processes, though - it's worth at least pretending to go through with it. –  Phoshi Jan 12 '10 at 12:31

Well, you could use par or QuickPar for Windows or run an MD5/SHA1 checksum on the iso.

share|improve this answer
    
The link to par2 is dead. –  Zian Choy Nov 11 '09 at 22:49
    
I removed the par2 link. –  hanleyp Nov 13 '09 at 15:43

Sounds like you need a checksum for your backup.

share|improve this answer

All of the answers have been good, but I would like to add one more suggestion. Replace your media on a regular schedule. We replace the back up media every six months. It is fairly inexpensive, especially compared to needing a backup and find that it is bad.

share|improve this answer

With WinRAR you can specify the use of a recovery volume which will allow an entire CD/DVD from a recovery set to be destroyed and still be able to recover from the remaining disks. For example, I use WinRAR to backup a set of files/directories by compressing them into a RAR archive, using the span options to set it to span to the size of a DVD(or whatever media you are spanning, the size of a tape backup, etc.) and also set a Recover Volume of 1 or more (found under Advanced tab in the Volumes section, and requires that you have set the Split To Volumes on the General tab, this is different from a Recovery Record so don't confuse them).

With a Recovery Volume an extra Rar file will be created in the set. Some sort of parity is used such that any one of the rar files in a set can be lost and the remaining files can still be used to recover all of the data. You could use Recovery Record as well to allow for a certain amount of damage in all of the rar files.

You can then test the integrity of the data periodically by performing a test extraction. I don't know if you'll get any feedback about the damage of the files. You could easily test this by purposely damaging a file in a set. With a Recovery Volume the data should still be extracted, but hopefully WinRAR will display some feedback in it's console about the damaged archive.

share|improve this answer

CFV seems to do the job (http://cfv.sourceforge.net/cfv.1.html).

Personally, I use the following 2 commands:

Creating:

cfv -rr -C -tsha1 -fc:\users\zian\desktop\out.txt

Verifying:

cfv -rr -tsha1 -fc:\users\zian\desktop\out.txt
share|improve this answer

The method I've used in the past is to take a backup server and periodically try to image it from my backup medium.

If it works - great. If not, I go get new media.

clarification for justin re:comments
My imaging process always includes starting the server from the restoration and verifying applications start as expected - otherwise it hasn't been tested, it's merely been written-to.

share|improve this answer
    
How does this detect silent data corruption? –  user23307 Jan 12 '10 at 12:53
    
well, if you can't recover from your backup (ie, data has become corrupted), then it's no good. –  warren Jan 12 '10 at 13:29
    
I don't think you know the meaning of "silent data corruption" The fact that you can recover from the backup does not mean the data has not been corrupted. –  user23307 Jan 12 '10 at 22:56
    
in my understanding, "silent data corruption" means the restore can occur, but data may be nuked. However, applications won't run with corrupted data, so therefore the restore wasn't successful (ie, when you startup, stuff doesn't work) –  warren Jan 13 '10 at 14:29
    
Applications WILL run with corrupted data, thus the phrase "silent data corruption" –  user23307 Jan 14 '10 at 12:23

I found a related question here on SuperUser: Which file system automatically stores hashes of files?.

share|improve this answer

You could use ExactFile to generate a file containing the MD5 checksums of a fresh backup. Before the next backup, you can then compare the MD5 file against the current situation. If there are differences, this means that the backup has gone corrupt.

share|improve this answer

Making sure that anything I want to keep exists not only in multiple places, but on multiple mediums. Medium degeneration shouldn't just include the physical media themselves, but also the reading of them. I have some old text files backed up on 3 (THREE!) floppy disks, but if I needed them, well, I don't have a floppy drive anymore. How long until the CD drive is deprecated - we're heading that way with netbooks, and 'cloud computing'! If I'd stored my files on a variety of media, I'd be less likely to be in this situation - far more likely I'd still be able to read them!

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.