1

We periodically write two copies of a DVD image. The images typically contain on the order of 5000 files, with relatively few files in each folder. We write two because these are backups of ongoing work, and one copy is carried to an offsite location for safety. There are presently about 500 of these discs in the offsite box.

We recently had occasion to restore a number of files from these backups, and found that many files could not be read. At least not on the first try. And in many cases they could be read from a different DVD drive. Needless to say, this has caused some concern about the health of these backups.

So we added a checksum step to our image generation, where we use sha1sum to collect checksums of every file on the disc. With a freshly burned disc in hand that includes the sums, we then proceeded to attempt to validate the process using sha1sum --check checksum.txt. Now we are even more concerned.

A typical result is that about the first 1/3 to 2/3 of the files check OK. Then sha1sum starts reporting "permission denied", and continues to do so for nearly every remaining file in the list.

Oddly, if the list is shortened to just the failed files, sha1sum proceeds to read most or all of those OK.

This is consistent with results of using robocopy to copy all of the files off a disc. Robocopy gets most of the files on the first pass, then gets the rest on subsequent passes.

Our fundamental question is do we have a hardware problem with the DVD writer, our DVD readers, or our PCs? Or is this a media problem with the quality of the blank media (the current stack says Memorex)? And how should we tell?

  • No, DVDs are not a reliable backup medium. They are way to sensitive to external influences. – Daniel B Jul 23 '14 at 18:56
  • How old are the DVD+R disks? Are they stored in a heated environment (i.e. regular room temperature), or are they exposed to temperature swings? Have you considered using something like PAR2 to add recovery data? – ChrisInEdmonton Jul 23 '14 at 18:56
  • 2
    The only really reliable backup medium is PaperBack. – Daniel R Hicks Jul 23 '14 at 19:58
  • 2
    (This of course reiterates the fact that there is a sore need for a reliable archive medium for digital data.) – Daniel R Hicks Jul 23 '14 at 21:04
  • 1
    At least DVD is optical, so it is still playing in the same sandbox as paper, just at a much higher resolution. And there are the M-Disc folks, claiming media that is etched and having a 1000 year lifetime. (Take that claim with a saltshaker, but note that normal media is in the sub 10-year lifetime ballpark.) But my problem right now isn't longevity. It is getting files back at all. – RBerteig Jul 23 '14 at 21:11
1

I would guess you are suffering from old DVDs, or the particular box of discs that you bought has a defect (someone left them in the sun for too long before they were put on the shelf).

On the other hand, if your DVD burner is old, it's likely that the laser is toast. Lasers have a hard life and don't last very long. A bad laser could affect the readers and writer, presuming they were bought at similar times. You replace the writer (or reader) and try again.

This ends my official answer. For unsolicited, but highly practical advice, keep reading.


Avoid the disposable media du jour

If your data is important, don't trust a 50¢ DVD to store it indefinitely. Additionally, don't trust SD cards, usb flash disks, zip disks, LS120s, or floppy disks, and all for the same reasons:

  • Small things get lost, stolen, run through the wash, and stepped on
  • Stuff that's archived offsite has to be moved; moving computer media has a fairly significant impact on reliability (see above)
  • Manufacturing tolerances are lower for cheap, extremely-mass-produced things
  • The reliability of media is always hugely overestimated (DVDs last for 100-200 years!)
  • Most of todays computing devices don't have optical drives. In 5 years, you might not have a flash card reader, a USB port, or even a regular computer.

For similar reasons, don't archive data (remove it from a computer and keep it only on removable media) either. If it's important enough to keep, or if you're not sure how important it is, then keep it on a computer. Otherwise, delete it.

Reliable Backups

The only reliable backups that I've ever known were those kept on real hard disks. You don't have a lot of data; 500 DVDs should fit onto a cheap hard disk with room to spare (and will weigh a great deal less).

  • Buy the cheapest external drive you can find. 2TB USB drives are about $70. In a few years, bigger (4TB+) external drives will likely be about $70.
  • Copy your important data on there, and get another one to do a computer backup (now you have three copies of the data).
  • Replace these drives every couple years, depending on your budget.
  • This goes without saying, but do not move or drop your backup disks! Leave them on your desk or (for network-attached disks) in your server closet. Moving them risks adding vibration, breaking cables, dropping, etc.

Hard disks are also way easier to use and way faster to write than DVDs (and other media).

(The one exception are tape backups, which can be thousands of times more reliable than hard disks, but not really practical unless you're backing up a bunch of servers).

Offsite Backups

Find an online provider (Backblaze etc.).

Don't consider an internet backup provider to be a permanent storage medium either. Businesses will fail, get hacked, and have IT accidents. Consider it to be an ultra emergency backup, and price that accordingly. Keep in mind that a $70 external drive replaced every two years costs less than $3 a month. Most online backup services are a lot more than that.

Rather than an online provider, get yourself some personal file synchronization software (something like Dropbox or AeroFS). Then set up a computer at your house that's connected to the internet, and use it to be your "online" backup provider. It's cheaper, but has a bit more overhead. It also has the benefit of being able to grab the drive and bring it into the office if there's an emergency (vs. trying to figure out how the online backup's "restore" option works).

Vs. Paper Backups

Paper is not fire, flood (mold), theft, or earthquake-proof. And, it's a bit speedy.

At 6 KB per page square inch, backing up a single (6GB) DVD would take about 6000 pages. Printing that will cost you about $60 in supplies (not including the cost of a high end laser printer or the upfront cost of toner), and take several hours, using about 2 kWh of electricity (releasing about a kg of CO2).

(My old answer was far more entertaining, but I misread per square inch as per page, sorry folks).

  • Uh, look again -- PaperBack will do 500KB per page, uncompressed. – Daniel R Hicks Aug 29 '14 at 2:53
  • And tape is not as reliable as you make it out to be. The oxide falls off the tape after 10-20 years. – Daniel R Hicks Aug 29 '14 at 2:55
  • @DanielRHicks - You're absolutely right, the "square inch" part of the paperback site wrapped funny, and I saw 6kb per ... paper. – Seth Aug 29 '14 at 4:12
  • 2
    And, incidentally, I still have at least one 20-year-old drive - it uses an early obscure interface (MFM maybe?). Finding working hardware to read it, let alone transfer it to a modern medium would be a challenge (if the drive still works, which I doubt). The main point is that you have to keep migrating data to modern technologies. Well planned tape systems should last at least 10 years, 20 would be a stretch, but that's still better than a hard disk (and they have a much better MTBF). We'll see how well SSDs do (just don't write to them too much). – Seth Aug 29 '14 at 4:24
  • Which is one major advantage of PaperBack (seriously!). So long as technology doesn't slide backwards there will always be scanners, and, given a written description of the algorithm (which can be printed out on one page in clear text), it will always be possible to recover the data. – Daniel R Hicks Aug 29 '14 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.