Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm considering backing up some data to DVDs/BDs. (I know about DVDisaster.) I've read somewhere that the outer sectors on a DVD are more likely to wear down than the inner ones. This suggests that over time, the blocks in the second half of an image are more likely to be corrupted than those in the first half. So my question is, is there a way to write the same data to 2 DVD images, such that

  1. one can mount either DVD, possibly in 2 steps (eg mount an iso file stored on the DVD), but without copying everything to HD first and manipulating it
  2. the main data should be written to the 2 DVDs in significantly different order

If I'm not being clear, here is what a possible solution could be.

Suppose for a second there existed a linux driver that could mount an iso image backwards. So, I give it an iso file, and when it wants the 1st sector, it reads the last 2048 bytes of the file backwards, instead of the first 2048 bytes in normal order. I don't know if such a driver exists, but if it did, it would be one solution to my problem because I could do this: put my data in an iso image; compute a second image as the first one backwards; encapsulate each image into a simple UDF filesystem with only one file; write the two UDF filesystems to different DVDs. Now, when I plug in either DVD, I could just mount the single iso file to get back my data. Furthermore, the data would be written to the 2 DVDs in different ways, so if both DVDs each lose the last 1/3 of their sectors, I could still retrieve it all (by hand, but it would be possible).

Other possible solutions would be a general driver to permute/rotate/reorder arbitrary-size blocks in an arbitrary block device. Or perhaps, is there a way to store a file in a UDF file system using a specific order of sectors? Given that UDF is a full-fledged file system, that is definitely possible, but is there a tool to do it?

Thanks!!

Edit: As I explained under the first reply, I don't mean to replace DVDisaster, but to complement it. Consider 2 strategies for backing up 4G of data. Strategy A: use 2 identical DVDs, each with 15% ecc from DVDisaster. Strategy B: use 2 DVDs, each with 15% ecc, but with the additional permutation thing I mention above (on 1 of the 2 DVDs). I claim that because of the wear patterns of DVDs (specifically, the correlation of errors), after a certain time, the probability of full recovery from B is significantly larger than from A (all other things being equal).

Edit2: To substantiate my claim that DVDisaster isn't a cure for everything, here is a script demonstrating how DVDisaster with 33% ECC data suffers data loss with only 1.3% corruption. The apparent contradiction is that 33% only applies to the "best case" corruption, not to "any" corruption. FYI, I'm creating a file spanning the entire filesystem in test.1.udf, changing only the last sector in the file to zero in test.2.udf, computing ecc data for both, and comparing the sectors, including the ecc data. The point is that if test.1.udf is your data and you lose the sectors that are different and only those, you cannot possibly recover test.1.udf because test.2.udf is another possibility.

n_blocks=8192
tdir=$(mktemp -d)
mkudffs -b 2048 test.1.udf $n_blocks
sudo mount test.1.udf $tdir -o bs=2048
sudo chown $USER.$USER $tdir
n=$(df -B 2K $tdir | tail -n 1 | awk '{print $4}')
let n-=1
dd if=/dev/urandom of=$tdir/file bs=2K count=$n 2>/dev/null
last=$(od <$tdir/file -Ad -t x1 | tail -n 2 | head -n 1 | cut -d ' ' -f 2-)
sudo umount $tdir
start_of_last_block=$(od <test.1.udf -Ad -t x1 | grep -A 1 "$last" | tail -n 1 | awk '{print $1}')
last_block=$(($start_of_last_block / 2048))
dd if=test.1.udf bs=2K count=$(($last_block - 1)) >test.2.udf 2>/dev/null
dd if=/dev/zero bs=2K count=1 >>test.2.udf 2>/dev/null
dd if=test.1.udf bs=2K skip=$last_block count=$(($n_blocks - $last_block)) >>test.2.udf 2>/dev/null
n_blocks_with_ecc=$(echo "$n_blocks * 133 / 100" | bc)
echo "add dvdisaster ecc data, using in total $n_blocks_with_ecc"
#run dvdisaster on the 2 files, then...
n_blocks_different=$(for i in $(seq 0 $(($n_blocks_with_ecc - 1))); do
  if [ $((($i / 100) * 100)) -eq $i ]; then
    echo "$i..." >&2
  fi
  diff -q <(dd if=test.1.udf bs=2K skip=$i count=1 2>/dev/null) \
      <(dd if=test.2.udf bs=2K skip=$i count=1 2>/dev/null) >/dev/null || echo $i
done | wc -l)
echo "number of blocks different: $n_blocks_different / $n_blocks_with_ecc ($(echo "scale=6; $n_blocks_different / $n_blocks_with_ecc * 100" | bc)%)"

Output:

number of blocks different: 145 / 10895 (1.330800%)
share|improve this question
    
Another possible solution would be to store the payload data on a linear virtual md-array split across a few files (e.g., 4). Then I could write those files in different orders on the 2 DVDs. The problem with this is that I need a loopback device for each chunk in order to mount the array, and those are limited. Also, I could only mount this if my system runs md (as opposed to an artificially fragmented file on a UDF, which every system should understand). –  Matei David Feb 23 '13 at 21:57

1 Answer 1

The problem you are describing already has a more elegant and efficient solution: Reed-Solomon Error correction. This works by inserting error correcting code at the end of the disk, such that you can lose a certain amount of the data from arbitrary locations on the single disk and still recover the whole file.

This is possible because the RS decoder does not differentiate between user data and error correction data. In the view of the RS decoder each block is a sequence of 100 bytes from which an arbitrary subset of 20 bytes can be recovered.

While some parts of the disk may be more likely to lose data, data loss can still occur in any location. In the two disk method described, you will lose data if any two failed sectors overlap. At the higher levels of data loss, this would be fairly common. In comparison, Reed-Solomon Error correction allows you to recover from losing 14.3% (normal mode) or 33.5% (high mode) of the disk without issues.

DVDisaster is designed to do just this and is perfectly capable of meeting your needs here. If you are felling particularly paranoid, you can set the redundancy to High (or custom) and still use less space and have higher reliability then you would with two disks.

Happy Back Ups!

enter image description here

share|improve this answer
    
Also mention par2 ;) –  0xC0000022L Feb 22 '13 at 21:10
    
I thought about it, but I wasn't sure how it well would work if you have files that went across multiple CD's. Also, I wanted to keep the length down. –  David Feb 22 '13 at 21:14
    
Thanks for the reply, but with my scheme I didn't mean to replace DVDisaster, only to complement it. –  Matei David Feb 23 '13 at 21:19
    
Some issues with what you said: 1. DVDisaster doesn't insert parity at "random" spots, but only at the end- I tested it. If you diff -qb <(dd if=orig.iso bs=2K count=$x) <(dd if=parity.iso bs=2K count=$x) with x equal to number of sectors in orig.iso, the will be equal. –  Matei David Feb 23 '13 at 21:27
    
2. I am 90% sure that the redundancy guarantee you mention (eg, 15%) means: "under the best circumstances, if 14.99% errors occur, you can recover them all; also, if 15.01% errors occur, you can certainly not recover them all." That's different than what you seem to suggest: "for every possible 14.99% errors, you can recover them all." Maybe par2 does that, but I bet (like I said, it's an informed guess) that to be efficient, DVDisaster splits the input in clusters and protects clusters independently of each other. To lose data, you only need to lose 15.01% from 1 cluster & its parity. –  Matei David Feb 23 '13 at 21:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.