Is it safe to use multiple different `--input-position` with ddrescue?

Question

I need to rescue data from some 2 TB large hard-drive and am doing so in some Live-Linux in some VM, where the problematic hard-drive is connected to using USB 3 and the VM provides a virtual disk of the needed size locally to receive the data. I then executed the following call, simply to see how things are going:

ddrescue -f /dev/sdc /dev/sdb /mnt/sda1/ddrescue.map

sdc is the broken device at USB, sdb is the virtual disk to receive the data, sda1 is for temporary storage and formatted using Ext4.

Things started to work quickly, ddrescue was able to read ~45 GB of data within minutes, then things slowed down massively, reading only at some Bytes per second for days. So the device was obviously broken at these parts and I tried to simply skip those using multiple invocations of different --input-position=[...]GB one after another. Depending on where I jumped to, things started to read fast again, until they got slow again and I jumped again using another invocation. The important thing to note is that input and output position printed by ddrescue have always been in sync! I didn't manually change anything in the provided map file as well or deleted it or whatever, it has always been one and the same file and only managed by ddrescueitself.

Afterwards I changed the approach a bit and decided to not use --input-position anymore manually, but the following instead:

ddrescue -f --min-read-rate=1MB --skip-size=1MB /dev/sdc /dev/sdb /mnt/sda1/ddrescue.map

So whenever ddrescue recognized slow parts, it skipped reasonable broken blocks of data and continued reading. Again, input and output position was in sync and the counter of read and rescued data increased all the time. Up to the point were ddrescue finished and said to have rescued ~650 GB of data.

The problem is that after finally looking at the virtual disk files themself, it seems that only ~160 GB of data is actually stored. Additionally, the last write timestamp was some days too old. So for some reason ddrescue thought it was reading lots of data, but didn't seem to write it properly at the places in the virtual disk where it read them from the broken disk. In the end, from my understanding the virtual disk should have had at least the size ddrescue said about the amount of data it rescued.

I have the feeling that ddrescue properly read all the data it said, but simply overwrote already rescued data in subsequent invocations. So while I guess it recognized --input-position to read from, it seems to have written always starting at position 0 again at the target.

Obviously I didn't specify the start position to write data to, but according the docs that shouldn't be necessary and ddrescue always printed input and output position to be the same anyway.

-o bytes
--output-position=bytes
Starting position of the image of the rescue domain in outfile, in bytes.
Defaults to '--input-position'. The bytes below bytes aren't touched if 
they exist and truncation is not requested. Else they are set to 0.

Of course I didn't request truncation, according the docs it is not enabled by default and wouldn't even have worked for the target drive I had specified:

-t
--truncate
Truncate outfile to zero size before writing to it. Only works for regular
files, not for drives or partitions.

So, any idea on what might have gone wrong? Were my multiple invocations with different values for --input-position wrong already? Does it have to do with reading and writing to drives instead of partitions or files?

Maybe a problem writing to some virtual disk? Though I don't see why that should make any difference and I need to write to some virtual disk and can't provide raw device storage of the needed size.

Thanks!

Answer 1

Is it safe to use multiple different --input-position with ddrescue?

Seems like I have missed that example before, but that is actually what I have done and it suggests that my approach is supported:

Example 5: While rescuing a partition in /dev/sda1 to the file hdimage, /dev/sda1 stops responding and begins returning read errors, causing ddrescue to mark the rest of the partition as non-scraped.
     ddrescue -n /dev/sda1 hdimage mapfile        <-- /dev/sda1 fails here
       (restart /dev/sda or reboot computer)
     ddrescue -n -A -i<pos> -O /dev/sda1 hdimage mapfile
       (if /dev/sda1 fails again, restart /dev/sda or reboot computer and
        then repeat the above command as many times as needed until it
        succeeds. <pos> is the position where the drive stopped responding)
     ddrescue -d -r3 /dev/sda1 hdimage mapfile

https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Examples

The second invocation clearly is documented to be repeated with different positions. Regarding how ddrescue works using its map file, this makes sense as well, simply because it always knows using that file which blocks have already been read.

So it seems chances are high that the problem in my case is different, especially the too old timestamp I think I recognized is strange. Maybe I have simply missed messages that ddrescue is not writing to the real target device for some reason. The VM itself was on another USB-drive as well, maybe there were some connection errors leading to the device being missed by the Live-Linux during runtime or such. I could have easily missed such errors in dmesg -T because of all the read errors logged.

Sounds like I need to repeat the whole process...

Answer 2

I read the ddrescue manual, and nowhere does it mention the possibility of multiple input-position parameters.

This parameter is always mentioned as "a" or "the", so it seems that it must be unique.

The source of your problem may be this phrase from the manual :

Note that you must keep the original offset between '--input-position' and '--output-position' of the original rescue run.

This seems to agree with the following other paragraph :

Ddrescue does not write zeros to the output when it finds bad sectors in the input, and does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps without wiping out the data already rescued.

This means that ddrescue remembers the parameters from the first run, so you are always supposed to keep the same parameters, or maybe just not to specify them on subsequent runs (I can't say which is right). It's entirely possible that some parameters were remembered and your new ones were ignored on following runs.

If parts of the meta tables of the disk were damaged, you might be seeing less data than was actually salvaged, because no file seems to include these parts.

The data that ddrescue cannot salvage needs to be recovered by other recovery products. This might take a long time and might even be impossible for the products at your disposal. If the data absolutely has to be recovered, a professional recovery company might be able to do it from the original disk, but these services are costly.

Answer 3

As the man page of ddrescue is long, the use of ddrescue is very different according the goal and user level. Basically, if you use Live Linux, you had better to run it on the phycical machine instead of VM, and also connect the disk to sATA without any sATA/USB adapter.
Among the other features ddrescue can bypass the kernel disk driver and buffers, hence it can reduce the useless repeated read of bad clusters. The mapfile (formerly called logfile) keeps information about all un/success read clusters and that's why you can simply repeat the crashed step. The ddrescue looks for mapfile before it starts its job, create it, if it does not exist, read it, if it is available and start to continue the rescue job at the last recorded position. You need not move the start position by hand every time the program crashes!

You can use various option to make the rescue process quicker and more safer. You also can, and it is recommended, you can make the rescue process in two or more steps:

First step: quickly read the good clusters and immediately skip the bad one.

Second step: deal with unread clusters from previous step and use a special options to trick out the disk features (NCQ, read ahead ...) in ought to read one sector at once. The adequate commands (I use) :

ddrescue -n -p -d -r1    /dev/sdd $IMGPATH/disk.img $IMGPATH/disk.log;
ddrescue       -d -r3 -R /dev/sdd $IMGPATH/disk.img $IMGPATH/disk.log;
#         |  |  |  |   |
#         |  |  |  |   revers reading
#         |  |  |  retry read 1x (3x)
#         |  |  direct access to disk (bypass the kernel)
#         |  preallocate diskspace      
#         nonscrap

If your disk heat too much or do not like to many read ops/s you can slow down the reading with the option: --max-read-rate=50M

So that is for the first touch only, but you can find many advices at specialised clubs or forums concerning for the ddrescue.