0

I need to rescue data from some 2 TB large hard-drive and am doing so in some Live-Linux in some VM, where the problematic hard-drive is connected to using USB 3 and the VM provides a virtual disk of the needed size locally to receive the data. I then executed the following call, simply to see how things are going:

ddrescue -f /dev/sdc /dev/sdb /mnt/sda1/ddrescue.map

sdc is the broken device at USB, sdb is the virtual disk to receive the data, sda1 is for temporary storage and formatted using Ext4.

Things started to work quickly, ddrescue was able to read ~45 GB of data within minutes, then things slowed down massively, reading only at some Bytes per second for days. So the device was obviously broken at these parts and I tried to simply skip those using multiple invocations of different --input-position=[...]GB one after another. Depending on where I jumped to, things started to read fast again, until they got slow again and I jumped again using another invocation. The important thing to note is that input and output position printed by ddrescue have always been in sync! I didn't manually change anything in the provided map file as well or deleted it or whatever, it has always been one and the same file and only managed by ddrescueitself.

Afterwards I changed the approach a bit and decided to not use --input-position anymore manually, but the following instead:

ddrescue -f --min-read-rate=1MB --skip-size=1MB /dev/sdc /dev/sdb /mnt/sda1/ddrescue.map

So whenever ddrescue recognized slow parts, it skipped reasonable broken blocks of data and continued reading. Again, input and output position was in sync and the counter of read and rescued data increased all the time. Up to the point were ddrescue finished and said to have rescued ~650 GB of data.

The problem is that after finally looking at the virtual disk files themself, it seems that only ~160 GB of data is actually stored. Additionally, the last write timestamp was some days too old. So for some reason ddrescue thought it was reading lots of data, but didn't seem to write it properly at the places in the virtual disk where it read them from the broken disk. In the end, from my understanding the virtual disk should have had at least the size ddrescue said about the amount of data it rescued.

I have the feeling that ddrescue properly read all the data it said, but simply overwrote already rescued data in subsequent invocations. So while I guess it recognized --input-position to read from, it seems to have written always starting at position 0 again at the target.

Obviously I didn't specify the start position to write data to, but according the docs that shouldn't be necessary and ddrescue always printed input and output position to be the same anyway.

-o bytes
--output-position=bytes
Starting position of the image of the rescue domain in outfile, in bytes.
Defaults to '--input-position'. The bytes below bytes aren't touched if 
they exist and truncation is not requested. Else they are set to 0.

Of course I didn't request truncation, according the docs it is not enabled by default and wouldn't even have worked for the target drive I had specified:

-t
--truncate
Truncate outfile to zero size before writing to it. Only works for regular
files, not for drives or partitions.

So, any idea on what might have gone wrong? Were my multiple invocations with different values for --input-position wrong already? Does it have to do with reading and writing to drives instead of partitions or files?

Maybe a problem writing to some virtual disk? Though I don't see why that should make any difference and I need to write to some virtual disk and can't provide raw device storage of the needed size.

Thanks!

0

Is it safe to use multiple different --input-position with ddrescue?

Seems like I have missed that example before, but that is actually what I have done and it suggests that my approach is supported:

Example 5: While rescuing a partition in /dev/sda1 to the file hdimage, /dev/sda1 stops responding and begins returning read errors, causing ddrescue to mark the rest of the partition as non-scraped.
     ddrescue -n /dev/sda1 hdimage mapfile        <-- /dev/sda1 fails here
       (restart /dev/sda or reboot computer)
     ddrescue -n -A -i<pos> -O /dev/sda1 hdimage mapfile
       (if /dev/sda1 fails again, restart /dev/sda or reboot computer and
        then repeat the above command as many times as needed until it
        succeeds. <pos> is the position where the drive stopped responding)
     ddrescue -d -r3 /dev/sda1 hdimage mapfile

https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Examples

The second invocation clearly is documented to be repeated with different positions. Regarding how ddrescue works using its map file, this makes sense as well, simply because it always knows using that file which blocks have already been read.

So it seems chances are high that the problem in my case is different, especially the too old timestamp I think I recognized is strange. Maybe I have simply missed messages that ddrescue is not writing to the real target device for some reason. The VM itself was on another USB-drive as well, maybe there were some connection errors leading to the device being missed by the Live-Linux during runtime or such. I could have easily missed such errors in dmesg -T because of all the read errors logged.

Sounds like I need to repeat the whole process...

  • I've tested again with 3 invocations using --input-position=XGB starting at 0, 2 and 5 GB of data, reading around 1 GG and all of those increased the memory of the virtual target disk I was writing to. So I'm pretty much sure that using multiple --input-position is safe and I did something else wrong. – Thorsten Schöning Oct 14 '18 at 14:22
  • The problem seems to have happened again: After ~23 hours of work, ~485 GiB of data have been rescued, ddrescue keeps reading and rescuing data, but none is written to the virtual disk anymore. No errors in dmesg -T about the virtual target disk, non in the host, the writes simply disappear somewhere. And because ddrescue thinks the data has been rescued, one doesn't know what is missing and where to read again or such. – Thorsten Schöning Oct 15 '18 at 13:41
0

I read the ddrescue manual, and nowhere does it mention the possibility of multiple input-position parameters.

This parameter is always mentioned as "a" or "the", so it seems that it must be unique.

The source of your problem may be this phrase from the manual :

Note that you must keep the original offset between '--input-position' and '--output-position' of the original rescue run.

This seems to agree with the following other paragraph :

Ddrescue does not write zeros to the output when it finds bad sectors in the input, and does not truncate the output file if not asked to. So, every time you run it on the same output file, it tries to fill in the gaps without wiping out the data already rescued.

This means that ddrescue remembers the parameters from the first run, so you are always supposed to keep the same parameters, or maybe just not to specify them on subsequent runs (I can't say which is right). It's entirely possible that some parameters were remembered and your new ones were ignored on following runs.

If parts of the meta tables of the disk were damaged, you might be seeing less data than was actually salvaged, because no file seems to include these parts.

The data that ddrescue cannot salvage needs to be recovered by other recovery products. This might take a long time and might even be impossible for the products at your disposal. If the data absolutely has to be recovered, a professional recovery company might be able to do it from the original disk, but these services are costly.

  • Seems I was unclear: I did not specify the argument multiple times per invocation, but always only once per invocation with different values. At first with e.g. 1GB, than with 2 or 500 and so on. And ddrescue always started to read where I said it should. From my understanding, as I specified raw devices, ddrescue should care or know of files as well and instead simply read and write arbitrary "stuff". – Thorsten Schöning Oct 11 '18 at 7:45
  • By the documentation you were not supposed to specify it multiple times, but always use the same value or none. The mode of operation of ddrescue supposes multiple identical runs. As I said, I don't know if multiple specifications are taken into account or not. And it doesn't know anything about files, just about disk sectors, same as dd. – harrymc Oct 11 '18 at 7:57
  • It does know about files: "infile and outfile may be files, devices or partitions." Example 5 clearly mentions different positions as well: "<pos> is the position where the drive stopped responding" gnu.org/software/ddrescue/manual/ddrescue_manual.html#Examples – Thorsten Schöning Oct 11 '18 at 12:46
  • Yes, but you are not using that option. – harrymc Oct 11 '18 at 15:00
  • "-i bytes --input-position=bytes" gnu.org/software/ddrescue/manual/… – Thorsten Schöning Oct 12 '18 at 6:40
-1

As the man page of ddrescue is long, the use of ddrescue is very different according the goal and user level. Basically, if you use Live Linux, you had better to run it on the phycical machine instead of VM, and also connect the disk to sATA without any sATA/USB adapter.
Among the other features ddrescue can bypass the kernel disk driver and buffers, hence it can reduce the useless repeated read of bad clusters. The mapfile (formerly called logfile) keeps information about all un/success read clusters and that's why you can simply repeat the crashed step. The ddrescue looks for mapfile before it starts its job, create it, if it does not exist, read it, if it is available and start to continue the rescue job at the last recorded position. You need not move the start position by hand every time the program crashes!

You can use various option to make the rescue process quicker and more safer. You also can, and it is recommended, you can make the rescue process in two or more steps:

First step: quickly read the good clusters and immediately skip the bad one.

Second step: deal with unread clusters from previous step and use a special options to trick out the disk features (NCQ, read ahead ...) in ought to read one sector at once. The adequate commands (I use) :

ddrescue -n -p -d -r1    /dev/sdd $IMGPATH/disk.img $IMGPATH/disk.log;
ddrescue       -d -r3 -R /dev/sdd $IMGPATH/disk.img $IMGPATH/disk.log;
#         |  |  |  |   |
#         |  |  |  |   revers reading
#         |  |  |  retry read 1x (3x)
#         |  |  direct access to disk (bypass the kernel)
#         |  preallocate diskspace      
#         nonscrap

If your disk heat too much or do not like to many read ops/s you can slow down the reading with the option: --max-read-rate=50M

So that is for the first touch only, but you can find many advices at specialised clubs or forums concerning for the ddrescue.

  • 1
    -1. OP raised few questions (including the one in the title). In my opinion you didn't give a direct answer to any of them. The discrepancy between allegedly rescued ~650 GB of data and a lot smaller virtual disk is still a mystery; we don't know what went wrong nor if --input-position is safe. Your answer may be technically right but it doesn't answer the question. – Kamil Maciorowski Oct 10 '18 at 12:22
  • Sorry, Kamil, I tried to show how to use the ddrescue in the better way to get results and not artefacts. You can post your answer to discover the mystery. Thanks for your help. – schweik Oct 10 '18 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.