Why is it impossible to compress on the fly images by ddrescue?

Question

I'm trying to recover files on a dying hard drive, using GNU's ddrescue. The drive is rather large (1TB), and all I have to store the image is another 1TB drive.

ddrescue fails near the end, with an error about disk space, even with the -S option.

Why isn't it possible to compress the image as it is created ? It is possible with dd and dd_rescue.

Also, man ddrescue says that the -S option doesn't work on all systems, but how can I know if it works on mine ?

Answer 1

gddrescue dosen't image the file in order - it goes back and retries, and fills in the blanks, and I suspect this is why you can't effectively pipe it into something else. The man page goes into that in detail

GNU ddrescue manages efficiently the status of the rescue in progress and tries to rescue the good parts first, scheduling reads inside bad (or slow) areas for later. This maximizes the amount of data that can be finally recovered from a failing drive.

The standard dd utility can be used to save data from a failing drive, but it reads the data sequentially, which may wear out the drive without rescuing anything if the errors are at the beginning of the drive.

Other programs switch to small size reads when they find errors, but they still read the data sequentially. This is a bad idea because it means spending more time at error areas, damaging the surface, the heads and the drive mechanics, instead of getting out of them as fast as possible. This behavior reduces the chances of rescuing the remaining good data.

Its by design, and the solution, unfortunately, is to get a bigger drive. For the -S argument to work, I believe the used space on the source drive must be smaller than the destination drive.

Answer 2

It is possible to compress images created with ddrescue on-the-fly. Thing is, ddrescue requires a seekable destination, since it will go over in several passes (and therefore must have the ability to jump backwards to fill in earlier gaps, as @JourneymanGeek explains in his answer). This means you cannot use a pipe as the output, since a pipe is not seekable. Therefore you cannot pipe to compression programs.

One way to get around this is to use transparent compression. Some filesystems (notably, Btrfs, amongst others) provide this built-in. Alternatively, you can use filesystem drivers to provide transparent, seekable compressed storage, e.g. fusecompress.

Answer 3

There's a good post over at brashear.me which creates a btrfs partition stored as a file. The partition supports compression and can be used by ddrescue transparently. The blog post is reproduced below, should it ever become unavailable.

As disk sizes explode, I've found myself having to mirror disks which I don't have enough storage for. My tool of choice is ddrescue. However, it doesn't support compression because it needs to be able to seek through the output as it rescues data. A solution I've found is to create a sparse file, format it btrfs, and mount it with the compression option. This allows ddrescue to operate normally, while giving me fast + decent compression.

Create a sparse file which is larger than your source disk:

dd if=/dev/zero bs=1 count=0 seek=4T of=image-repository.img

Mount the file as a loop device:

sudo losetup /dev/loop0 image-repository.img

Partition the loop device:

sudo gdisk /dev/loop0 

Create new GPT by pressing o followed by y

Create new partition by pressing n and pressing enter a few times to accept defaults.

Write the table, by pressing w then y

Reread partitions from the loop device:

sudo partprobe /dev/loop0

Format the loop0p1 partition:

sudo mkfs.btrfs /dev/loop0p1

Mount the filesystem with compression enabled. Valid compression options are zlib, lzo, zstd:

mkdir /mnt/img-repo
sudo mount -o compress=zstd /dev/loop0p1 /mnt/img-repo

Set the c attr on the mount directory:

sudo chattr +c /mnt/img-repo

Now, when you create a ddrescue image inside of the /mnt/img-repo/ folder, it will be transparently compressed!

At a later date, you can access your ddrescue'd image using:

losetup --find --show image-repository.img
partprobe /dev/loopX     //where X is the output of the previous command
mkdir /mnt/img-repo
mount -o compress=zstd /dev/loopXp1 /mnt/img-repo
chattr +c /mnt/img-repo
cd /mnt/img-repo

And you can inspect the contents of the ddrescue'd image using:

losetup --read-only --find --show /mnt/img-repo/disk.img
mkdir /mnt/recovered_files/
mount /dev/loopXp1 /mnt/recovered_files/     //where X is the output of the previous command