10

I'm trying to recover files on a dying hard drive, using GNU's ddrescue. The drive is rather large (1TB), and all I have to store the image is another 1TB drive.

ddrescue fails near the end, with an error about disk space, even with the -S option.

Why isn't it possible to compress the image as it is created ? It is possible with dd and dd_rescue.

Also, man ddrescue says that the -S option doesn't work on all systems, but how can I know if it works on mine ?

3
  • 1
    Is there a reason you can't just ddrescue [...] | gzip -9 > /path/to/destination/file? Aug 12, 2013 at 14:21
  • 1
    I've not tried this, but I wonder if getting a single larger drive was not an option, you might be able to get a second disk, create a larger LVM than your source disk, and store the image there.
    – Journeyman Geek
    Aug 12, 2013 at 14:30
  • 1
    @AaronMiller Yes: ddrescue requires a destination with seek support, due to its mode of operation (cf. JourneymanGeek's answer). Therefore, it cannot output to pipes.
    – Bob
    Aug 12, 2013 at 14:38

3 Answers 3

12

gddrescue dosen't image the file in order - it goes back and retries, and fills in the blanks, and I suspect this is why you can't effectively pipe it into something else. The man page goes into that in detail

GNU ddrescue manages efficiently the status of the rescue in progress and tries to rescue the good parts first, scheduling reads inside bad (or slow) areas for later. This maximizes the amount of data that can be finally recovered from a failing drive.

The standard dd utility can be used to save data from a failing drive, but it reads the data sequentially, which may wear out the drive without rescuing anything if the errors are at the beginning of the drive.

Other programs switch to small size reads when they find errors, but they still read the data sequentially. This is a bad idea because it means spending more time at error areas, damaging the surface, the heads and the drive mechanics, instead of getting out of them as fast as possible. This behavior reduces the chances of rescuing the remaining good data.

Its by design, and the solution, unfortunately, is to get a bigger drive. For the -S argument to work, I believe the used space on the source drive must be smaller than the destination drive.

4
  • 2
    A bigger drive might not be necessary. There are some filesystems and other modules/programs that can provide transparent compression. Really, ddrescue should work just fine with any seekable destination file, which can be compressed.
    – Bob
    Aug 12, 2013 at 14:40
  • I consider lots of big drives essential for recovery and forensics. Else its just to painful with every other variable we have on hand - especially messed up drives.
    – Journeyman Geek
    Aug 12, 2013 at 14:45
  • 3
    If it's your job, sure. But for a one-time thing, where you might be able to just sneak past, sometimes the purchase of a new drive is not desirable. Also, everything critical is backed up. Right? Right? (Seriously, @Manu, back up everything you can't afford to lose. Storage can and will fail.)
    – Bob
    Aug 12, 2013 at 14:49
  • Yes, lesson learned :) I have a NAS and a cloud based solution, but one was too old, the other did not copy everything. I should have checked sooner.
    – Manu
    Aug 12, 2013 at 16:58
10

It is possible to compress images created with ddrescue on-the-fly. Thing is, ddrescue requires a seekable destination, since it will go over in several passes (and therefore must have the ability to jump backwards to fill in earlier gaps, as @JourneymanGeek explains in his answer). This means you cannot use a pipe as the output, since a pipe is not seekable. Therefore you cannot pipe to compression programs.

One way to get around this is to use transparent compression. Some filesystems (notably, Btrfs, amongst others) provide this built-in. Alternatively, you can use filesystem drivers to provide transparent, seekable compressed storage, e.g. fusecompress.

4
  • 2
    Of course, compressibility of your source data can vary a lot. Especially since this is a drive image, which will not see 'free space' (very compressible) - it will actually copy whatever was in the blocks, whether they were 'deleted' or not. Also, the level of compression will vary across different methods used, and transparent compression often sacrifices size for speed, though that may be adjustable.
    – Bob
    Aug 12, 2013 at 14:51
  • 1
    I have thought about compressed file systems, but I have no experience at all with them.
    – Manu
    Aug 12, 2013 at 16:56
  • @Manu The fusecompress one looks fairly simple.
    – Bob
    Aug 12, 2013 at 17:56
  • 1
    brashear.me/blog/2017/11/14/… has instructions for using btrfs+lzo (I'm going to need to do something like this, since I ran badblocks on the drive and stopped it while it was filled with 0xAAAAAAAAA..., then wrote data to it, then it died and I'm trying to recover it. So it won't sparsify the output file because all the empty space is 0xAAAA instead of 0x0000. 🙄)
    – endolith
    Jan 23, 2020 at 21:57
0

There's a good post over at brashear.me which creates a btrfs partition stored as a file. The partition supports compression and can be used by ddrescue transparently. The blog post is reproduced below, should it ever become unavailable.

As disk sizes explode, I've found myself having to mirror disks which I don't have enough storage for. My tool of choice is ddrescue. However, it doesn't support compression because it needs to be able to seek through the output as it rescues data. A solution I've found is to create a sparse file, format it btrfs, and mount it with the compression option. This allows ddrescue to operate normally, while giving me fast + decent compression.

Create a sparse file which is larger than your source disk:

dd if=/dev/zero bs=1 count=0 seek=4T of=image-repository.img

Mount the file as a loop device:

sudo losetup /dev/loop0 image-repository.img

Partition the loop device:

sudo gdisk /dev/loop0 

Create new GPT by pressing o followed by y

Create new partition by pressing n and pressing enter a few times to accept defaults.

Write the table, by pressing w then y

Reread partitions from the loop device:

sudo partprobe /dev/loop0

Format the loop0p1 partition:

sudo mkfs.btrfs /dev/loop0p1

Mount the filesystem with compression enabled. Valid compression options are zlib, lzo, zstd:

mkdir /mnt/img-repo
sudo mount -o compress=zstd /dev/loop0p1 /mnt/img-repo

Set the c attr on the mount directory:

sudo chattr +c /mnt/img-repo

Now, when you create a ddrescue image inside of the /mnt/img-repo/ folder, it will be transparently compressed!

At a later date, you can access your ddrescue'd image using:

losetup --find --show image-repository.img
partprobe /dev/loopX     //where X is the output of the previous command
mkdir /mnt/img-repo
mount -o compress=zstd /dev/loopXp1 /mnt/img-repo
chattr +c /mnt/img-repo
cd /mnt/img-repo

And you can inspect the contents of the ddrescue'd image using:

losetup --read-only --find --show /mnt/img-repo/disk.img
mkdir /mnt/recovered_files/
mount /dev/loopXp1 /mnt/recovered_files/     //where X is the output of the previous command

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.