ddrescue, “size on disk” lower than total size, with possible impact on performance when writing to NTFS

Question

The backstory is in my previous question and my own answer to it.

At one moment I had two partial images created by ddrescue: one file on NTFS filesystem and the other on ext4.

I had noticed quite early in the process that the “size on disk” for both images was way lower than the total size, indicating (if I'm not mistaken) that those files had been written as “sparse”, i.e. that the empty data had not been actually allocated on the corresponding volumes, only the data which had already been rescued was accounted for. But at no point did I use the -S switch in my ddrescue commands, which specifies that the output file should be written as “sparse”.

_{Side note: What I did was using the -R switch (“reverse”) at the beginning, figuring that it would allocate the whole size of the input HDD right away (the idea was that it would result in a “cleaner” output, writing all the data sequentially on the receiving partition, so as to preserve the integrity of the image file even if something would go wrong with the file system and I would have to recover the recovery…) ; it did indeed increase the displayed size of the file to 931.5GB, but in fact the “size on disk” was only increased by whatever small amount of data which was copied during that step.}

So the main question would be: how can this sparseness be explained? Why is the ddrescue copy not sequential by default?

Then, as I had two partial images, both containing some valid data missing from the other, I did the following:

• I attempted to copy the rescued areas from the second image on the ext4 partition, missing in the first image, to that first image on the NTFS partition, which should have been very fast, both images being on the same healthy 2TB HDD (a Seagate ST2000DX001 with a max write speed close to 200MB/s). But it turned out that it was very slow: only 660KB/s.
• So I stopped and did the opposite: I made ddrescue copy the rescued areas from the first image (on NTFS), missing in the second image, to that second image (on ext4). And now I got a copying rate of 43000KB/s or 43MB/s, which was considerably higher, and closer to a normal copying rate within the same HDD of that class and capacity.

The second question: could this weird behaviour be related to the performance issue which I experienced when writing to NTFS? Is the Linux NTFS driver known to have trouble dealing with large “sparse” files?

Answer 1

This answer investigates the behavior of ddrescue to address the main question. If you're not interested in testing procedure then you may skip to my conclusions and interpretation near the end.

Testbed

$ uname -a
Linux foo 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

$ ddrescue -V
GNU ddrescue 1.17
…

The filesystem is btrfs; it shouldn't matter though as long as it supports sparse files.

Testing

At first I got 8 MiB of random data:

dd if=/dev/urandom of=random.chunk bs=1M count=8

Then I made it a loopback device and remembered its name:

loopdev=`sudo losetup -f --show random.chunk`

Next I created yet another device that consisted of

• chunk 0: unreadable, 1 MiB
• chunk 1: zeros, 2 MiB
• chunk 2: unreadable, 4 MiB
• chunk 3: data from random.chunk, 8 MiB
• chunk 4: unreadable, 16 MiB

The code (it uses here document syntax):

sudo dmsetup create mydevice << EOF
    0  2048 error
 2048  4096 zero
 6144  8192 error
14336 16384 linear $loopdev 0
30720 32768 error
EOF

I confirmed with gdisk -l /dev/mapper/mydevice that the total size is 31 MiB as it should be.

Actual reading is done with:

ddrescue     /dev/mapper/mydevice  normal.raw  normal.log
ddrescue -R  /dev/mapper/mydevice normalR.raw normalR.log
ddrescue -S  /dev/mapper/mydevice  sparse.raw  sparse.log
ddrescue -RS /dev/mapper/mydevice sparseR.raw sparseR.log

And the results of ls -hls *.raw are

 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normal.raw
 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normalR.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparse.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparseR.raw

To be sure, I confirmed with cmp that all four files are identical when you read them. Four logfiles contained the same map of erroneous and healthy sectors.

Notice that

• 15 MiB means the last chunk is missing;
• 10 MiB indicates chunk 1 and chunk 3;
• 8 MiB indicates chunk 3 only.

Cleaning

sudo dmsetup remove mydevice
sudo losetup -d $loopdev
unset loopdev
rm random.chunk normal.raw normal.log normalR.raw normalR.log sparse.raw sparse.log sparseR.raw sparseR.log

---

Conclusions

• When it comes to file size, it doesn't matter whether you read in reverse (-R) or not.
• Unreadable chunk at the very end of the input file doesn't contribute to the overall size of the output file.
• Unreadable chunks that do contribute to overall file size are always sparse (if target filesystem supports this, of course).
• The -S option only affects blocks of zeros that were actually read from the input file.

Interpretation

Above there were facts. This section is more like my opinion.

It appears ddrescue tries to save you diskspace whenever it can do this without additional work. When you use -S the tool has to do some computations to check if a given data block is all zeros. If there's a read error it doesn't need to compute anything, it can make the fragment sparse in the output file with no cost.

Solution

You wrote:

using the -R switch (“reverse”) at the beginning, figuring that it would allocate the whole size of the input HDD right away

We just saw it's a false assumption. In fact you described what -p does. ddrescue -p will preallocate space on disk for output file. When I did this during my tests the output file had 31 MiB and was not sparse (even with -S).

Answer 2

I made a different test on my own.

– I created a simple template ddrescue log/map file containing this :

0x00000000  0x100000  ?
0x100000  0x3FE00000  +
0x3FF00000  0x100000  ?

(Which means : within one GB of data in total, the first and last MB haven't been tried, the rest is considered as “rescued”.)

– I ran ddrescue with that log/map file, using this command (with the rescued image from the recovery of that 1TB HDD as input, cutting the output at 1GB) :

ddrescue -s 1073741824 [rescued_image_file] [test1GB] [test1GB.log]

The resulting [test1GB] file has a total size of 1GB as expected, but a “size on disk” of 2MB, meaning that only the data which was actually copied (first and last MB) has been allocated.

– Then I ran ddrescue with that 1GB file as input, with no template this time, first without and then with the -S switch (“sparse writes”).

ddrescue [test1GB] [test1GB-NS] [test1GB-NS.log]
ddrescue -S [test1GB] [test1GB-S] [test1GB-S.log]

And it appears that :

• [test1GB-NS] (non-sparse) has a “size on disk” of 1GB -- so the whole file has been allocated and copied, even the empty sectors ; whereas...
• [test1GB-S] (sparse) has a “size on disk” of only 1,2MB or 1114112 bytes -- meaning that the empty sectors have not been allocated, even those contained in the first and last MB.

I thought that “sparseness” was an all-or-nothing concept, just like file compression, yet apparently there is such a thing as a “partially sparse” file, and indeed ddrescue appears to be saving space that way -- which is not necessarily an advantage (and might indeed have an impact on performance) ; there should be a switch to make it allocate the full size of the output file on-the-fly (as opposed to pre-allocating which can be very long if the input is large), just like it does (obviously) when writing directly to a device or partition.