0

The backstory is in my previous question and my own answer to it.

At one moment I had two partial images created by ddrescue: one file on NTFS filesystem and the other on ext4.

I had noticed quite early in the process that the “size on disk” for both images was way lower than the total size, indicating (if I'm not mistaken) that those files had been written as “sparse”, i.e. that the empty data had not been actually allocated on the corresponding volumes, only the data which had already been rescued was accounted for. But at no point did I use the -S switch in my ddrescue commands, which specifies that the output file should be written as “sparse”.

Side note: What I did was using the -R switch (“reverse”) at the beginning, figuring that it would allocate the whole size of the input HDD right away (the idea was that it would result in a “cleaner” output, writing all the data sequentially on the receiving partition, so as to preserve the integrity of the image file even if something would go wrong with the file system and I would have to recover the recovery…) ; it did indeed increase the displayed size of the file to 931.5GB, but in fact the “size on disk” was only increased by whatever small amount of data which was copied during that step.

So the main question would be: how can this sparseness be explained? Why is the ddrescue copy not sequential by default?

Then, as I had two partial images, both containing some valid data missing from the other, I did the following:

  • I attempted to copy the rescued areas from the second image on the ext4 partition, missing in the first image, to that first image on the NTFS partition, which should have been very fast, both images being on the same healthy 2TB HDD (a Seagate ST2000DX001 with a max write speed close to 200MB/s). But it turned out that it was very slow: only 660KB/s.
  • So I stopped and did the opposite: I made ddrescue copy the rescued areas from the first image (on NTFS), missing in the second image, to that second image (on ext4). And now I got a copying rate of 43000KB/s or 43MB/s, which was considerably higher, and closer to a normal copying rate within the same HDD of that class and capacity.

The second question: could this weird behaviour be related to the performance issue which I experienced when writing to NTFS? Is the Linux NTFS driver known to have trouble dealing with large “sparse” files?

  • Your question was almost a wall of text, with the whole backstory which is not really important because one can understand the current issue without it (and if someone is interested then he or she can still follow the link given). I made the question body shorter, easier to read. I think it should be two separate questions: (1) about sparseness, (2) about NTFS behavior presumably because of sparseness. As long as there are no answers yet that address both issues, you can shrink this question to cover one topic and ask another question about the other. IMO this would be the right thing to do. – Kamil Maciorowski Sep 9 '17 at 8:29
  • Well, I tried to apply your suggestions, but it seems like it's never formal enough, and I'm quite puzzled by the rigorousness of the rules for posting on this otherwise excellent website (maybe you'll tell me that it's excellent only because it's rigorous ! :^p). I mean, it's already not so easy to formulate such technical issues clearly in approximately good english (which I think I did quite well so far), it can be discouraging to then have to be lost in seemingly endless editing so as to meet a certain standard, which, past a certain point, won't improve the quality/clarity significantly. – GabrielB Sep 9 '17 at 12:27
  • Here you suggest me to ask two separate questions, but to me they're one and the same, for I'm not even sure if I use the correct terminology, and if my interpretations / assumptions are even remotely correct. I'm just trying to recover as much as possible from a failing HDD using a tool called ddrescue. I know next to nothing about “sparseness”, I discovered the concept quite recently, I'm not sure how it operates at a logical level, and how it translates when it comes to the actual data written on the storage device, I couldn't say for sure if what I obseved is indeed related to sparseness. – GabrielB Sep 9 '17 at 12:39
  • (1) I apologize if you found my remarks intimidating. (2) I'm just a random guy on the Internet, you may ignore me. (3) Nevertheless I have some experience on this site to tell what questions attract good answers. (4) This question has a potential but it used to be awfully long, I invested my time to make it more readable. If I thought it wasn't formal enough I would have downvoted it but I didn't. – Kamil Maciorowski Sep 9 '17 at 20:04
  • 1
    "and I'm quite puzzled by the rigorousness of the rules for posting on this otherwise excellent website" - this excellence website is made possible by those rules that make it possible for you to find excellence answers to other people's excellent questions and hopefully by improving the quality of this question receive an excellent answer to your own question. Without those rules the amount of noise would make it impossible to find questions with quality answers. – Ramhound Sep 9 '17 at 23:56
1

This answer investigates the behavior of ddrescue to address the main question. If you're not interested in testing procedure then you may skip to my conclusions and interpretation near the end.

Testbed

$ uname -a
Linux foo 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/issue
Ubuntu 14.04.5 LTS \n \l

$ ddrescue -V
GNU ddrescue 1.17
…

The filesystem is btrfs; it shouldn't matter though as long as it supports sparse files.

Testing

At first I got 8 MiB of random data:

dd if=/dev/urandom of=random.chunk bs=1M count=8

Then I made it a loopback device and remembered its name:

loopdev=`sudo losetup -f --show random.chunk`

Next I created yet another device that consisted of

  • chunk 0: unreadable, 1 MiB
  • chunk 1: zeros, 2 MiB
  • chunk 2: unreadable, 4 MiB
  • chunk 3: data from random.chunk, 8 MiB
  • chunk 4: unreadable, 16 MiB

The code (it uses here document syntax):

sudo dmsetup create mydevice << EOF
    0  2048 error
 2048  4096 zero
 6144  8192 error
14336 16384 linear $loopdev 0
30720 32768 error
EOF

I confirmed with gdisk -l /dev/mapper/mydevice that the total size is 31 MiB as it should be.

Actual reading is done with:

ddrescue     /dev/mapper/mydevice  normal.raw  normal.log
ddrescue -R  /dev/mapper/mydevice normalR.raw normalR.log
ddrescue -S  /dev/mapper/mydevice  sparse.raw  sparse.log
ddrescue -RS /dev/mapper/mydevice sparseR.raw sparseR.log

And the results of ls -hls *.raw are

 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normal.raw
 10M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 normalR.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparse.raw
8.0M -rw-rw-r-- 1 kamil kamil 15M Sep 10 00:37 sparseR.raw

To be sure, I confirmed with cmp that all four files are identical when you read them. Four logfiles contained the same map of erroneous and healthy sectors.

Notice that

  • 15 MiB means the last chunk is missing;
  • 10 MiB indicates chunk 1 and chunk 3;
  • 8 MiB indicates chunk 3 only.

Cleaning

sudo dmsetup remove mydevice
sudo losetup -d $loopdev
unset loopdev
rm random.chunk normal.raw normal.log normalR.raw normalR.log sparse.raw sparse.log sparseR.raw sparseR.log

Conclusions

  • When it comes to file size, it doesn't matter whether you read in reverse (-R) or not.
  • Unreadable chunk at the very end of the input file doesn't contribute to the overall size of the output file.
  • Unreadable chunks that do contribute to overall file size are always sparse (if target filesystem supports this, of course).
  • The -S option only affects blocks of zeros that were actually read from the input file.

Interpretation

Above there were facts. This section is more like my opinion.

It appears ddrescue tries to save you diskspace whenever it can do this without additional work. When you use -S the tool has to do some computations to check if a given data block is all zeros. If there's a read error it doesn't need to compute anything, it can make the fragment sparse in the output file with no cost.

Solution

You wrote:

using the -R switch (“reverse”) at the beginning, figuring that it would allocate the whole size of the input HDD right away

We just saw it's a false assumption. In fact you described what -p does. ddrescue -p will preallocate space on disk for output file. When I did this during my tests the output file had 31 MiB and was not sparse (even with -S).

  • I would be interested in the testing procedure, but I'm quite lost in the details here ! :^p (Quite new to Linux, barely know enough syntax to run ddrescue and a few other related tools.) I made a simpler test on my own (see my answer) which seems to confirm your findings, and my earlier observations. Regarding the -p switch, I tried it at first but as I explained it proved to be a very long process as it seems to actually write the full empty file to the output (1TB in this case), instead of merely allocating its size, which should/could (?) be done instantaneously. – GabrielB Sep 11 '17 at 13:44
  • @GabrielB How about fallocate -l <exact_size_here> filename beforehand? My ddrescue or fallocate need about two seconds to allocate 70+ GiB, so 1 TB shouldn't take very long. I work on btrfs though, cannot test ext4 at the moment. – Kamil Maciorowski Sep 11 '17 at 13:57
  • I began the whole process with an NTFS partition (before switching to ext4, as explained in the first question), and that's when I tried the -P switch. So we might be on to something here... If pre-allocating a large volume on an ext4 partition is also done in a matter of seconds, it would indicate that the current NTFS support on Linux doesn't allow to perform that kind of operation as efficiently as with a native Linux file system (I never heard of btrfs by the way). But could it explain such a major slowdown ? – GabrielB Sep 11 '17 at 14:41
0

I made a different test on my own.

– I created a simple template ddrescue log/map file containing this :

0x00000000  0x100000  ?
0x100000  0x3FE00000  +
0x3FF00000  0x100000  ?

(Which means : within one GB of data in total, the first and last MB haven't been tried, the rest is considered as “rescued”.)

– I ran ddrescue with that log/map file, using this command (with the rescued image from the recovery of that 1TB HDD as input, cutting the output at 1GB) :

ddrescue -s 1073741824 [rescued_image_file] [test1GB] [test1GB.log]

The resulting [test1GB] file has a total size of 1GB as expected, but a “size on disk” of 2MB, meaning that only the data which was actually copied (first and last MB) has been allocated.

– Then I ran ddrescue with that 1GB file as input, with no template this time, first without and then with the -S switch (“sparse writes”).

ddrescue [test1GB] [test1GB-NS] [test1GB-NS.log]
ddrescue -S [test1GB] [test1GB-S] [test1GB-S.log]

And it appears that :

  • [test1GB-NS] (non-sparse) has a “size on disk” of 1GB -- so the whole file has been allocated and copied, even the empty sectors ; whereas...
  • [test1GB-S] (sparse) has a “size on disk” of only 1,2MB or 1114112 bytes -- meaning that the empty sectors have not been allocated, even those contained in the first and last MB.

I thought that “sparseness” was an all-or-nothing concept, just like file compression, yet apparently there is such a thing as a “partially sparse” file, and indeed ddrescue appears to be saving space that way -- which is not necessarily an advantage (and might indeed have an impact on performance) ; there should be a switch to make it allocate the full size of the output file on-the-fly (as opposed to pre-allocating which can be very long if the input is large), just like it does (obviously) when writing directly to a device or partition.

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.