2

In an attempt to recover as much data as possible off of a failing 3TB HDD, I proceeded like this :

  • I made a surface scan with HD Sentinel, which identified two small damaged areas and about 100 bad sectors (before that the count was at 16).
  • Then I identified which files were affected by bad sectors using various methods.
  • I moved those files (six large video files) to a special folder, and copied the rest of the files and folders, by decreasing order of importance ; everything was copied successfully, except one unimportant .eml file, which happened to be located close to the already identified bad sectors.
  • Then I figured that the safest way to get the most out of the remaining files (TV broadcasts which are no longer online and for which I have no backup) would be to use ddrescue – but since the only empty HDD I had was a 500GB one, I couldn't image everything. Some of those files are massively fragmented (6000 to 12000 fragments each – they were downloaded simultaneously, I guess that's why they were written in an “interlaced” pattern causing that level of fragmentation, because otherwise the HDD had plenty of free space), so I could not recover them simply by extracting the sectors they occupied, but I thought that by imaging the first 10GB, usually containing the whole MFT and all the other system files, plus the four areas where those files were located, I would be able to extract them easily from the image, using WinHex or R-Studio.

But unfortunately, I didn't get the whole MFT : some of it (as I later found out examining the complete nfi.exe listing of that partition I had made earlier) is located around the 200GB mark, and a third chunk is located at the very end of the partition, close to the 3TB mark. I didn't think that the HDD's state would deteriorate so quickly during the recovery attempt (now it's got more than 12000 reallocated sectors plus 9000 pending sectors, just a few hours later !...), and I didn't take the precaution to save the MFT from WinHex when I could. Now, with ddrescue it has become painfully slow, I probably won't get the whole MFT. Also, if I open that partial image with WinHex, it uses the same volume snapshot that was created when I examined the physical device, the files I want are listed with their correct size and dates, if I click on them it displays the correct first sector, but it still can't extract them (only 0 byte files are extracted), apparently the volume snapshot doesn't contain all the required data regarding the allocated sectors, WinHex seems to rely on the MFT at that point, so that won't work either.

But I have recovered a good portion of the chunks of data containing those six files, and I have for each of them a detailed list of the sectors / clusters they occupy (obtained with three different tools : nfi.exe, Recuva, HD Sentinel). Now, how can I rebuild those files with that information, using an automated script ? (It would be an impossible task to do this manually.)

With ddrescue I could use the -i (input position) -o (output position) and -s (input size) switches, but how could I automate the process, and run those thousands of commands all at once ?

On Windows, I know a command line tool called dsfo which can extract data from any source to a destination file with a command like this :

dsfo [source] [offset] [size] [destination]

I could edit my list of sectors/clusters with a combination of Calc and TEDNotepad, to create a list of dsfo commands, but it would create thousands of chunks, which I would then have to join somehow. Is there a better way to do this in one step ?



EDIT :

So I took the list of clusters / sectors for one of these files, generated by HD Sentinel, which is presented like this :

R:\fichiers corrompus\2017_07_2223_58 - Arte - Pink Floyd - The Dark Side of the Moon Live.mp4
Total Size: 883 787 365 bytes   Position: 0     Attributes: Arc
Number of file fragments: 6040
VCN: 0  LCN: 516530293  Length: 4288    sectors: 4132506536 - 4132540839
VCN: 4288   LCN: 516534613  Length: 16  sectors: 4132541096 - 4132541223
VCN: 4304   LCN: 516534645  Length: 64  sectors: 4132541352 - 4132541863
VCN: 4368   LCN: 516534725  Length: 16  sectors: 4132541992 - 4132542119
VCN: 4384   LCN: 516534757  Length: 48  sectors: 4132542248 - 4132542631
VCN: 4432   LCN: 516534853  Length: 32  sectors: 4132543016 - 4132543271
VCN: 4464   LCN: 516534901  Length: 16  sectors: 4132543400 - 4132543527
VCN: 4480   LCN: 516534933  Length: 48  sectors: 4132543656 - 4132544039
VCN: 4528   LCN: 516535013  Length: 16  sectors: 4132544296 - 4132544423
...
VCN: 215760 LCN: 568126709  Length: 9   sectors: 4545277864 - 4545277935

The first field probably stands for “Virtual Cluster Number” (haven't found a detailed description in the integrated help), anyway, this value obviously represents the cluster number relative to the begining of the file. The second value must be the “Logical Cluster Number” and is the cluster number relative to the begining of the partition (see below, I had it wrong at first, thinking that this value was relative to the whole device). The third value represents the length of each fragment, also measured in clusters. Those three values should suffice for my intents and purposes.

I imported that into TED Notepad, and used the “Tools” > “Lines” > “Columns, numbers” function, selected columns 2, 3, 1 with tabs as separators, which produced this output :

LCN: 516530293  Length: 4288    VCN: 0
LCN: 516534613  Length: 16  VCN: 4288
LCN: 516534645  Length: 64  VCN: 4304
LCN: 516534725  Length: 16  VCN: 4368
LCN: 516534757  Length: 48  VCN: 4384
LCN: 516534853  Length: 32  VCN: 4432
LCN: 516534901  Length: 16  VCN: 4464
LCN: 516534933  Length: 48  VCN: 4480
LCN: 516535013  Length: 16  VCN: 4528
...
LCN: 568126709  Length: 9   VCN: 215760 

Then I imported that into Calc with tabs and spaces as separators, added a column to calculate the input offset from the cluster number (=LCN*8*512), another to calculate the length in bytes from the length in clusters (=Length*8*512) and finally another to get the output offset from the VCN value (=VCN*8*512), pasted the formulas to all the other lines, removed the extra columns, replaced “LCN:” with “ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i”, replaced “Length:” with “-s”, replaced “VCN:” with “-o”...
Now I've got this (except there are 6000-12000 lines for each file) :

ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115708080128 -s 17563648 -o 0
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115725774848 -s 65536 -o 17563648
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115725905920 -s 262144 -o 17629184
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726233600 -s 65536 -o 17891328
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726364672 -s 196608 -o 17956864
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726757888 -s 131072 -o 18153472
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726954496 -s 65536 -o 18284544
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115727085568 -s 196608 -o 18350080
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115727413248 -s 65536 -o 18546688
...
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2327047000064 -s 36864 -o 883752960

So, what is the simplest way to run this huge series of commands on a Knoppix live system ? What is in Linux the equivalent of a batch script for the command prompt in Windows ?

(I could find that particular file on a P2P network, so it will allow me to test if this method works flawlessly, and if it does, to assess the level of damage. No such luck for the five others. One of those is not fragmented so I could extract it as one chunk of data : there are many blank sectors near the end, but the rest is readable. So there remain four files to extract that way.)

8
  • By insisting working on the affected drive you are making things worse. I suggest you take the time to buy a new 3TB drive and ddrescue a whole copy of the damaged one. Then you can extract the files with RecuperaBit (disclosure: I am the author). – Andrea Lazzarotto Nov 14 '17 at 6:59
  • Precisely, at this point it might be too late to make a whole copy, and I can't even get the areas where the MFT is located (which would be necessary to extract fragmented files from any data recovery software – I tried R-Studio to no avail). Besides, I already copied almost everything when the drive was still in not-so-bad condition. That's why I'm trying to salvage what is salvageable of those 6 remaining files by extracting them from the image as it is, just with a list of the sectors they occupy. I'm pretty sure it can be done this way, I just need an answer to the questions in bold type. – GabrielB Nov 14 '17 at 17:25
  • Oh, I see. The Linux equivalent of a batch script is a shell script. The most common shell is called Bash. – Andrea Lazzarotto Nov 14 '17 at 19:25
  • Also, you should consider adding a final command to cat the output files together. – Andrea Lazzarotto Nov 14 '17 at 19:26
  • I made a quick research in the meantime : indeed a shell script seems quite easy to use (only difference with Windows is that it has to be made executable, and run with "./" before its name, otherwise it's just a list of commands in plain text). Normally, with the way these commands are typed, and the way ddrescue works, it should write all the fragments together into the same output file, right ? That's why this method seems preferable to the Windows one with dsfo, even though I'm less familiar with Linux. If anybody knows a Windows tool which can do this in one step, let me know. – GabrielB Nov 14 '17 at 22:57
2

So I did run those ddrescue scripts (first made them executable with the “chmod +x” command, then called them with ./name_of_the_script) :

– At first the commands didn't work, ddrescue gave only errors, I had to edit the scripts again so that the parameters would be placed before the names of the input and output files. The commands then looked like this :

ddrescue -P -i 2115843346432 -s 17563648 -o 0  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861041152 -s 65536 -o 17563648  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861172224 -s 262144 -o 17629184  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861499904 -s 65536 -o 17891328  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115861630976 -s 196608 -o 17956864  ST3000DM001-2.dd 201707222358.mp4
ddrescue -P -i 2115862024192 -s 131072 -o 18153472  ST3000DM001-2.dd 201707222358.mp4
...
ddrescue -P -i 2327182266368 -s 36864 -o 883752960  ST3000DM001-2.dd 201707222358.mp4

(Total size of that file : 883787365, or 883789824 with the slack space.)
(“-P” stands for “preview”, “-i” for “input position”, “-s” for “size”, “-o” for “output position”.)
(The paths could be omitted as the scripts, the image file and the expected output files were all in the same directory.)

– Then the first attempt produced an unreadable file, without a correct MP4 header. Why ? Because the list provided by Hard Disk Sentinel gives the physical/absolute sector numbers, but the logical cluster numbers (I verified by opening the image file with WinHex), so I had to add 264192x512 to the input offset calculation (the partition offset being 264192 sectors, or 129MB).

– Then it worked. It took just a few minutes and produced five video files, which are mostly readable, skippable to the end, with their expected content — I haven't watched them completely, but it seems as flawless as can be.

enter image description here

(I made all this on a secondary computer running on Knoppix live from a memory card, and used TeamViewer to command it from my primary computer on Windows 7, and also to be able to transfer the script files easily. Maybe there's a simpler setup for such purposes, but, well, it works ! :^p)

– But of course there are corrupted parts, since there were unreadable sectors in that partial image. How could I know where, quickly and reliably ? Well...
I had the idea to use ddrescue's “generate” mode, which creates logfiles (or mapfiles as they're called now) by parsing the output and considering that totally empty sectors are unread sectors, marked “?”, the rest being marked “+”. Since ddrescue expects an input file and an output file, but only the output file is actually parsed in that mode, I created dummy input files with this command, which copies only 1MB but extends the size to the size of the output files (just to save time and space) :

ddrescue -s 1048576 -x 883789824 201707222358.mp4 201707222358copy.mp4

Then I ran the “generate” command :

ddrescue -G 201707222358copy.mp4 201707222358.mp4 201707222358-generate.log

And then I opened those files with ddrescueview :

enter image description here enter image description here

(Three of the six files are seriously damaged like the first one above, with large chunks of empty data, the three others only have a few corrupted sectors like the second one. The second one is the one which was not fragmented, I extracted it with a single ddrescue command.)

And then I patted myself on the back with one hand, while I was slapping my face with the other for having used that 3TB HDD every day for months with no backup... (At first it was supposed to store only temporary stuff, while I would be making room on other HDDs, but it took longer than anticipated, and I ran out of space to store such videos, and even my personal pictures and videos at some point, it could have been a major disaster, but “it's only a glitch”, as Dick Jones would have said.)

4
  • 1
    You may rearrange your question and the answer so the question clearly states the initial problem and the answer is step-by-step solution as a whole. For now it seems the question contains a part of the solution and the answer just continues. Don't get me wrong; editing your question to indicate some progress was a right thing to do then, tiding the things up is nice thing to do now. But regardless of whether you'll do it or not, you have +1 from me. I hardly ever work with NTFS, I will probably never use nor test your solution, but I believe it works and I appreciate you share it. – Kamil Maciorowski Nov 15 '17 at 6:59
  • Thanks for the nice comment ! At least my efforts of the night weren't completely pointless... Regarding your suggestion, I wonder if it's worth the trouble now to turn this into a step-by-step guide : this was a very specific issue, where I've been both extra careful (for saving the lists of sectors/clusters) and extra careless (for not having a backup, and failing to save the whole MFT when I could before I attempted any serious recovery of the problematic files, which I knew were highly fragmented). Still, that's an interesting problem, and the solution might be applicable to other tasks. – GabrielB Nov 15 '17 at 7:25
  • OK, maybe it's not worth the trouble. One way or another this question and answer (read as a whole) is a good piece of work. – Kamil Maciorowski Nov 15 '17 at 7:43
  • ST3000DM001 are bad hdd – v1nce Apr 8 at 20:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.