In an attempt to recover as much data as possible off of a failing 3TB HDD, I proceeded like this :
- I made a surface scan with HD Sentinel, which identified two small damaged areas and about 100 bad sectors (before that the count was at 16).
- Then I identified which files were affected by bad sectors using various methods.
- I moved those files (six large video files) to a special folder, and copied the rest of the files and folders, by decreasing order of importance ; everything was copied successfully, except one unimportant .eml file, which happened to be located close to the already identified bad sectors.
- Then I figured that the safest way to get the most out of the remaining files (TV broadcasts which are no longer online and for which I have no backup) would be to use ddrescue – but since the only empty HDD I had was a 500GB one, I couldn't image everything. Some of those files are massively fragmented (6000 to 12000 fragments each – they were downloaded simultaneously, I guess that's why they were written in an “interlaced” pattern causing that level of fragmentation, because otherwise the HDD had plenty of free space), so I could not recover them simply by extracting the sectors they occupied, but I thought that by imaging the first 10GB, usually containing the whole MFT and all the other system files, plus the four areas where those files were located, I would be able to extract them easily from the image, using WinHex or R-Studio.
But unfortunately, I didn't get the whole MFT : some of it (as I later found out examining the complete nfi.exe listing of that partition I had made earlier) is located around the 200GB mark, and a third chunk is located at the very end of the partition, close to the 3TB mark. I didn't think that the HDD's state would deteriorate so quickly during the recovery attempt (now it's got more than 12000 reallocated sectors plus 9000 pending sectors, just a few hours later !...), and I didn't take the precaution to save the MFT from WinHex when I could. Now, with ddrescue it has become painfully slow, I probably won't get the whole MFT. Also, if I open that partial image with WinHex, it uses the same volume snapshot that was created when I examined the physical device, the files I want are listed with their correct size and dates, if I click on them it displays the correct first sector, but it still can't extract them (only 0 byte files are extracted), apparently the volume snapshot doesn't contain all the required data regarding the allocated sectors, WinHex seems to rely on the MFT at that point, so that won't work either.
But I have recovered a good portion of the chunks of data containing those six files, and I have for each of them a detailed list of the sectors / clusters they occupy (obtained with three different tools : nfi.exe, Recuva, HD Sentinel). Now, how can I rebuild those files with that information, using an automated script ? (It would be an impossible task to do this manually.)
With ddrescue I could use the -i (input position) -o (output position) and -s (input size) switches, but how could I automate the process, and run those thousands of commands all at once ?
On Windows, I know a command line tool called dsfo which can extract data from any source to a destination file with a command like this :
dsfo [source] [offset] [size] [destination]
I could edit my list of sectors/clusters with a combination of Calc and TEDNotepad, to create a list of dsfo commands, but it would create thousands of chunks, which I would then have to join somehow. Is there a better way to do this in one step ?
So I took the list of clusters / sectors for one of these files, generated by HD Sentinel, which is presented like this :
R:\fichiers corrompus\2017_07_2223_58 - Arte - Pink Floyd - The Dark Side of the Moon Live.mp4 Total Size: 883 787 365 bytes Position: 0 Attributes: Arc Number of file fragments: 6040 VCN: 0 LCN: 516530293 Length: 4288 sectors: 4132506536 - 4132540839 VCN: 4288 LCN: 516534613 Length: 16 sectors: 4132541096 - 4132541223 VCN: 4304 LCN: 516534645 Length: 64 sectors: 4132541352 - 4132541863 VCN: 4368 LCN: 516534725 Length: 16 sectors: 4132541992 - 4132542119 VCN: 4384 LCN: 516534757 Length: 48 sectors: 4132542248 - 4132542631 VCN: 4432 LCN: 516534853 Length: 32 sectors: 4132543016 - 4132543271 VCN: 4464 LCN: 516534901 Length: 16 sectors: 4132543400 - 4132543527 VCN: 4480 LCN: 516534933 Length: 48 sectors: 4132543656 - 4132544039 VCN: 4528 LCN: 516535013 Length: 16 sectors: 4132544296 - 4132544423 ... VCN: 215760 LCN: 568126709 Length: 9 sectors: 4545277864 - 4545277935
The first field probably stands for “Virtual Cluster Number” (haven't found a detailed description in the integrated help), anyway, this value obviously represents the cluster number relative to the begining of the file. The second value must be the “Logical Cluster Number” and is the cluster number relative to the begining of the partition (see below, I had it wrong at first, thinking that this value was relative to the whole device). The third value represents the length of each fragment, also measured in clusters. Those three values should suffice for my intents and purposes.
I imported that into TED Notepad, and used the “Tools” > “Lines” > “Columns, numbers” function, selected columns 2, 3, 1 with tabs as separators, which produced this output :
LCN: 516530293 Length: 4288 VCN: 0 LCN: 516534613 Length: 16 VCN: 4288 LCN: 516534645 Length: 64 VCN: 4304 LCN: 516534725 Length: 16 VCN: 4368 LCN: 516534757 Length: 48 VCN: 4384 LCN: 516534853 Length: 32 VCN: 4432 LCN: 516534901 Length: 16 VCN: 4464 LCN: 516534933 Length: 48 VCN: 4480 LCN: 516535013 Length: 16 VCN: 4528 ... LCN: 568126709 Length: 9 VCN: 215760
Then I imported that into Calc with tabs and spaces as separators, added a column to calculate the input offset from the cluster number (=LCN*8*512), another to calculate the length in bytes from the length in clusters (=Length*8*512) and finally another to get the output offset from the VCN value (=VCN*8*512), pasted the formulas to all the other lines, removed the extra columns, replaced “LCN:” with “ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i”, replaced “Length:” with “-s”, replaced “VCN:” with “-o”...
Now I've got this (except there are 6000-12000 lines for each file) :
ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115708080128 -s 17563648 -o 0 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115725774848 -s 65536 -o 17563648 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115725905920 -s 262144 -o 17629184 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726233600 -s 65536 -o 17891328 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726364672 -s 196608 -o 17956864 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726757888 -s 131072 -o 18153472 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115726954496 -s 65536 -o 18284544 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115727085568 -s 196608 -o 18350080 ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2115727413248 -s 65536 -o 18546688 ... ddrescue /media/sdb1/ST3000DM001-2.dd /media/sdb1/201707222358.mp4 -i 2327047000064 -s 36864 -o 883752960
So, what is the simplest way to run this huge series of commands on a Knoppix live system ? What is in Linux the equivalent of a batch script for the command prompt in Windows ?
(I could find that particular file on a P2P network, so it will allow me to test if this method works flawlessly, and if it does, to assess the level of damage. No such luck for the five others. One of those is not fragmented so I could extract it as one chunk of data : there are many blank sectors near the end, but the rest is readable. So there remain four files to extract that way.)