0

I need someone to please sanity check my manual NTFS recovery process so far because it's lead me to a brick wall.

Background:

  • I had a 1TB NTFS external drive (WD Elements) which contained mostly photos.
  • Somehow, it's corrupted and appears as a raw disk on Windows.
  • It appears on a Linux system in the /dev/disk/by-path (and by-id, by-uuid etc.) directories and appears as /dev/sdb.
  • EaseUS is able to find (nearly?) all of my photos on there with a quick scan (not the heavy lifting deep scan)
  • EaseUS finds about 70GB of files (mostly photos).
  • I think the NTFS records are corrupted i.e. it's not a mechanical failure.
  • I'd like to attempt recovery myself for fun and profit.
  • I don't have another drive large enough to make a complete image of the corrupted drive.

I need to parse the NTFS MFT $File records because:

  • I'd like to get the original file names and directory structure back.
  • If an image is not written in contiguous clusters I won't successfully recover it just by looking for image file signatures.

The plan is:

  1. Image a portion of the corrupted disk.
  2. Parse it to identify and read MFT $File records.
  3. Use the $File records (and specifically the $Data attribute thereof) to determine the data runs of each file.
  4. Knowing the data runs for a file, I can pick out the bytes of a file from the image I created using ddrescue.
  5. Rinse and repeat until I'm done.

Firstly - does that sound like a reasonable plan?

What I've done:

  1. Found a bunch of $File records
  2. Parsed one to get the data runs
  3. Read the raw bytes at the location specified by the data run.

Specifically:

  1. Used ddrescue to image 100GB (starting at 0) of the corrupted disk.
    1. I imagine that nearly all of the actual data I need is written within the first 100GB since the total volume of interesting data is 70GB. I can repeat this whole process over subsequent 100GB portions if necessary.
    2. The command I used to image the first 100GB was ddrescue /dev/sdb ~/mydisk.img ~/mydisk.map --size=100G.
    3. ddrescue did encounter I/O errors and reported that it only recovered about 99.57%.
    4. The start of the image (first 20MB or so) seems empty so this might be the reason for the drive failure. I'm ignoring that for now.
  2. Read over the 100GB image and located all instances of the ASCII string "FILE" which denotes the start of a $File record in the MFT.
    1. This has also triggered on false positives such as the word "PROFILE" in the middle of an arbitrary file.
    2. Therefore I'm only considering results where the distance between one occurrence of "FILE" and the next is <= 1024 bytes since that's the maximum MFT record size. If there's 3MB between one occurrence of "FILE" and the next then it's unlikely to be a $File record.
  3. Iterate over presumed $File records (size <= 1024 bytes) and extracted $Data attributes.
  4. Parsed it for data runs (ignoring the discussion about resident vs non-resident attributes which I think I understand but isn't part of my question).

So I went through the above process and selected one of the $File records and identified its data runs. Here's the $Data attribute (header and contents):

80 00 00 00 48 00 00 00  01 00 00 00 00 00 01 00
00 00 00 00 00 00 00 00  FA 03 00 00 00 00 00 00
40 00 00 00 00 00 00 00  00 B0 3F 00 00 00 00 00
F4 AC 3F 00 00 00 00 00  F4 AC 3F 00 00 00 00 00
32 FB 03 ED C8 11 00 00  FF FF FF FF 82 79 47 11

The data run details are the first half of the last row, just before the FF FF FF FF end of attribute marker:

  • length byte: 32
  • cluster run number: FB 03 (little endian) = 1019 clusters in the run
  • cluster start number: ED C8 11 = 1165549 is the first cluster of the run
  • the next 00 indicates that there are no more runs.

Now, considering there are 512 bytes per sector and 128 sectors per cluster, I read (1019 * 128 * 512) bytes from the 100GB image starting at (1165549 * 128 * 512).

Unfortunately that left me with a 66.8MB file of mostly 0x00 although towards the end there was some data. I'm pretty sure I've got something wrong and I just found some data by accident (although I do happen to end on a JPG end-of-file marker (DD F9).

Is my approach to this whole task reasonable and I've just made a small error somewhere?

Or have I misunderstood something fundamental about NTFS and this is completely the wrong idea?

3 Answers 3

1

Firstly - does that sound like a reasonable plan?

No. I mean I didn't look in depth at you method for decoding runs but, start clusters are relative to start of file system ( = start of volume / partition when we talk NTFS). So any MFT entry that points to a cluster number, points to cluster number relative to start of partition, not relative to your arbitrary 100 MB part of the volume.

Also, MFT is 'self referencing', so first thing to try is to find start of MFT from which you can then derive all MFT entries. If you can not find start of MFT try locating the mirror of the MFT instead.

So, to properly decode a MFT entry and get the data it is referencing we need:

  • Offset to start of partition.
  • Correct cluster size

So if we decode start cluster we can do: (cluster no. * sectors / cluster) + offset partition

How did you determine 128 sectors per cluster? this does not seem right at all! See defaults here: https://www.disktuna.com/default-cluster-sizes-for-fat-exfat-and-ntfs/.

0

It's unprofessional to expose a dammaged disc to the read load you described. The first requirement is to duplicate that drive to a stable healthy one to avoid the loss of additional sectors and to have no unreadable sectors anymore. The fact that the unreadable sectors could not be copied is sad but the key is to avoid you or a recovery programm to handle read errors.

Your recovery attempts may very well have already caused additional dammage on that drive.

As metadata structures and data are not necessarily laid out in a linear manner you cannot compensate the lack of owning at least one drive that is big enough by subsequently reading 100GB slices somewhere into free drive storage elsewhere. You need random access. Unfortunately in your case once a structure points into your next 100GB slice you have to empty your 100GB storage area.

If you are already parsing structures in your preferred programming language there is no need to run unix commands like dd for the copy job. ddrescue is needed just once at the beginning of your recovery attempt.

If you just want to learn NTFS internals that's fine and a cool thing to do but you can learn that already on storage like USB sticks. Big drives are not necessary.

Please think about for what reason Easeus did not recover your desired metadata like file names and folder structures.

1
  • EaseUS did find a lot of the files but I'm reluctant to fork out the $70 first without at least giving it a go myself. RE: the non-linearity of the drive - I thought drives were linear starting at byte 0 of sector 0 of cluster 0 and going all the way up to the end of the physical drive. NTFS describes, among other things, how to traverse the drive to the start of the next NTFS record/file/interesting bit of info. Are you saying that, if I use a tool like ddrescue to read bytes 0 - 1000000, I shouldn't expect to have read the NTFS metadata which describes the layout? Jul 16, 2020 at 21:27
0

I thought drives were linear starting at byte 0 of sector 0 of cluster 0 and going all the way up to the end of the physical drive.

Yes and no. Storage addresses are linear, starting with byte 0. Sector addresses are linear as well.

Cluster addresses are linear and starting with cluster number zero but only relative to their respective partition. There is no disc-wide cluster labeling only a partition wide cluster labeling starting from zero.

Are you saying that, if I use a tool like ddrescue to read bytes 0 - 1000000, I shouldn't expect to have read the NTFS metadata which describes the layout?

The NTFS metadata involves not only the boot sector but the master file table and other files as well. When reading 1 MByte from the start of the drive you will only read a fraction of the master file table provided that it is located there. Its position is not fixed and is likely to change upon defragmentation.

Use the strategy to experiment and learn about your file system here: https://forum.cgsecurity.org/phpBB3/viewtopic.php?p=31304#p31304

When trying to decrypt NTFS you are so much better knowing the intended outcome already. It is a kind of deciphering process then like a "known plaintext attack" where you know the whole plaintext.

Somebody just downvoted my answers. If that was you please explain the reason. Is there an error somewhere? I'd like to know! Thank you.

4
  • Hi. I didn't downvote your answers but I'll admit that I feel your second answer provides more constructive feedback than your first. I'm taking a look at the link you provided and it seems that the solution there was to re-write the root directory cluster rather than parse for file locations and manually read the bytes to a new location. I hadn't thought of that approach so thanks. I'll think some more on that! Jul 17, 2020 at 16:18
  • The solution depends on the individual dammage. The key here is not the found solution but to learn about the file system using a healthy one where you know what you should find so that you can verify yourself that your code behaved correctly.
    – r2d3
    Jul 17, 2020 at 21:38
  • Here's another question I had from earlier in the process for more context around the damage: superuser.com/questions/1567193/what-is-the-likely-ntfs-state . I'm fairly confident my code did what I intended (I carefully compared is parsing of the data runs for the file with calculations by hand) but it seems my intention was wrong. I just don't understand how I ended up with a $File record with a $Data attribute that pointed to a 66.8MB image. But maybe I was waylaid by incorrect values for bytes/sector or sectors/cluster? Jul 18, 2020 at 20:47
  • I do not understand what "my intention was wrong" means. The same applies to "I just don't understand how I ended up" Do not compare your code to your manual operation. Have your code work on a good file system where you verify the outcome by accessing your file system in the regular manner. It seems that you do not understand the reasoning behind my advice.
    – r2d3
    Jul 20, 2020 at 6:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .