2

I have several hundred photo files that were inadvertently moved from a Windows NTFS formatted, compressed external drive, to a second external drive that was formatted by Mac OS X. The move was performed under the Mac OS X operating system. Unfortunately, the file move process was completed without all of the original files being "uncompressed" by the Windows file system.

Is anyone aware of any recovery service, application or manual method of conversion that will allow me to recover these photo files, all of which are now seen as "corrupted" JPG and RAW files? Any assistance or suggestions you may be able to offer would be greatly appreciated.

  • 2
    Does the OS X NTFS driver not understand the Compressed attribute? That's... kind of terrible. You could try copying the files back onto the NTFS volume using the same driver, but you would need to re-set the Compressed attribute on them, too. – CBHacking Oct 14 '15 at 23:49
  • No, the Mac OS X driver doesn't understand a compressed NTFS file. The "uncompressing" function should have been handed by Windows, but the files were moved out of the Windows environment prior to the uncompress operation being completed by Windows. Copying the files back to a compressed NTFS file system didn't work. Could you please provide steps on setting the compressed attribute of the files? I'm willing to try anything, even going to the bit level if necessary. – GMason Oct 14 '15 at 23:56
  • You'd need to add a file attribute to the master file table. The ntfs-3g driver might understand how to do that. I have a better idea, though, and I'll explain it in an answer. – CBHacking Oct 15 '15 at 0:09
  • For what it's worth: it won't matter. Photos and videos won't compress much even with the best of compression algorithms, let alone with the high-speed low-compression algorithm the NTFS file system uses. – Mahmoud Al-Qudsi Oct 15 '15 at 0:23
  • Yes, I realize that photos don't compress very much, but the photo files in question were sitting on an NTFS compressed drive that held many different types of data. It was easy to simply compress the entire drive, rather than to pick and choose which sub-directories were compressed, and which ones were not compressed. Obviously, I now wish I had made the effort... Isn't hind-sight a wonderful thing! – GMason Oct 15 '15 at 1:11
1

NTFS compression is indicated by a flag in the file metadata, which is stored in the master file table (MFT). Setting it directly is difficult, because usually when you change that flag, the file system driver will compress or decompress the file for you. You could probably add this flag manually - it's just a single bit in the file attributes DWORD - but directly hacking the MFT is probably not the best approach. Instead, try the following:

  1. On an NTFS volume (could be a flashdrive, if you format it with NTFS), create one (or a bunch of) compressed files (using Windows). For the sake of these instructions, we'll call these files rescue1, rescue2, etc.
  2. Connect the NTFS drive to your Mac system (without first decompressing the files). Mount the volume read/write. Let's say it's mounted at /mount/ntfs (I don't actually use OS X enough to have memorized where it mounts external volumes).
  3. Using a tool that copies file contents, such as the command-line utility dd, copy the contents of your compressed-without-metadata files into the contents of the compressed files on the NTFS volume. The command would be something like dd if=/path/to/bad/file of=/mount/ntfs/rescue1 bs=4M.
  4. Reconnect the drive to Windows, and see if the compressed files can be opened correctly now. If they can, you can then safely have Windows decompress them (either in-place, or by copying to a drive that doesn't support compression like a FAT32 flashdrive).
  5. If this approach works, you can use it to rescue all of your hundreds of files. Just create as many compressed files on the NTFS drive as you need, name them in whatever manner seems best (could be the original names), and copy the contents from the Mac.

Note that you don't have to compress the whole NTFS volume; that just causes all directories to inherit the "Compressed" flag, and each file inherits it from the file's directory. It wouldn't hurt to do so, though.

I do recommend using a backed-up or throw-away NTFS volume, though; if the OS X NTFS driver is that bad, it may corrupt the MFT when you try writing to a Windows volume.

More complicated alternate approach, if the above doesn't work:

  1. Create a bunch of compressed files on Windows. The hard part is that they need to match the sizes of the bad files. If you want to rescue a file that is 30913 bytes in size, you will need your compressed NTFS file to be that size after compression. I'll be honest; I'm not sure how to arrange that. At a minimum, make the rescue files at least the size (on disk) as the files that need rescuing. Making the size match within 4k is best, as that's the default size of an NTFS cluster (the allocation chunks used for file data).
  2. Use the fsutil utility in Windows to get the extents of the rescue files. The extents are the actual offsets on the volume where the file's data is stored in the partition.
    • The command is fsutil file queryextents <filename> and depending on the location, you may need to run it as admin.
    • The output of the command looks something like this: VCN: 0x0 Clusters: 0x2 LCN: 0x48000. What that says is that file occupies two clusters, logical cluster number 0x48000 (which is offset 4096*0x48000=1207959552) to volume offset 0x48001, which is 8k (2 x 4k/cluster) total space. In practice the end of that is usually unused space.
    • There may be multiple lines of output (this happens when a file is fragmented; it's unlikely to happen on a freshly-formatted volume if you size each file before creating the next one). The first value (the virtual cluster number) in subsequent lines will not be zero, it will be the offset (in clusters) within the file where that extent starts (for example, if it's 0x3, that means that the extent starts 12k into the file).
  3. Disconnect the drive and connect to OS X. Do not mount the volume this time. Instead, find the correct volume identifier. On Linux, this would be something like /dev/sdb1. (Second drive -> sdb1, first partition -> sdb1).
  4. Using dd, copy directly into the raw volume from your bad compressed files.
    • For example, let's say your file has two extents:
      • VCN: 0x0 Clusters: 0x15 LCN: 0x13c
      • VCN: 0x15 Clusters: 0x3f6 LCN: 0xab20
    • You would use the following commands to dd, with file names adjusted as needed. See the dd manpage for more info, but for now note that all parameters to dd are conveniently in multiples of the block size (bs param) but that hex values have been converted to decimal (not sure if dd can handle hex correctly; some dd-like programs can but not all):
      • dd if=/path/to/bad/file of=/dev/sdb1 bs=4K count=21 seek=316
      • dd if=/bath/to/bad/file of=/dev/sdb1 bs=4k count=1014 seek=337 skip=21
  5. Once you've copied the file data to the raw partition, you should be able to connect the drive to Windows and read the compressed (rescued) files. If the file sizes didn't match exactly then you may find that there's a little garbage at the end of the files, but hopefully that's not going to be a problem.

The other approach (possibly easier, even much easier) is to find the routine in ntfs-3g that handles NTFS decompression. Run that routine on the files directly. Alternatively, see if you can find a copy of the ntfsdiskedit utility (seems to have been discontinued) and see if you can use it to manually set the "Compressed" bit on the file after copying them (as uncompressed files) to an NTFS volume.

  • Thanks very much for the detailed suggestion. I'll give it a try, with fingers crossed. BTW - when I originally copied the compressed files from the external drive, I was using the native NTFS driver of OS X, which is read-only. OS X can perform read-write operations on FAT and FAT32, but read-only on NTFS, unless using third-party drivers. – GMason Oct 15 '15 at 0:53
  • See if you can force the built-in drive to mount R/W. If you use ntfs-3g, which is the driver that Linux uses, it will probably respect the "Compressed" attribute and try re-compressing the data you write to it. – CBHacking Oct 15 '15 at 0:57
  • I'll give it a try. Thanks again - I really appreciate your suggestions. – GMason Oct 15 '15 at 0:59
  • When I examine the "corrupted photo" files with a hex editor, I realize that the "bad" files are actually void files - 00's from the first to the last byte of the file. Even though filenames and file sizes were correct, the content of the files are all zero's. Something obviously went terribly wrong during the original process of copying the files from an NTFS drive to an OS X formatted drive. My assumption of a file compression issue was misguided. Looks like the issue was related to the file copy process itself. Thanks for all your time, attention, and detailed suggestions on a fix. – GMason Oct 16 '15 at 8:49
  • Ouch. I'm really sorry to hear that. Something should have occurred to me earlier, though: how did you MOVE files off of NTFS if the OS X NFTS driver is read-only? Copy and then delete the originals using Windows, or reformat the drive? – CBHacking Oct 16 '15 at 20:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.