5

There's an existing question similar to this that didn't get answered in the proper context (a massive volume of data to compare) at all, so I'm trying again here.

I'm using SuperDuper to maintain a cloned copy of my Mac's internal hard drive on an external drive. Due to data integrity measures built into that program, it's possible for your destination volume to require massively more space than the source drive, but that's beside the point.

What I'm looking to do is run a "folder" comparison at the volume level. At its crudest, I want to essentially diff -r /Volumes/MacintoshHD /Volumes/BackupDrive and have a decent navigation interface to work with the results. The big problem with this is that these are 1TB disks containing about 800 GB of data and on the order of 3.1 million individual files. I've tried many diff tools, and none of them seem to be able to cope with such a huge amount of data. Here's a brief list of applications that totally and utterly fail at this:

  • diff (command line). Since it streams its results to the console, this is more "reliable" than the others, but it produces an overwhelming volume of unusable text output (there's no human-friendly interface for "browsing" for the differences.)

  • FileMerge (from Apple's Dev Tools). With smaller file sets the interface works great, but on this huge scale it will hang for many hours trying to scan and eventually crash before finishing.

  • Kaleidoscope. After many hours of scanning without an accurate progress bar, it will eventually show an A/B comparison window, but will randomly drop the 'B' side completely if you try to navigate at all, forcing you to start over. (Which is pointless because it will just drop the B side again.)

  • Araxis Merge. Came the closest out of everything I tried. It did produce a comparison eventually, but was significantly slower both to generate it and to navigate it, to the point of being intolerable. When you're talking about 18 hours of scanning time before you can start working with the data set, and 20-40 second hangs every time to try to change directories in the diff, speed and stability make a huge difference.

So my question is: Are there any diff tools out there tailored specifically to massive comparisons? How do people "normally" compare large data sets measured in triple-digit-gigabytes? Are there perhaps forensic tools my searches haven't found? In particular, I'm only concerned about the differences, not the identical portions. I expect there to be a relatively small percentage of change, maybe 1-5%, so a tool that "throws away" the similarities would have a huge advantage with a dataset this large.

1
  • I'm curious how Beyond Compare folder level diff would behave in this scenario. I'm not sure how it would scale, it probably doesn't have UI virtualization and all these other to scale features. I don't know I've never put it to the test but I'm curious now. The individual file diffs are designed per file extension as to what diff rendering is put forth. I believe before it does full binary content diffs it does crc diffs at first pass. This is all in the realm content comparisons scootersoftware.com/v4help/index.html?opcompare.html
    – jxramos
    Aug 12, 2022 at 9:25

2 Answers 2

0

DupScan (for Mac) will find all the duplicate files. It has many options for how to compare files, checksum being one.

DupScan result is a list of files with the number of duplicates and an easy way to remove them.

Of course... whatever you do will take hours.

1
  • I should have clarified: I'm expecting there to be a large percentage of duplicates. In my case, I'm not concerned with what is the same, I'm concerned with what is different. There may only be a couple thousand files that are not duplicates in the entire comparison-- those are the files I want to review.
    – beporter
    Jun 17, 2014 at 14:15
0

Your question and my curiosity about Beyond Compare's tooling in this space lead me to this

Yes, you can compare hard drives on two different computers using a Beyond Compare snapshot.

https://forum.scootersoftware.com/forum/beyond-compare-4-discussion/general/14563-comparing-two-computers-hard-drives

Looks like a great feature, sort of like a manifest document for your hard drive.

https://www.scootersoftware.com/v4help/index.html?snapshots.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .