There's an existing question similar to this that didn't get answered in the proper context (a massive volume of data to compare) at all, so I'm trying again here.
I'm using SuperDuper to maintain a cloned copy of my Mac's internal hard drive on an external drive. Due to data integrity measures built into that program, it's possible for your destination volume to require massively more space than the source drive, but that's beside the point.
What I'm looking to do is run a "folder" comparison at the volume level. At its crudest, I want to essentially diff -r /Volumes/MacintoshHD /Volumes/BackupDrive
and have a decent navigation interface to work with the results. The big problem with this is that these are 1TB disks containing about 800 GB of data and on the order of 3.1 million individual files. I've tried many diff tools, and none of them seem to be able to cope with such a huge amount of data. Here's a brief list of applications that totally and utterly fail at this:
diff
(command line). Since it streams its results to the console, this is more "reliable" than the others, but it produces an overwhelming volume of unusable text output (there's no human-friendly interface for "browsing" for the differences.)FileMerge (from Apple's Dev Tools). With smaller file sets the interface works great, but on this huge scale it will hang for many hours trying to scan and eventually crash before finishing.
Kaleidoscope. After many hours of scanning without an accurate progress bar, it will eventually show an A/B comparison window, but will randomly drop the 'B' side completely if you try to navigate at all, forcing you to start over. (Which is pointless because it will just drop the B side again.)
Araxis Merge. Came the closest out of everything I tried. It did produce a comparison eventually, but was significantly slower both to generate it and to navigate it, to the point of being intolerable. When you're talking about 18 hours of scanning time before you can start working with the data set, and 20-40 second hangs every time to try to change directories in the diff, speed and stability make a huge difference.
So my question is: Are there any diff tools out there tailored specifically to massive comparisons? How do people "normally" compare large data sets measured in triple-digit-gigabytes? Are there perhaps forensic tools my searches haven't found? In particular, I'm only concerned about the differences, not the identical portions. I expect there to be a relatively small percentage of change, maybe 1-5%, so a tool that "throws away" the similarities would have a huge advantage with a dataset this large.