2

I just rsync-ed 2,000,000 files (3TB) from one RAID to another.

I want to make sure my data is intact.

rsync -c takes a really long time.

diff doesn't show me what it's doing.

Is there an alternative that's (a) faster, and (b) will show me progress while it's comparing?

(I'm on Mac, and brew search diff gives me apgdiff colordiff diffstat diffutils fmdiff libxdiff open-vcdiff podiff rfcdiff vbindiff bsdiff diffpdf diffuse dwdiff kdiff3 ndiff perceptualdiff rdiff-backup tkdiff wdiff ... would one of these do the job?)

6
  • Duplicate of superuser.com/questions/708001/…. Keep questions to 1 exchange site.
    – spuder
    Jan 28, 2014 at 4:44
  • Calculate the sha1sums for both sets of files and then diff the chucksums?
    – Zoredache
    Jan 28, 2014 at 5:20
  • I agree with Zoredache. Using sha1sum or md5sum to compute unique file hashs on both hosts and then using diff to compare the result is the less network heavy solution. Although i believe it will not be much faster/better than rsync -c.
    – Biapy
    Jan 28, 2014 at 6:07
  • 2
    Please do not crosspost.
    – Marco
    Jan 28, 2014 at 9:43
  • 1
    Duplicate of unix.stackexchange.com/questions/111251/… which has more answers
    – pd12
    Apr 15, 2016 at 15:09

1 Answer 1

2

cmp is the tool you're probably looking for: it compares two files byte by byte, reading one block of each at a time. As soon as it finds a difference, it bails out. It's hence both faster (doesn't require to read whole files when they differ) and more reliable (it actually performs byte by byte comparison) than checksumming. It's also faster than diff as it doesn't require complex compuation to show you where the files differ.

You could easily write a small script to recurse over your two directories and output progress based on the amount of files (or their cumulative size) processed.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .