I have two identically sized files, and I need to do a binary diff to check whether they're the same.

I'm currently runnnig diff file1.img file2.img but it's taking quite a while to process my 4 GB files. Is this the most efficient way to do this?

cmp is designed to find differences in binary files. You might also try checksumming (sum) and compare the hashes.

One of the most common ways of determining if two files are identical (assuming their sizes match) is using a program to create a "hash" (essentially a fingerprint) of a file. The most common ones are md5sum and sha1sum.

For example:

$ md5sum file1 file2
e0e7485b678a538c2815132de7f9e878  file1
4a14aace18d472709ccae3910af55955  file2

If you have many files that you need to check, for example if you are transferring a directory full of files from one system to another, you can redirect the output from the original system to a file, then md5sum/sha1sum can automatically use that file to tell you which files are different:

$ md5sum file1 file2 > MD5SUMS
... copy file1, file2, MD5SUMS across
$ md5sum --check MD5SUMS
file1: OK
file2: OK
  • 2
    MD5 is not always a reliable for this: digg.com/security/… – Jon Cage Apr 29 '10 at 12:30
  • 27
    Actually, MD5 is reliable for basic integrity checks. It just isn't considered as cryptographically strong as we once thought. If you are worried about hackers, don't use MD5, but if you just want to know if some files copied from a CD got corrupted, or if your compiler spits out identical files every time, MD5 is more than adequate. – Adam Batkin Apr 29 '10 at 12:55
up vote 16 down vote accepted

Found a solution - the cmp tool which comes with most Linux flavours.

If I just want to know if they are the same, I prefer to use sha1sum if it's available, or md5 as a fallback.

If I want to know how different they are, or where they're different, one thing that works is to crank them both through od ('octal dump', which usually has a hex option) to make temporary files and then diff those.

  • 2
    If you want to know whether they are two files are the same, I don't think using sha1sum (or md5sum for that matter) can be more efficient than just diff (as this was the original question), because even if the two (large) files differ at the very beginning), you will read both of them entirely before knowing they differ. – Pierre Apr 10 '14 at 21:29
  • @Pierre BUT, hashing and crypto-signing works across remote devices. – VasyaNovikov Dec 15 '15 at 17:26

I just ran some benchmarks on a 100+ MB file. diff was the fastest, while cmp came second, and using md5sum came in last.

# time diff file1 file2; echo $?

real    0m0.122s
user    0m0.009s
sys 0m0.113s
0
# time cmp file1 file2; echo $?

real    0m0.213s
user    0m0.097s
sys     0m0.117s
0
# time md5sum file1 > /tmp/test; time md5sum file2 > /tmp/test2; diff /tmp/test /tmp/test2; echo $?

real    0m0.279s
user    0m0.211s
sys     0m0.066s

real    0m0.316s
user    0m0.225s
sys     0m0.073s
0

I reran the exercise with a 4.3 GB file, and had to delete and recreate the file with dd since RAM caching was greatly affecting the results.

$time diff file1 file2; echo $?

real    0m19.325s
user    0m0.995s
sys 0m5.280s
0

$time cmp file1 file2; echo $?

real    0m36.106s
user    0m4.399s
sys 0m6.147s
0

$time md5sum file1 > /tmp/test; time md5sum file2 > /tmp/test2; diff /tmp/test /tmp/test2; echo $?

real    0m10.441s
user    0m8.054s
sys 0m2.357s

real    0m24.682s
user    0m8.027s
sys 0m3.442s
0

Based on these results I would recommend moving the files to a RAMFS mount, and sticking with diff.

  • I like that you actually did a benchmark, but 100 Mb is not representative of the OP's case. 1,000Mb would be much better. – jpaugh Sep 11 at 18:48
  • 1
    agreed which is why i ran the 4.3 gig tests a few months later. It actually took effort to bypass the OS caching. – forbidder Sep 11 at 21:40
  • I imagine it does. I apologize for not reading reading the middle of your answer. (I still thought it was good enough to upvote, even with only seeing the first benchmark.) FWIW, there's some kernel magic for disabling file caching. I'd have to do the benchmark myself to see which ones actually work or are necessary. – jpaugh Sep 12 at 14:11

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.