41

I have two identically sized files, and I need to do a binary diff to check whether they're the same.

I'm currently runnnig diff file1.img file2.img but it's taking quite a while to process my 4 GB files. Is this the most efficient way to do this?

Improve this question

5 Answers 5

Reset to default
47

cmp is designed to find differences in binary files. You might also try checksumming (sum) and compare the hashes.

Improve this answer
21

One of the most common ways of determining if two files are identical (assuming their sizes match) is using a program to create a "hash" (essentially a fingerprint) of a file. The most common ones are md5sum and sha1sum.

For example:

$ md5sum file1 file2
e0e7485b678a538c2815132de7f9e878  file1
4a14aace18d472709ccae3910af55955  file2

If you have many files that you need to check, for example if you are transferring a directory full of files from one system to another, you can redirect the output from the original system to a file, then md5sum/sha1sum can automatically use that file to tell you which files are different:

$ md5sum file1 file2 > MD5SUMS
... copy file1, file2, MD5SUMS across
$ md5sum --check MD5SUMS
file1: OK
file2: OK
Improve this answer
4
  • 2
    MD5 is not always a reliable for this: digg.com/security/…
    – Jon Cage
    Commented Apr 29, 2010 at 12:30
  • 28
    Actually, MD5 is reliable for basic integrity checks. It just isn't considered as cryptographically strong as we once thought. If you are worried about hackers, don't use MD5, but if you just want to know if some files copied from a CD got corrupted, or if your compiler spits out identical files every time, MD5 is more than adequate.
    – Adam Batkin
    Commented Apr 29, 2010 at 12:55
  • This is a needlessly complex solution, diff file_A file_B would work, even with binary files.
    – Edwin Buck
    Commented Jun 22, 2020 at 14:35
  • @EdwinBuck Sure, if you happen to have the two files right next to each other and you only care about a one-off check then diff is a little less work. But for all the other situations that one might encounter, checksums/hashes are waaaay more flexible
    – Adam Batkin
    Commented Jun 23, 2020 at 16:46
17

Found a solution - the cmp tool which comes with most Linux flavours.

Improve this answer
3

If I just want to know if they are the same, I prefer to use sha1sum if it's available, or md5 as a fallback.

If I want to know how different they are, or where they're different, one thing that works is to crank them both through od ('octal dump', which usually has a hex option) to make temporary files and then diff those.

Improve this answer
2
  • 2
    If you want to know whether they are two files are the same, I don't think using sha1sum (or md5sum for that matter) can be more efficient than just diff (as this was the original question), because even if the two (large) files differ at the very beginning), you will read both of them entirely before knowing they differ.
    – Pierre
    Commented Apr 10, 2014 at 21:29
  • @Pierre BUT, hashing and crypto-signing works across remote devices.
    – VasyaNovikov
    Commented Dec 15, 2015 at 17:26
1

I just ran some benchmarks on a 100+ MB file. diff was the fastest, while cmp came second, and using md5sum came in last.

# time diff file1 file2; echo $?

real    0m0.122s
user    0m0.009s
sys 0m0.113s
0
# time cmp file1 file2; echo $?

real    0m0.213s
user    0m0.097s
sys     0m0.117s
0
# time md5sum file1 > /tmp/test; time md5sum file2 > /tmp/test2; diff /tmp/test /tmp/test2; echo $?

real    0m0.279s
user    0m0.211s
sys     0m0.066s

real    0m0.316s
user    0m0.225s
sys     0m0.073s
0

I reran the exercise with a 4.3 GB file, and had to delete and recreate the file with dd since RAM caching was greatly affecting the results.

$time diff file1 file2; echo $?

real    0m19.325s
user    0m0.995s
sys 0m5.280s
0

$time cmp file1 file2; echo $?

real    0m36.106s
user    0m4.399s
sys 0m6.147s
0

$time md5sum file1 > /tmp/test; time md5sum file2 > /tmp/test2; diff /tmp/test /tmp/test2; echo $?

real    0m10.441s
user    0m8.054s
sys 0m2.357s

real    0m24.682s
user    0m8.027s
sys 0m3.442s
0

Based on these results I would recommend moving the files to a RAMFS mount, and sticking with diff.

Improve this answer
5
  • I like that you actually did a benchmark, but 100 Mb is not representative of the OP's case. 1,000Mb would be much better.
    – jpaugh
    Commented Sep 11, 2018 at 18:48
  • 1
    agreed which is why i ran the 4.3 gig tests a few months later. It actually took effort to bypass the OS caching.
    – Michael C. Chen
    Commented Sep 11, 2018 at 21:40
  • I imagine it does. I apologize for not reading reading the middle of your answer. (I still thought it was good enough to upvote, even with only seeing the first benchmark.) FWIW, there's some kernel magic for disabling file caching. I'd have to do the benchmark myself to see which ones actually work or are necessary.
    – jpaugh
    Commented Sep 12, 2018 at 14:11
  • thanks for the link on kernel magic, very helpful. And thought about this some more, and the best method would depend on what is being done. e.g. like someone mentioned if the files are on different machines, hashing and then comparing would be better. If the same file needs to be checked against other files multiple times, hashing would also win. The only thing where diff would win is when you have both files on the same system locally and you just need to compare once.
    – Michael C. Chen
    Commented Aug 12, 2020 at 19:26
  • It's really pleasant to think that every addition I make to the world can come back to me, years later. :-) May you have the fruits of your labor, also.
    – jpaugh
    Commented Aug 26, 2020 at 13:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .