I'm working on OSX and I have multiple CSV files in a directory. I want to check if any two of them are identical.

I've tried:

cmp file1.csv file2.csv || echo "files are different"

This tells me that the first two files are different.

I've also tried:

 diff -q --from-file *.csv

This tells me that the first file is different from every other file.

But how can I check this for every file, without having to do it manually?


The following works for me in Linux. I do not have a Mac Terminal to test if this works there.

$ md5sum * | sort | uniq --check-chars=32 --group
36da49e13da596245e85e9e47ab8a04f *y

8c3c2e6aa3be26d9f1bbbcd9a401e88e *x
8c3c2e6aa3be26d9f1bbbcd9a401e88e *z

The program uniq requires sorted input. Check its man page for other output options.

  • What version of uniq are you using? I cannot find one that has the --group option. – hymie May 29 '15 at 19:10
  • uniq (GNU coreutils) 8.23 – Steven May 29 '15 at 19:15
  • The most recent I've seen in production is v8.21. However that is not a big deal since the -d option does almost the same thing – Davide Feb 7 '17 at 23:04

In addition to Steven's answer, you should enter this into the terminal:

shasum * | sort | uniq 

And that should provide you the results.


What you really want is a nice shell script. A brain dead shell script would be:

for i in `ls`; do diff -sq --from-file=$i `ls` 2> /dev/null; done | grep identical

The problem with this is, of course, that it checks each file against every other file, including itself, even if the check has already been made. I'm sure there is a nice awk script that will do the right thing in even less space but I am not feeling very awk-ish right now.

Other options to diff that you might like to consider are -w if white space is of no consequence, -b if the presence of white space is important but you really don't care how much or what kind is there, --strip-trailing-cr if some of the files were made on a system that thinks it needs both a new line and a carriage return.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.