10

There's a great tool, fdupes, for finding duplicate files across two (or more) directories.

I'm looking for a simple tool/command that can output the complementary set - the paths of those files that do not have a duplicate.

5 Answers 5

11
find DIR1 DIR2 -type f -exec sha1sum '{}' \+ | sort | \
    uniq -c --check-chars 40 | egrep '^ *1 ' | cut -c 51-
1
  • 1
    Minor modification to use uniq functionality instead of grep: find DIR1 DIR2 -type f -exec sha1sum {} + | sort | uniq -u --check-chars 40 | cut -c 43- May 20, 2012 at 11:47
5

jdupes gained the option you're looking for in June 2020, available in v1.17.0 or higher.

Try this (-r = recurse, -u = only print files that didn't match any other files, a.k.a. "uniques"):

jdupes -ru dir1/ dir2/
1

I know this is a Linux area, however, I stumbled across this while doing a search for an answer for OSX under 10.15.

However,i f you are a Mac user and your OSX version is less than OSX 10.15 (such as Mojave or less) then you can use File Buddy 10.

I do not know if it will work with 10.15. File Buddy is the only reason I have stuck with 10.14.6. Although I will probably make a dual boot for 10.15 and 10.14.6 so I can keep it and a few games going.

This little program has all sorts of useful functions. One of them is to find identical or unique files across any volumes or directories that you can throw at it. I have included a little pic with some of the unique file search options displayed:

File Buddy Search Options

0
0

As an alternative that will keep the partial matching of jdupes (much faster than doing a full hash)

jdupes -r -T -T "$DIR1" "$DIR2" | awk -v p1="$DIR1" -v p2="$DIR2" '$0 ~ p1 && $0 ~ p2'  RS="\n\n" ORS="\n\n" | sort > dupes
find "$DIR1 $DIR2" -type f | sort > all
comm -23 all dupes

I create two lists: one with duplicated files one with all files. The awk statement ensures that the dupes files only contains files that appear both in $DIR1 AND $DIR2 (--isolate does something else apparently)

Then I use the comm utility to compare them for files that are unique to only the all list (not the dupes list). The only caveat is that jdupes only checks the first 4K of each file, which may not be enough to ensure a unique match, so for personal use I bumped that up to 1MB and it still runs very fast.

Note: If you only want files unique to $DIR1 then remove the $DIR2 search in the find command.

1
  • jdupes now has (as of 2020-06-11) a -u/--printunique option which outputs ONLY a list of files that had no matches after all files are scanned. I do not recommend matching against only the first X bytes of a file. The reason the -T switch requires doubling is because it's extremely dangerous; please avoid recommending its use as it can easily lead to data loss if combined with any destructive file operation. Jul 10, 2020 at 20:30
0

I once had the same problem with finding these unique files and I did NOT want to checksum them (because they were too large and too many), so I wrote a script based on the filename and filesize:

isolated-files.py --source folder1 --target folder2

This will show any files (recursively) within folder2 which are not in folder1 (also recursively). It can also be used on SSH connections and with multiple folders.

see https://github.com/ezzra/isolated-files

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .