There's a great tool, fdupes, for finding duplicate files across two (or more) directories.
I'm looking for a simple tool/command that can output the complementary set - the paths of those files that do not have a duplicate.
find DIR1 DIR2 -type f -exec sha1sum '{}' \+ | sort | \
uniq -c --check-chars 40 | egrep '^ *1 ' | cut -c 51-
uniq
functionality instead of grep
: find DIR1 DIR2 -type f -exec sha1sum {} + | sort | uniq -u --check-chars 40 | cut -c 43-
May 20, 2012 at 11:47
I know this is a Linux area, however, I stumbled across this while doing a search for an answer for OSX under 10.15.
However,i f you are a Mac user and your OSX version is less than OSX 10.15 (such as Mojave or less) then you can use File Buddy 10.
I do not know if it will work with 10.15. File Buddy is the only reason I have stuck with 10.14.6. Although I will probably make a dual boot for 10.15 and 10.14.6 so I can keep it and a few games going.
This little program has all sorts of useful functions. One of them is to find identical or unique files across any volumes or directories that you can throw at it. I have included a little pic with some of the unique file search options displayed:
As an alternative that will keep the partial matching of jdupes (much faster than doing a full hash)
jdupes -r -T -T "$DIR1" "$DIR2" | awk -v p1="$DIR1" -v p2="$DIR2" '$0 ~ p1 && $0 ~ p2' RS="\n\n" ORS="\n\n" | sort > dupes
find "$DIR1 $DIR2" -type f | sort > all
comm -23 all dupes
I create two lists: one with duplicated files one with all files. The awk statement ensures that the dupes files only contains files that appear both in $DIR1 AND $DIR2 (--isolate
does something else apparently)
Then I use the comm
utility to compare them for files that are unique to only the all
list (not the dupes list). The only caveat is that jdupes only checks the first 4K of each file, which may not be enough to ensure a unique match, so for personal use I bumped that up to 1MB and it still runs very fast.
Note: If you only want files unique to $DIR1 then remove the $DIR2 search in the find command.
-u/--printunique
option which outputs ONLY a list of files that had no matches after all files are scanned. I do not recommend matching against only the first X bytes of a file. The reason the -T
switch requires doubling is because it's extremely dangerous; please avoid recommending its use as it can easily lead to data loss if combined with any destructive file operation.
Jul 10, 2020 at 20:30
I once had the same problem with finding these unique files and I did NOT want to checksum them (because they were too large and too many), so I wrote a script based on the filename and filesize:
isolated-files.py --source folder1 --target folder2
This will show any files (recursively) within folder2 which are not in folder1 (also recursively). It can also be used on SSH connections and with multiple folders.