1

I'm reorganizing some files on my harddrive and have an old copy of a folder, what I want to do is to ensure that every file in folder A (the old copy) exists in folder B. Each folder contains at least 200 files, most of which are likely duplicates.

Caveat: I can't just check the filenames, I must also ensure that they have the same content (à la fdupes).

I checked the manpage for fdupes if there was some way to do the inverse operation (i.e. figure out which files don't have any copies) but came up empty handed, even if the functionality would exist I still have files that exist only in folder B.

These two questions were similar but were Windows only so not much help:

Thanks in advance.

2
  • I don't have fdupes installed. Does it produce a list of duplicate files?
    – choroba
    Dec 3, 2015 at 11:37
  • @choroba Yes :)
    – erb
    Dec 3, 2015 at 11:52

1 Answer 1

1

If you already have a list of duplicate files, just compare it to the list of all files to find the single ones:

fdupes -r dir1 dir2 | grep -vwFf- <(find dir1 dir2 -type f)
  • -w only mathces whole words (not substrings, so file abcd doesn't get matched by abc)
  • -F consideres the patterns to be literal, not regular expressions
  • -f uses the specified file (- in this case, i.e. standard input) as a list of expressions to match against.
  • -v as you probably know, shows the nonmatching lines.

This should output a list contains the files only found in one of the folders, it should be easy to verify that they are all in folder B by skimming or grepping for the name of folder A (below called dir1) like this:

fdupes -r dir1 dir2 | grep -vwFf- <(find dir1 dir2 -type f) | grep -F dir1
2
  • Worked wonders, just had to add the -r flag to fdupes, edited the post so that it recurses. Thanks!
    – erb
    Dec 3, 2015 at 12:20
  • Using duplicate scanners will generally not work as well as using rsync -rcnv for this. The options specify recursion (follow subdirectories), verify files by checksum rather than name/date, "dry run" (don't actually do anything), and verbose (show detailed results). Filter out directories by piping to | grep -v '/$' and you should not see any of your files listed in the output if everything is identical between directories. jdupes has a -I/--isolate option for situations like this, but it doesn't currently work. Jul 10, 2020 at 20:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .