I've merged multiple backups together of all my photography files and came across multiple copies of many files. Not all files are copies and some are photoshopped ones. Now I noticed 'fdupes' that can help me identify duplicates and delete each.

Now there is one issue..

Let's say I've got this folder structure:

/Photoshoot X copy 1/*
/Photoshoot X copy 1/selection/*

/Photoshoot X copy 2/*
/Photoshoot X copy 2/selection/*

Many, if not all, files in copy 2 are duplicates of those in copy 1. Those can be deleted. However! Some, if not all, files inside the folder selection are direct duplicates of those found in the root folder. Those should not be deleted, even though they are duplicates. They should not be deleted because, even though they are duplicates, they are copied because these are the important ones out of the whole collection.

Now my question: is it possible to batch compare files relative to it's path?

So only both root folders are compared, not the root folder compared to the content of the folder selection.

  • Would a workflow like moving the whole /Photoshoot X copy 2/selection/* directory some place protected, or archiving it someplace, running fdupes, and then restoring the protected copies work for you? – infixed Mar 24 '16 at 14:42
  • I already was thinking of such method. I would do it differently. Copy all relative content to a work folder and run fdupes on that. – Sander Schaeffer Mar 24 '16 at 14:44
  • Oops. I pressed enter too quickly. However.. I've got about a thousand separate sets of content to scan.. So it will be quite a task. Therefor I would like a bit more optimal suggestion/work flow.. – Sander Schaeffer Mar 24 '16 at 14:44
  • Well, on linux I could get a list of (probably) duplicated files with something like find basedirectory -type f -exec md5sum {} \; | grep -v "/selection/" | sort> list.txt, and look for lines with duplicate md5's. (probably with a perl script) But I don't think that's your environment – infixed Mar 24 '16 at 14:55
  • All data is stored on a Synology NAS which as far as I know it's os is based on Linux. I've got access to both a Mac and Windows machine directly connected to the NAS. So you tell me what I need :p :) – Sander Schaeffer Mar 24 '16 at 14:57

In a Linux environment one could install fdupes

Step 1, get a list of duplicate files

fdupes -f basedirectory/ > list.txt

Step 2, filter out the duplicates in the /selection/ directories

grep -v "/selection/" list.txt > script.sh

Step 3 make script.sh into a script to delete the files using global search and replace to put a delete command, rm at the start of every line. Use your favorite text editor, or sed if you feel up to it. Check your work here, these are the files that go away. Take care if you have file names with spaces or other special characters in them. You might have to put a rm " at the start of the line and a " at the end to work around that.

perl -p -i -e 's/^(.*)$/rm "\1"/' script.sh

should work, but check out the file afterward before executing to make sure it is what you want.

Step 4 execute your new script. This is pulling the trigger.

sh -x script.sh

Step 5 clean up by getting rid of the temporary files list.txt and script.sh

No doubt one could modify the workflow for windows command line with suitable subsitutions.

  • A quick thought of which I preferred to private message you.. Do you think you can realize a script that only searches the direct content of the folders 'copy 1', 'copy 2', 'copy '3, etc. which resize in some of the folders inside /shoots/; e.g.: '/shoots/modelA/copy 1', '/shoots/modelB/copy 2', '/shoots/modelC/copy 3'. So if random folder inside /shoots/ doesn't contain a folder called 'copy #' it skips the parent folder (e.g. 'modelC'). - And it doesn't scan any deeper but the folder 'copy #'. .. Do you understand? xD See illustration: i.imgur.com/uUgBJ84.jpg only scan folder /w check – Sander Schaeffer Mar 24 '16 at 18:48
  • The reason for this is sometimes a folder doesn't contains a folder called 'copy' and not all additional folders are called 'selection'. So to 'easify' this process, only the folders 'copy #' should be processed. - If this can be realized, please add it as a secondary part of your answer. I couldn't thank you enough. Update: I'll make a new question for this task as this is somewhat different. – Sander Schaeffer Mar 24 '16 at 18:51
  • New question: superuser.com/questions/1056972/… - Big thanks if you are willing to help. – Sander Schaeffer Mar 24 '16 at 19:06

Run the perl script findDups.pl below on your mac (replacing dir_to_scan with the directory to scan for duplicates):

$ findDups.pl dir_to_scan > rmdups.sh

have a final look at the script rmdups.sh before running it with:

$ sh rmdups.sh

# Usage: findDups.pl dir_to_scan
use strict;
use warnings;
use File::Find 'find';
use Digest::MD5 'md5';

my $dir = shift @ARGV || '.';
my %files;

find(\&wanted, $dir);

sub wanted {
  my $localname = $_;
  my $path = $File::Find::name;
  return if $path =~ m#/selection/#;
  return unless -f $localname;
  my $md5 = md5($localname);
  push(@{$files{$md5}}, $path);

while (my($key, $value) = each(%files)) {
  while (@{$files{$key}} > 1) {
    my $doomed = shift @{$files{$key}};
    print 'rm \'' . $doomed . '\'' . $/;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.