I'm trying to compare 2 volumes, which mainly have the same content, but in completely different folder structures. (Around 2 million files.)

I want to see which files are ONLY in 1 folder, not the other -- based on size & name.

I tried this:

find /Volumes/1/ /Volumes/2/ -not -empty -type f -printf "%D\t%s\t%f\n" | sort -t$'\t' -k2 -n | uniq -uf1 | sort

The idea is to get a list of all files, with device, size, and name, then remove all the duplicates.

... but it's not working. I still get files common to both drives in my result.

Ideally I'd also want to include the full path in the results list, but I don't know how to include that without messing up the sorting/uniq-ing.

  • Do you actually trust two files to be identical if they share name and size, or do you also require md5/sha1/sha2 verification? – MariusMatutiae Feb 2 '14 at 9:13
  • I trust ... since one drive was recently rsynced from the other, but now there's a bit of a mess left over. Just looking for scraps left over that need to be copied. – Ze'ev Feb 2 '14 at 9:15

EDIT: answer regarding the need of preserving the device

find dir1/ dir2/ -type f -printf "%D\t%s\t%f\n" | \
    sort -t$'\t' -k2 | \
    uniq -uf2

find /Volumes/1/ /Volumes/2/ -type f -printf "%f\n" | sort | uniq -u will show files missing only by filename

find /Volumes/1/ /Volumes/2/ -type f -printf "%s-%f\n" | sort | uniq -u will show files missing by matching filename and size

The second option will output files twice if there is a size mismatch. If that's not what you want, you can refilter the list like this:

find /Volumes/1/ /Volumes/2/ -type f -printf "%s\t%f\n" | sort | uniq -u | \
    cut -d$'\t' -f 2|sort|uniq
  • Yes, but then I don't know which drive the files came from. That's basically what I had, without the %D for device ... I need the results to include info about which volume the file resides on. – Ze'ev Feb 4 '14 at 6:30
  • 1
    @Ze'ev check my edited answer. I wasn't clear wether you needed the duplicate result in case of size mismatch. – GnP Feb 4 '14 at 15:12

Use rsync with -R (relative option to show full path)

rsync -rvcnR --delete path_to_sync/ path_to_orig/

The other command line switches r, v, c and n tell rsync (check the man page for details) to perform a verbose, recursive, checksum-based synchronization of the two directories, but only for show: -n

Edit to allow for the deletion and movement of files which caused the corruption of file structure after the initial rsync which the originator clarified after my answer.

change your find to something like this:

find /Volume/1/ /Volume/2/ -printf '%f %s\t%D%p\n' | sort -uk1,1 | cut -f 2- -d ' '

Use printf to create the first field using file-name followed by the fields you require, sort by this first field for unique files then delete field 1 leaving the unique files. If you use space es in file names you would need to use a different delimiter to the space I have used for the cut separator.

  • Does this really work? OP claims files excist bt in completely different folders. – MariusMatutiae Feb 2 '14 at 11:34
  • @MariusMatutiae if the files were put there via rsync so therefore they can be checked via rsync. – Antony Feb 2 '14 at 11:41
  • They were subsequently moved. I had begun to delete apparent duplicates, moving them to the Trash, then later discovered some were wrongly identified, so now all files on first volume are in one big folder. I need an 'rsync' that ignores folder structure. Beyond Compare can do it, but it chokes on a job this big. – Ze'ev Feb 2 '14 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.