Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Does anyone of you know how to perform the following task: Suppose you have two directory trees, which I will call source and target. They may have a very different structure, but could contain many duplicate files.

An example would be a structured collection of photographs on one hand (the destination), and just a tmp directory in which you unload everything from your camera on the other (the source).

Maybe some of these files are already in the structured directory tree. I would like to rsync from the source to the target, such that only the files from the source that are not in the destination at any location are copied.

share|improve this question
sounds like you want to compare the contents of two directories recursively. try md5deep, perhaps with a little sorting and a diff. – Sirex Oct 26 '11 at 9:51
Thanks Sirex, I didn't know md5deep. – doetoe Oct 26 '11 at 15:40
Do you want to simply remove duplicates? Because otherwise, where do you want to put the non-dupes that aren't sorted? – Chris2048 Apr 6 '12 at 23:20

Try fdupes:

fdupes is a program written by Adrian Lopez to scan directories for duplicate files, with options to list, delete or replace the files with hardlinks pointing to the duplicate. It first compares file sizes and MD5 signatures, and then performs a byte-by-byte check for verification.

This wikipedia entry also lists similar programs.

share|improve this answer
What is fdupes? What does it do? Why does it work so well or not so well? Please refrain from just adding links, but rather add to it, by adding a description of what it does and why it works. Sometimes it's even good to give instructions on how to use it. – KronoS Apr 7 '12 at 18:51
@KronoS I haven't used fdupes enough to compare it to other similar programs, but a dupe-finding program seems to be what might be needed here, and this page lists many such programs. – Chris2048 Apr 9 '12 at 13:04
Thanks Chirs2048. I'll have a look. – doetoe May 21 '12 at 16:51

Since your example describes photos; Adobe Lightroom does exactly that. It can import pictures into its directory structure while ignoring duplicates. If you want this function for other types of documents you need another solution though.

share|improve this answer
Thanks Mattias. I was looking more for something (lightweight) to operate on existing directory structures though – doetoe May 21 '12 at 16:52

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.