Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I try to consolidate old drives to new ones of larger capacity. Sometimes files have been renamed, but are otherwise identical. Sometimes an old directory has just a few more files in it than a newer directory with the same name. Sometimes a file has the same name but the size differs.

So I often find myself asking the question:

Are there any files on this old drive or directory that I haven't already copied to the new drive?

I just want to know that I have the files, I don't want to try and sync stuff automatically (Syncing tools tend to just sync, creating duplicate folder structures and other problems, so I prefer to do it by hand).

Basically, if an old drive has a file called "" ten directories deep, and my new big drive has an identical file called "" in the root, I just want a "yes you have it" or "no, unique files exist".

Is there a free tool, a script or a quick and easy method (Mac/Unix or Windows) to get the answer?

share|improve this question
Python could do this pretty trivially, knowing a scripting language is great for situations like this – Phoshi Mar 30 '10 at 17:06

Duplicate Cleaner does this

You can find files by specifying to search for duplicates by any combination of:

* Same Content (MD5 checksum)
* Same File Name
* Same Size
* Same Date (Note - this is the file's Modified Date)
share|improve this answer

To my knowledge there's no such tool, as it's pretty much a program that would only suite a few people.

Possible solution for this would be to scan everything (well you could define a list of extensions to limit the scan as the list of all files might be huge and would take a lot of unnecessary files) on one drive and create CRC32 checksum for each file. Then on another run the program would have to verify (compare the CRC32 checksum) each file it find on new drive with the list it created earlier and announce missing files.

Maybe if there were enough interest some talented programmer/scripter could fix this to get his budget up and running.

share|improve this answer

A tool for detecting "duplicate files" would be able to produce a report of the files that have been backed up.

share|improve this answer

Easy Duplicate Finder (now commercial, freeware version is no longer available) can do this for you. It'll even export lists of duplicates. Just point it to the source dir and the destination dir.

Might I mention that the only reliable way to do this is via a Hash check (crc32 or md5 or sha1)? Easy Duplicate Finder uses CRC, so you can be sure.

share|improve this answer

For Linux (and maybe other systems, I'm not sure) there is fdupes program. I use it to find duplicate files (but it doesn't list unique files).

In addition, fdupes page at Wikipedia also lists many other similar tools.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .