Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the duplication. (I've looked at this question, but all the one I've looked at from the list seem to only do byte-count comparison...)

Are there any (Windows) dedup applications that can do a similarity-match and point the user to the files in question for examination? Freeware is good, free trial is acceptable. Even just a list of similarities to tell me where to look would probably work.

EDIT: Sorry, I should have mentioned; these are text-based files, primarily DOC, PPT and PDF. The most likely thing to have changed is the content, but formatting might differ as well. Even just picking up on text changes would probably be helpful though...

share|improve this question
I am also looking for a duplicate image finder - since I edited lots of pictures I clicked and my album from 2 years ago is a mess. – KalEl Sep 22 '10 at 5:19
Knowing the types of files would be really helpful in this case, as would the types of changes between near duplicates. Are you talking about sequential documents (text or word processing files) having changes made to their beginning, end or throughout? Or are you talking about image files (JPG, PNG, etc.) that have had local changes (crops, paint, text overlays, etc) or global changes (sharpening, blurring, contrast, etc.). Because you are looking for similar data, not identical data, the method depends heavily on the type of data and the expected differences between near duplicates. – Mike Fitzpatrick Sep 22 '10 at 5:23
@Mike Updated in response. – Margaret Sep 22 '10 at 5:28
@Margaret: DOC, PPT and PDF files are actually BINARY and not TEXT. and all these filetypes have to be rendered and then visually compared to see if they are "close enough". – akira Sep 22 '10 at 7:11
@Margaret: As @akira mentioned, these are mostly binary files. I am not aware of any package that will do what you want across a range of file formats. I would personally tackle this with a combination of command line utilities to extract strings from the binary files and then try to do some basic statistical analysis. But the approach would be different per file type (DOC, PPT, etc.) and I would ONLY go this route if I couldn't do it manually in the space of a few hours. I hope someone else can provide a pointer to an off-the-shelf solution. – Mike Fitzpatrick Sep 22 '10 at 9:11

You could try a plagiarism detector. Plagiarism and updates don't present exactly the same kind of similarities, so it may or may not give useful results, but there are a lot to choose from, so if one doesn't help, another might. I don't have a particular program to recommend; you could try to ask any teacher or professor you know (preferably outside computer science since they're more likely to be familiar with programming plagiarism than with natural language plagiarism).

share|improve this answer

I don't know of any applications, but if most of the content is the same between versions you could do a Windows Search on the directory with the "word or phrase in the file" option. Your query would be a particular phrase that doesn't change (or at least that you don't think changes) much between versions and is fairly unique to that particular document/set of documents. This type of search should work for PDF, DOC, and PPT despite the fact that they are not straight text files. This won't get you the exact output you're looking for, but if you choose your search phrase well and your content between versions doesn't vary wildly it should work pretty well.

share|improve this answer
I was doing this for a bit, but the problem is that there are about 1000 files, of which maybe 100 are duplicates - and doing manual check for each file was excruciating... :S – Margaret Sep 24 '10 at 0:34
Ugh, sounds terrible. I'd look into the plagiarism detector idea since even if you could automate the search process, you'd still have to come up with a way to generate the search phrase and generating the search phrase is basically the plagiarism detection problem itself. – Littleman Sep 25 '10 at 18:48

Look for ssdeep and sdhash.

I've never tried sdhash but I read it's better than ssdeep. Anyway, both provide a CLI that allows the computation of fuzzy hashes and their respective similarities.

Should work fairly well for your goal.

PS: Sorry for the brevity and the lack of links but I'm mobile ATM.

share|improve this answer

protected by Community Dec 28 '15 at 15:17

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.