6

I have a folder full of files, with a number of duplicate files in it. Unfortunately, in a number of cases, one version is an updated version of another, so a straight byte-match doesn't locate the duplication. (I've looked at this question, but all the one I've looked at from the list seem to only do byte-count comparison...)

Are there any (Windows) dedup applications that can do a similarity-match and point the user to the files in question for examination? Freeware is good, free trial is acceptable. Even just a list of similarities to tell me where to look would probably work.

EDIT: Sorry, I should have mentioned; these are text-based files, primarily DOC, PPT and PDF. The most likely thing to have changed is the content, but formatting might differ as well. Even just picking up on text changes would probably be helpful though...

6
  • I am also looking for a duplicate image finder - since I edited lots of pictures I clicked and my album from 2 years ago is a mess.
    – KalEl
    Sep 22, 2010 at 5:19
  • Knowing the types of files would be really helpful in this case, as would the types of changes between near duplicates. Are you talking about sequential documents (text or word processing files) having changes made to their beginning, end or throughout? Or are you talking about image files (JPG, PNG, etc.) that have had local changes (crops, paint, text overlays, etc) or global changes (sharpening, blurring, contrast, etc.). Because you are looking for similar data, not identical data, the method depends heavily on the type of data and the expected differences between near duplicates. Sep 22, 2010 at 5:23
  • @Mike Updated in response.
    – Margaret
    Sep 22, 2010 at 5:28
  • @Margaret: DOC, PPT and PDF files are actually BINARY and not TEXT. and all these filetypes have to be rendered and then visually compared to see if they are "close enough".
    – akira
    Sep 22, 2010 at 7:11
  • @Margaret: As @akira mentioned, these are mostly binary files. I am not aware of any package that will do what you want across a range of file formats. I would personally tackle this with a combination of command line utilities to extract strings from the binary files and then try to do some basic statistical analysis. But the approach would be different per file type (DOC, PPT, etc.) and I would ONLY go this route if I couldn't do it manually in the space of a few hours. I hope someone else can provide a pointer to an off-the-shelf solution. Sep 22, 2010 at 9:11

3 Answers 3

3

You could try a plagiarism detector. Plagiarism and updates don't present exactly the same kind of similarities, so it may or may not give useful results, but there are a lot to choose from, so if one doesn't help, another might. I don't have a particular program to recommend; you could try to ask any teacher or professor you know (preferably outside computer science since they're more likely to be familiar with programming plagiarism than with natural language plagiarism).

1

Look for ssdeep and sdhash.

I've never tried sdhash but I read it's better than ssdeep. Anyway, both provide a CLI that allows the computation of fuzzy hashes and their respective similarities.

Should work fairly well for your goal.

PS: Sorry for the brevity and the lack of links but I'm mobile ATM.

1
  • Thanks for the info. But I'm not clear - how do these help? Will the hashes of near-exact files be the same? Jan 2, 2021 at 17:11
0

I don't know of any applications, but if most of the content is the same between versions you could do a Windows Search on the directory with the "word or phrase in the file" option. Your query would be a particular phrase that doesn't change (or at least that you don't think changes) much between versions and is fairly unique to that particular document/set of documents. This type of search should work for PDF, DOC, and PPT despite the fact that they are not straight text files. This won't get you the exact output you're looking for, but if you choose your search phrase well and your content between versions doesn't vary wildly it should work pretty well.

2
  • I was doing this for a bit, but the problem is that there are about 1000 files, of which maybe 100 are duplicates - and doing manual check for each file was excruciating... :S
    – Margaret
    Sep 24, 2010 at 0:34
  • Ugh, sounds terrible. I'd look into the plagiarism detector idea since even if you could automate the search process, you'd still have to come up with a way to generate the search phrase and generating the search phrase is basically the plagiarism detection problem itself.
    – Littleman
    Sep 25, 2010 at 18:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .