4

Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?

Improve this question

3 Answers 3

Reset to default
3

You can try Near Duplicates Finder. It is not exactly what you are asking for, but it is Java based, so it will work on any platform where Java works (including Windows, OS X and Linux). Is has no GUI interface, but it will find near duplicates for text based files, like TXT, DOC, HTML, PDF, etc.

Example Usage

Near Duplicates Finder v.0.1.0
Usage: java com.softcorporation.neardup.DuplicatesFinder parameters ...

Parameters format: -parameter [value]
    -start filename[,filename]  directory / file(s) to search for duplicates (mandatory)
    -report filename            report file (by default report goes to ./report.log file)
    -score                      the score to report the duplicate (default is 0.6)
    -onlynew                    find the duplicates only for new documents
    -gram                       number of words in a phrase
    -purge                      clear files list from past runs
    -db                         location of db directory with files list
    -delete criteria            remove duplicates by criteria (old, new, small, large)
    -deletepath pattern         remove only matching pattern files (mandatory for delete)
    -verbose                    display progress information (on standard output)

Example: Find duplicates from text files in directory 'docs' and save report in 'report.log'
    java com.softcorporation.neardup.DuplicatesFinder -start docs -report report.log

For more information visit web site: http://www.softcorporation.com/products/neardup
Improve this answer
0

I've found SSDeepFE for ssdeep.

Also, sdhash seems to have a web-based GUI but it isn't very useful yet.

Improve this answer
0

I think this is the duplicate file finder that you are looking for. There is similarity mode which works very nice on source code files too (text files), but it is available in the commercial PRO version.

Improve this answer
2
  • Have you tried it?
    – Alix Axel
    May 22, 2013 at 23:45
  • Yes I have. I have already shared that it works very nice on source files (text). The similarity mode does not use the so called FUZZY search, but rather than that it performs some deep analysis and can find similar documents even if entire paragraphs are swapped. This tool worked the best for me, but the similarity mode is slower and should be used on a smaller subset of files. The similarity mode works in the FREE version with some limitations:link
    – Jack Dorsey
    May 23, 2013 at 11:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .