Is there any GUI application (either for Windows or Linux) that implements fuzzy hashing algorithms like ssdeep or sdhash to find near duplicate files (mostly text files)?
3 Answers
You can try Near Duplicates Finder. It is not exactly what you are asking for, but it is Java based, so it will work on any platform where Java works (including Windows, OS X and Linux). Is has no GUI interface, but it will find near duplicates for text based files, like TXT, DOC, HTML, PDF, etc.
Example Usage
Near Duplicates Finder v.0.1.0
Usage: java com.softcorporation.neardup.DuplicatesFinder parameters ...
Parameters format: -parameter [value]
-start filename[,filename] directory / file(s) to search for duplicates (mandatory)
-report filename report file (by default report goes to ./report.log file)
-score the score to report the duplicate (default is 0.6)
-onlynew find the duplicates only for new documents
-gram number of words in a phrase
-purge clear files list from past runs
-db location of db directory with files list
-delete criteria remove duplicates by criteria (old, new, small, large)
-deletepath pattern remove only matching pattern files (mandatory for delete)
-verbose display progress information (on standard output)
Example: Find duplicates from text files in directory 'docs' and save report in 'report.log'
java com.softcorporation.neardup.DuplicatesFinder -start docs -report report.log
For more information visit web site: http://www.softcorporation.com/products/neardup
I've found SSDeepFE for ssdeep.
Also, sdhash seems to have a web-based GUI but it isn't very useful yet.
I think this is the duplicate file finder that you are looking for. There is similarity mode which works very nice on source code files too (text files), but it is available in the commercial PRO version.
-
-
Yes I have. I have already shared that it works very nice on source files (text). The similarity mode does not use the so called FUZZY search, but rather than that it performs some deep analysis and can find similar documents even if entire paragraphs are swapped. This tool worked the best for me, but the similarity mode is slower and should be used on a smaller subset of files. The similarity mode works in the FREE version with some limitations:link May 23, 2013 at 11:24