Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Is there a way I can have a hash value as input when searching for files and a complete list of files and their locations as output?

This could be helpful when trying to pin point file duplicates. I often times find myself in situations where I have a bunch of files that I know I already have stored in some location but I don't know where. They are essentially duplicates.

For instance, I could have a bunch of files on a portable hard drive, and also hard copies of those files on the internal hard drive of a desktop computer... but I'm not sure of the location! Now if the files are not renamed, I could do a file name search to try to locate the hard copy on the desktop. I could then compare them side by side and in case they are the same I could delete the copy I have on the portable hard drive. But if the files have been renamed on either one of the hard drives this would probably not work (depending on how much the new names differ from the original).

If a file is renamed, but not edited, I could calculate its hash value, e.g. SHA1 value is 74e7432df4a66f246b5214d60b190b67e2f6ce52. I would then like to have this value as input when searching for files and have the operating system search through a given directory or the entire file system for files with this exact SHA1 hash value and output a complete list of locations where these files are stored.

I'm using Windows, but I am generally interested in knowing how something like this could be achieved, regardless of operating system.

share|improve this question
    
Unless the file system keeps a table of hashes (most don't) you need to calculate those as part of the search. I would rather use a program that does this for you–it will likely use hashes internally as one mechanism to compare files–than make your own solution. If you do make your own solution, I'd recommend using something like md5 for the hashing. While not cryptographically secure, it's faster than SHA* and provides good enough entropy for the application, for files not intentionally forged to create collisions. –  nitro2k01 Dec 24 '13 at 13:04
    
Hashing a file will rarely be faster than comparing the data in two files (most will fail fairly quickly) –  Bandrami Dec 24 '13 at 13:15
    
If hashing is not a good option, then by what other means can I uniquely identify a file? –  sammyg Dec 24 '13 at 13:53
    
Approximately, how long will it take to hash 60 GiB in 135000 files? This is the entire content of my Users folder. Is there any upper limit on how big files I can hash? I know that small files are hashed fairly quickly, but the big ones might take several minutes to hash. –  sammyg Dec 24 '13 at 13:59

2 Answers 2

This is an intriguing question. I have been using a tool called fdupes to accomplish something similar. Fdupes will recursively search through directories and compare every file with every other file. First it compares size, and if the sizes are identical then it creates hashes of the files and compares that, if the hashes are the same then in actually goes through each file byte by byte and compares it.

When if finds all the files that are truly identical you can have it do several things. I have it delete the duplicate and create a hardlink in it's place (thus saving me HDD space), although you can have it simply output the locations of the duplicate files and not do anything with them. This is the scenario you are asking about.

Some downsides with fdupes are that as far as I know it's Linux only, and since it compares every file to every other file it takes quite a bit of I/O and time to run. It does not "search" for a file per say, but it would list all the files that have an identical hash.

I would highly recommend it and I set it to run in a cron job every day so that I never have any unnecessary duplicates of my data (it excludes my backups of course).

Fdupes Source Page

share|improve this answer

There's a tool ($) called FileLocator Pro that can search by file hash (SHA-x or MD5).

Excerpt from this page: http://www.mythicsoft.com/filelocatorpro/help/en/advanced_criteria.htm

Note: If the expression type is set to 'File Hash' then the containing text box can include a comma separated list of hash values or a pointer to a file containing a list of hash values, e.g.

5A9C9B42A16F5E1985B7B0A019114C7A,675C9B42A16F5E1985B7B0A019114C7A

or,

=c:\FileHashTable.txt

The actual algorithms used to calculate the hash, e.g. SHA1, MD5, are specified in the Options tab.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.