Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking for a tool to maintain the integrity of a filesystem and it's contents using checksums. Effectively storing a list of checksums/filename pairs somewhere on the filesystem in a way that can be verified later if files are somehow damaged or lost.

Git does what I want, but because it stores the contents of every file in it's object database, the disk usage will at least double. And the fact that it does not provide a progress bar when scanning files tells me it was not designed for the multi-terabyte filesystem I have in mind.

I can do this crudely by storing the output of md5deep, but is there a tool specifically designed for this purpose, using whatever smarts possible to make the process efficient?

share|improve this question
what filesystems/OS? – Sathya Aug 27 '11 at 14:13
NTFS, Mac OSX . – Jesse Aug 27 '11 at 14:26
up vote 2 down vote accepted

md5deep and related tools are in fact designed to verify file integrity. If MD5/SHA hash algorithms are too slow, it's because they were designed for cryptographic security first – if that is not required, CRC32 checksums are the next best choice. (For example, the Btrfs filesystem uses CRC32 for data integrity.) Search for "SFV" utilities (iSFV, MacSFV) that will automate the checking.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .