I'm looking for a tool to maintain the integrity of a filesystem and it's contents using checksums. Effectively storing a list of checksums/filename pairs somewhere on the filesystem in a way that can be verified later if files are somehow damaged or lost.

Git does what I want, but because it stores the contents of every file in it's object database, the disk usage will at least double. And the fact that it does not provide a progress bar when scanning files tells me it was not designed for the multi-terabyte filesystem I have in mind.

I can do this crudely by storing the output of md5deep, but is there a tool specifically designed for this purpose, using whatever smarts possible to make the process efficient?

  • what filesystems/OS?
    – Sathyajith Bhat
    Aug 27, 2011 at 14:13
  • NTFS, Mac OSX .
    – Jesse
    Aug 27, 2011 at 14:26

1 Answer 1


md5deep and related tools are in fact designed to verify file integrity. If MD5/SHA hash algorithms are too slow, it's because they were designed for cryptographic security first – if that is not required, CRC32 checksums are the next best choice. (For example, the Btrfs filesystem uses CRC32 for data integrity.) Search for "SFV" utilities (iSFV, MacSFV) that will automate the checking.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.