0

I would like to run sha1sum or md5sum on all my files recursively, store the results to a file (or a couple of files) and then be able to scan through the files again to verify the integrity of the files.

tools like sha1sum / md5sum does not allow for recursive, and I would also like to spread the load so it uses more than one CPU core.

I do not yet feel safe putting my files on BTRFS and besides BTRFS does not yet work properly on raid5/6 so I am looking for a "quick" way to checksum all my files and verify them whenever I want. That way I would at least know if silent data corruption have hampered any of my files.

Does any tools exist that is designed for this purpose (yes, I am aware of SnapRAID)

  • How about ZFS? It is available on Linux, too, from what I've heard. I mean, I would avoid manual periodic scripts. This is not very reliable. – Martin Sugioarto Oct 9 '16 at 19:38
  • What about find -f / - exec md5sum {} \; > logfile.md5sum ? and use that file after? – Hastur Oct 9 '16 at 20:04
  • @Hastur : In Debian your command does not work. If anything it have to look like this: 'find -type f /path/ -exec md5sum {} \; >checksums' and to check it you could use 'md5sum -c checksums | grep ":FAILED" >checksumfailures' – Waxhead Oct 9 '16 at 21:10
1

These tools are intended for more or less this use case:

http://md5deep.sourceforge.net/

  • Hashdeep is especially useful. Thanks for sharing! – MariusMatutiae Oct 10 '16 at 5:39
1

Or you could write something on your own;) I wrote this script in Python merging other people snippets from the web. It can be a starting point.

#!/usr/bin/python

import os
import hashlib
from joblib import Parallel, delayed
import json

BASE_DIR = "/home/p/music"
NUM_PROC = 2
OUT_FILE = "hashes.json"

# list of files to be processed
file_list = []
for root,dir,files in os.walk(BASE_DIR):
    for name in files :
        file_list.append(os.path.join(root, name))


# -- md5sum computation for a single file 
def hashOne(f):
    BLOCKSIZE = 65536
    hasher = hashlib.md5()
    with open(f, 'rb') as afile:
        print f
        buf = afile.read(BLOCKSIZE)
        while len(buf) > 0:
            hasher.update(buf)
            buf = afile.read(BLOCKSIZE)
    return hasher.hexdigest()

# parallel calculation of md5sum    
md5 = Parallel(n_jobs=NUM_PROC)(delayed(hashOne)(f) for f in file_list) 

# make an hash of file_names -> md5sum
out_hash = {}
for i in range(len(file_list)):
    out_hash[file_list[i]] = md5[i]

# write out the hash "filename -> md5sum" to disk for future 
dump = json.dumps(out_hash)
with open(OUT_FILE, "wb") as file:
    file.write(dump)

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.