Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Context: I have a large terabyte drive with various types of large media files, iso files etc. I would like to verify its contents using md5sum on the first megabyte due to speed/performance.

You can create a sum like this:

FILE=four_gig_file.iso
SUM=$(head -c 1M "$FILE" | md5sum)
printf "%s *%s\n" ${SUM%-} "$FILE" >>test.md5

How would you verify this as the first megabyte's signature is different than the whole file's?

I've seen this done in other languages, but I am wondering how to do it in bash. I've experimented with various md5sum -c permutations involving pipes and whatnot.


Instead of using md5sum -c, would you have to recompute the hashes into a new file, then 'diff' them?

You can use a

find /directory/path/ -type f -print0 | xargs -0 md5sum blah blah 

to work on a large number of files.

PS: Rsync is not an option

UPDATE 2: So as it stands -- Using head, find, and md5sum; one could then create a file from the source directory fairly quickly, then check it with diff on the other side after computing on the destination. Any thoughts, suggestions. Has anybody got any clever one-liners or scripts to share? Thanks again!

share|improve this question

1 Answer 1

up vote 1 down vote accepted

To verify contents by only sampling the first megabyte of a file will likely not detect if some of the larger files have been corrupted, damaged or altered in one way or another. The reason for that is you're only giving the hashing algorithm one megabyte of data when there might be hundreds of other megabytes that could be off. Even one bit in the wrong position would give a different signature.

If data integrity is what you want to verify, you're better off with the CRC32 algorithm. It's faster than MD5. Although it it is possible to forge/modify a file to appear to have the correct CRC32 signature, it is not likely that random bits of corruption will ever do that.

Update:

Here's a nice one-liner to do the 1 megabyte based md5 checksum on every file:

find ./ -type f -print0 | xargs -0 -n1 -I{} sh -c "echo '{}' >> output.md5 && head -c 1M '{}' | md5sum >> output.md5"

Replace md5sum with cksum if you feel like it. Notice that I chose to include the filename in the output. That's because the filename string does not get passed on when you're not giving md5sum the whole file.

share|improve this answer
    
There is that risk, yes. The issue is the speed and performance of md5sum across a large capacity drive containing very large files. To checksum the entire file is not practical. Thank you for the CRC32 tip, I will definitely check into that. –  Bubnoff Jul 29 '11 at 21:15
    
Does the Linux command cksum use CRC32? Both man and info say 'CRC' but not CRC32. –  Bubnoff Jul 29 '11 at 21:26
    
Yes, cksum uses CRC32. There are a few variations of CRC32 (or CRC) which could be confusing if you were to compare checksums from other tools. Check out the wikipedia article for more details: en.wikipedia.org/wiki/Cyclic_redundancy_check –  jesper Jul 29 '11 at 21:40
    
- Thanks! Good to know. I'll do some tests and weigh options from there. Since this is mostly to test the presence/absence of a file, 1 MB might be an OK risk as far as corruption. If CRC32 is fast enough though, maybe I don't need to take that risk. –  Bubnoff Jul 29 '11 at 21:44
    
used time with both md5sum and cksum - I show that cksum is slower than md5sum. –  Bubnoff Jul 29 '11 at 21:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.