I want to do checksumming of large files and stream in unix/linux, and I want get many checksums from every large part of file/stream, every 1 MB or every 10MB.

For example, I have disk image, compressed disk image, and the copy of the original disk. Some parts of images may be modified. The disk is 50 GB, and there is around 50000 of 1 MB blocks. So for every file I want to get 50 000 md5sum or sha1sums to get overview of modifications. Single md5sum will not help me to locate modification offset.

This task is easy for uncompressed disk image, with using dd tool in for loop in bash with computing offsets and selecting (skipping) every 1MB part of file. The same with the disk:

for a in `seq 1 50000`; do echo -n "$a: "; dd if=image.src bs=1M count=1 skip=$a | md5sum; done

But now I want to compare compressed image and uncompressed one without unpacking it to the disk. I have 7z unpacker which can unpack the image to stdout with high speed, up to 150-200 MB/s (options 7z e -so image.7z |). But what can I write after the | symbol to get md5sum of all file parts.


Something simple like this Perl script probably would suffice.

$amount = 1_000_000;
while (read(STDIN, $buffer, $amount) > 0) {
    open MD5, "|md5";
    print MD5 $buffer;
    close MD5;

Put this in foo.pl and invoke it as perl foo.pl at the end of your pipeline.

  • Cool version, thank you. I just finished my own using Crypt::Rhash (md5+sha1+tiger - still rather fast). You can change my code for better readability and style. – osgx May 19 '14 at 23:42
  • And the script shows that my images are similar to my hard drive, but not the same. I almost destroyed 50 GB of unique data. – osgx May 19 '14 at 23:44

split from coreutils (the default on most Linux distributions) has a --filter option which you can use:

7z e -so image.7z | split -b 1000000 --filter=md5sum
  • Nifty. I did not know about this. – Kyle Jones May 20 '14 at 6:06

It seems to me that you're looking for this kind of a tool.

From the Readme file of BigSync:

Bigsync is a tool to incrementally backup a single large file to a slow destination (think network media or a cheap NAS). The most common cases for bigsync are disk images, virtual OSes, encrypted volumes and raw devices.

Bigsync will read the source file in chunks calculating checksums for each one. It will compare them with previously stored values for the destination file and overwrite changed chunks if checksums differ.

This way we minimize the access to a slow target media which is the whole point of bigsync's existence.

  • "Bigsync will read the source file in chunks calculating checksums for each one" - this part is right, but I don't want to modify anything. – osgx May 19 '14 at 23:33

It was easy to write small 1MB hasher using rhash tools (librhash library). There is simple perl script which creates checksums of each 1MB part of standard input stream. It needs Crypt::Rhash bindings from cpan:

$ cpan
(cpan) install Crypt::Rhash
$ cat rhash1M.pl
# Compute md5 and sha1 sum of every 1 MB part of stream

use strict;
use local::lib;
use Crypt::Rhash;

my ($buf, $len, $i);
my $r=Crypt::Rhash->new(RHASH_MD5|RHASH_SHA1);
# we can add more hashes, like RHASH_TIGER etc
binmode STDIN;
while($len= read STDIN,$buf,1024*1024){
    print "$i+$len: \t"; # print offset
    print "md5:",$r->hash(RHASH_MD5), " sha1:", $r->hash(RHASH_SHA1),"\n";
    $r->reset(); # reset hash calculator

This public domain script will output decimal offset, then +, then block size, then md5 and sha1 sums of input.

For example 2 MB of zeroes has sums:

$ dd if=/dev/zero of=zerofile bs=1M count=2
$ ./rhash1M.pl < zerofile 
0+1048576:  md5:b6d81b360a5672d80c27430f39153e2c sha1:3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3 
1048576+1048576:    md5:b6d81b360a5672d80c27430f39153e2c sha1:3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3

rsync works like this, computing a checksum to see if there are differences in parts of the file before sending anything.

I'm not sure how well it would function with files this large, although I have never heard of it having any file size limitation.


Pipe the output to this Python 2 script, for example 7z e -so image.7z | python md5sum.py:

import sys, hashlib
CHUNK_SIZE = 1000 * 1000
for chunk in iter(lambda: sys.stdin.read(CHUNK_SIZE), ''):
    print hashlib.new('md5', chunk).hexdigest()
  • 1
    What is the input? How does this work? What does the user do, from step 1 to step x? This is a low quality answer as is. – Canadian Luke May 20 '14 at 3:29
  • Please add more information to this post (what each line does, etc. Just add comments). I hit Looks Good in the LQP que because I see how it could be fixed easily but as it is it's a LQP. – Jon May 20 '14 at 4:34
  • @CanadianLuke, I've explained how to use it. – Cristian Ciupitu May 20 '14 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.