42
sha1sum ./path/to/directory/* | sha1sum 

the above was posted as a way to compute a sha1sum of a directory which contains files. This command fails if the directory includes more directories. Is there a way to recursively compute the sha1sum of a directory of directories universally (without custom fitting an algorithm to the particular directory in question)?

1
  • treesum could presumably be molded into a shape that fits your requirements. The problem is that just like in your example the order of directory entries can vary and so the aggregate SHA-1 hash you aim to compute would vary across runs. So you'd have to sort the hashes by some criteria (which will negate some of the parallel nature of treesum) and make sure that the representation of path names is normalized e.g. with or without leading ./; only ever / instead of \\ on Windows ... etc. Jan 7 at 12:05

12 Answers 12

17

Thanks to this SO post

find . -type f \( -exec sha1sum "$PWD"/{} \; \) | awk '{print $1}' | sort | sha1sum

Warning: This code is untested! Edit this question if it's wrong and you can fix it; I'll approve your edit.

4
  • Sorry; I couldn't resist! ;-) Recursion is fun. Of course there's a way. I'll write up a proper answer now. Aug 6 '12 at 20:12
  • 5
    This is won't generate same hash for the exact same folders on the different machines because the output also contains <hash> and <file path> which file path is different on different machines and cause different hashed on different machines. Correct line should be like find . -type f \( -exec sha1sum "$PWD"/{} \; \) | awk '{print $1}' | sort | sha1sum @allquixotic
    – alper
    Aug 3 '18 at 21:40
  • 2
    In addition to this, hashes of files should be in ordered which will also cause different hashes if the sort order is different on different machines.
    – alper
    Aug 3 '18 at 21:40
  • What is the purpose of the escaped parentheses here? I get the same result with and without them.
    – matthiash
    Dec 1 at 8:08
49

I generally like the "find | xargs" pattern, like so:

find ./path/to/directory/ -type f -print0  | xargs -0 sha1sum

You have to use the "-print0" and "-0", in case there are spaces in file names.

However, this is very similar to the "find -exec cmd {}" pattern.

See a discussion comparing the two patterns here: https://stackoverflow.com/questions/896808/find-exec-cmd-vs-xargs

2
  • 2
    Your answer only returns hash of the files. Hash of the folder should be obtained using find . -type f -print0 | xargs -0 sha1sum | awk '{print $1}' | sha1sum.
    – alper
    Aug 3 '18 at 20:54
  • You can omit the -print0 and -0 parts by using xargs -I "{}" sha1sum "{}". Full command: find configs -type f | xargs -I "{}" sha1sum "{}" | sha1sum | cut -f1 -d' ' I explicitly include the names of the files to ensure, that even a file rename shows up in the config hash.
    – ST-DDT
    May 6 at 15:27
6

INTRODUCTION

A few years ago, I wrote and presented (in this very thread) a script that can check the hash signatures of all individual files in the current directory structure and output it as a list in a text file.

Since then, I've refined this formula several times. I've decided to re-post my new and improved script here as a separate answer. It's written for sha256 but anyone still wanting to use sha1 can do a simple search and replace in gedit to swap sha256 with sha1. Personally, I haven't used sha1 for a couple of years and I would not recommend it as it's become antiquated and google has demonstrated how it can be compromised.

Here is what my new script does:

  1. You can simply use the script by going to the directory you want to hash and inputting:

    sha256rec
    

    Alternatively, you can call this script from another directory by doing:

    sha256rec "/path/to/target/directory/you/want/hash"
    
  2. Script will detect if you have write privileges in current dir. If you do, results will be saved in the current directory. If you don't have write privileges or if your current directory is in a read-only system (such as a cdrom), the results will be saved to current user's home directory.

  3. Script will detect if some of the sub directories are not accessible at current user privileges. If all are readable then no elevation of privilege takes place, if they aren't, then the user's privileges are elevated to root.

  4. Find is used to find all the files in current dir structure (including all the sub-directories). Sort is used to make sure the results are outputted alphabetically. The resulting list undergoes sha256sum and is outputted to a text file.

  5. Since writing the old script I've adopted a design philosophy that temp files are evil and should be avoided when possible as they leave users open to snooping and tampering by malicious third parties. So all data in this new script are manipulated as variables until the very last minute where the results are outputed as a text file.

  6. The resulting file itself is hashed and the path/hash are outputed in the terminal. I like to take pictures of these hashes with an old school offline camera to be able to ensure that the results file hasn't been tampered with when I refer to it at a later date.

  7. Old result files are ignored in the tally. It makes comparing results easier.

Here is an example of the terminal output when running my script:

kernelcrunch@ubuntu:/usr/src/linux-headers-4.13.0-16-generic$ sha256rec
======================================================================= 
sha256rec:         
=======================================================================        
Current Folder : /usr/src/linux-headers-4.13.0-16-generic   
Target Folder  : /usr/src/linux-headers-4.13.0-16-generic
Output File    : /home/kernelcrunch/000_sha256sum_recurs_linux-headers-4.13.0-16-generic_d_22-04-2018_t_02.17.txt


Seems you're currently in either a Read-Only system or a root owned directory as a regular user. You can find the hash results in your home folder.
f3ddb06212622c375c6bcc11bd629ce38f6c48b7474054ca6f569ded4b4af9d8  /home/kernelcrunch/000_sha256sum_recurs_linux-headers-4.13.0-16-generic_d_22-04-2018_t_02.17.txt
Operation Length: 10 Seconds.
=======================================================================
kernelcrunch@ubuntu:/usr/src/linux-headers-4.13.0-16-generic$ 

Here is a snippet of the output that can be found in 000_sha256sum_recurs_linux-headers-4.13.0-16-generic_d_22-04-2018_t_02.17.txt:

79c3f378a42bd225642220cc1e4801deb35c046475bb069a96870ad773082805  ./.9491.d
2e336c69cde866c6f01a3495048d0ebc2871dd9c4cb5d647be029e0205d15ce6  ./.config
174f23ff7a7fba897bfb7cf17e9a501bcecacf7ef0c0d5cf030414c1e257d4e3  ./.config.old
389d83f546b250304a9a01bb3072ff79f9d9e380c8a2106cadbf714a872afe33  ./.missing-syscalls.d
035dc77da819101cb9889b4e515023dddd2c953f00d2653b87c6196a6560903e  ./Module.symvers
b28054d7995233e6d003ceb9ed119a0b3354f5ccf77b8d687fc0353ae3c5bfb8  ./arch/x86/include/generated/asm/.syscalls_32.h.cmd
01cf821170e3e6e592e36a96e8628377151c762ac2ee3210c96004bfaef22f5f  ./arch/x86/include/generated/asm/.syscalls_64.h.cmd
111efa83187c58a74a9b0170fd496b497b0682d109a7c240c17e2ffcc734f4f4  ./arch/x86/include/generated/asm/.unistd_32_ia32.h.cmd
fcba4e8abf9e95472c31708555db844ac43c87260fb0ba706b6f519404bf9aba  ./arch/x86/include/generated/asm/.unistd_64_x32.h.cmd
3264438a54cbf7e62b05d38a93c5df8fe4202ac782a5d83ed202cba9eee71139  ./arch/x86/include/generated/asm/.xen-hypercalls.h.cmd
4bd7a45837da7de379b87242efe562ce06bf9d8ab8f636c205bb5ef384c8f759  ./arch/x86/include/generated/asm/clkdev.h
0d96461abd23bbf2da522822948455413a345f9ef8ac7a7f81c6126584b3c964  ./arch/x86/include/generated/asm/dma-contiguous.h
b1a54c24a12ce2c0f283661121974436cdb09ae91822497458072f5f97447c5d  ./arch/x86/include/generated/asm/early_ioremap.h
dd864107295503e102ea339e0fd4496204c697bdd5c1b1a35864dfefe504a990  ./arch/x86/include/generated/asm/mcs_spinlock.h
782ce66804d000472b3c601978fa9bd98dcf3b2750d608c684dc52dd1aa0eb7e  ./arch/x86/include/generated/asm/mm-arch-hooks.h
cd9913197f90cd06e55b19be1e02746655b5e52e388f13ec29032294c2f75897  ./arch/x86/include/generated/asm/syscalls_32.h
758ce35908e8cfeec956f57a206d8064a83a49298e47d47b7e9a7d37b5d96d59  ./arch/x86/include/generated/asm/syscalls_64.h
1147ca3a8443d9ccbdf9cd1f4b9b633f0b77f0559b83ec5e4fa594eadb2548be  ./arch/x86/include/generated/asm/unistd_32_ia32.h
ca5223fbf8f03613a6b000e20eb275d9b8081c8059bc540481a303ce722d42f3  ./arch/x86/include/generated/asm/unistd_64_x32.h
31703052c0d2ab8fe14b4e5dfcc45fcbd5feb5016b0a729b6ba92caa52b069e2  ./arch/x86/include/generated/asm/xen-hypercalls.h
c085ff1b6e9d06faa3fc6a55f69f9065c54098d206827deec7fe0a59d316fc99  ./arch/x86/include/generated/uapi/asm/.unistd_32.h.cmd
7929c16d349845cebb9e303e0ff15f67d924cac42940d0f7271584f1346635fc  ./arch/x86/include/generated/uapi/asm/.unistd_64.h.cmd
9aa492c5a75f5547f8d1dc454bef78189b8f262d1c4b00323a577907f138a63e  ./arch/x86/include/generated/uapi/asm/.unistd_x32.h.cmd
f568e151bbbb5d51fd531604a4a5ca9f17004142cd38ce019f0d5c661d32e36b  ./arch/x86/include/generated/uapi/asm/unistd_32.h
c45cf378498aa06b808bb9ccf5c3c4518e26501667f06c907a385671c60f14ae  ./arch/x86/include/generated/uapi/asm/unistd_64.h
a0088d8d86d7fd96798faa32aa427ed87743d3a0db76605b153d5124845161e2  ./arch/x86/include/generated/uapi/asm/unistd_x32.h
e757eb6420dffa6b24b7aa38ca57e6d6f0bfa7d6f3ea23bbc08789c7e31d15fa  ./arch/x86/kernel/.asm-offsets.s.cmd
f9e703e4f148d370d445c2f8c95f4a1b1ccde28c149cff2db5067c949a63d542  ./arch/x86/kernel/asm-offsets.s
7971fb3e0cc3a3564302b9a3e1ad188d2a00b653189968bbc155d42c70ce6fbf  ./arch/x86/purgatory/.entry64.o.cmd
8352d79fe81d2cf694880f428e283d79fd4b498cea5a425644da25a9641be26b  ./arch/x86/purgatory/.kexec-purgatory.c.cmd
37f3edbee777e955ba3b402098cb6c07500cf9dc7e1d44737f772ac222e6eb3e  ./arch/x86/purgatory/.purgatory.o.cmd
bb8b895cbd2611b69e2f46c2565b4c2e63a85afb56cff946a555f2d277ee99b2  ./arch/x86/purgatory/.purgatory.ro.cmd
bcc2365c9d3d027f1469806eb4f77b0f3ede6eb0855ea0fcd28aa65884046a54  ./arch/x86/purgatory/.setup-x86_64.o.cmd
872229f334fdcc8562e31b9f6581008c1571ac91f12889cd0ff413590585155a  ./arch/x86/purgatory/.sha256.o.cmd
6fb0cbef120aadee282f7bc3b5ea2f912980f16712281f8f7b65901005194422  ./arch/x86/purgatory/.stack.o.cmd
cd1b61063ae3cf45ee0c58b2c55039f3eac5f67a5154726d288b4708c4d43deb  ./arch/x86/purgatory/.string.o.cmd
e5826f0216fd590972bbc8162dd175f87f9f7140c8101505d8ca5849c850ec91  ./arch/x86/purgatory/entry64.o

(it goes on for another 7000+ lines like this but you get the idea)

INSTALLATION

  1. Open a terminal and input the following commands:

    cd /usr/bin
    sudo su
    echo '#!/bin/bash'> /usr/bin/sha256rec
    chmod +x /usr/bin/sha256rec
    touch /usr/bin/sha256rec
    nano /usr/bin/sha256rec
    
  2. In nano, use Shif+Ctrl+v to paste. Ctrl-O and Enter to save. Ctr-X exits. Paste my script in there:

(paste after the #!/bin/bash)

  #FUNCTIONS OR FUNCTYOU?
  function s_readonly { err=$(date +%s%N); cd "$1"; mkdir $err 2> /tmp/$err; rmdir $err 2>/dev/null; echo $(cat /tmp/$err|grep -i "Read-only file system"|wc -l);shred -n 0 -uz /tmp/$err; }
  function w_denied { echo $(err=$(date +%s%N); cd "$1"; mkdir $err 2> /tmp/$err; rmdir $err 2>/dev/null; cat /tmp/$err|grep -i "Permission denied"|wc -l;shred -n 0 -uz /tmp/$err); }
  function r_denied { echo $(err=$(date +%s%N); cd "$1" >/dev/null 2> /tmp/$err; find . >/dev/null 2>> /tmp/$err; cat /tmp/$err|grep -i "Permission denied"|wc -l;shred -n 0 -uz /tmp/$err); }
  function rando_name { rando=$(echo $(date +%s%N)|sha256sum|awk '{print $1}'); rando=${rando::$(shuf -i 30-77 -n 1)}; echo $rando;}
  function ms0 { ms0=$(($(date +%s%N)/1000000)); }; function mstot { echo $(($(($(date +%s%N)/1000000))-$ms0));}
  function s0 { s0=$(date +%s); }; function stot { echo $(($(date +%s)-$s0));}
  s0

  #CHECK IF A TARGET DIR WAS SPECIFIED (-t= or --target= switch)
  if [ ! -z "$1" ]; then arg1="$1"; arg1_3=${arg1::3}; arg1_9=${arg1::9};fi
  if [ "$arg1_3" = "-t=" -o "$arg1_9" = "--target=" ]; then 
    switch=$(echo $arg1|awk -F '=' '{print $1}')
    switch_chr=$((${#switch}+1))
    target=${arg1:$switch_chr}
    current=$(pwd)
    cd "$target"
    arg1="" #<- cancels the not path in the find line
  else
    current=$(pwd)
    target=$(pwd) 
  fi

  echo -e  "=======================================================================\
    \nsha256rec: \
          \n=======================================================================\
          \nCurrent Folder : $current \
    \nTarget Folder  : $target"

  #GETS DEFAULT_USER, ASSUME'S YOU'RE USER 1000, IF 1000 DOESN'T EXIST SEARCHES 999, THEN 1001, 1002
  default_user=$(awk -v val=1000 -F ":" '$3==val{print $1}' /etc/passwd)
  if [ -z "$default_user" ]; then default_user=$(awk -v val=999 -F ":" '$3==val{print $1}' /etc/passwd); fi
  if [ -z "$default_user" ]; then default_user=$(awk -v val=1001 -F ":" '$3==val{print $1}' /etc/passwd); fi
  if [ -z "$default_user" ]; then default_user=$(awk -v val=1002 -F ":" '$3==val{print $1}' /etc/passwd); fi

  if [ "$(users | wc -l)" = "1" ]; then USER=$(users|awk '{print $1}'); else USER=$default_user;fi #not perfect but meh...

  #running rando_name in this very specific spot between USER detection and Permission detection, some interfers somehow with detection functions... 
  #the rando function placed underneath the user detection is somehow turning c=$current from the dir path to whatever rando_name puts out.

  #FIGURE OUT WHERE TO PUT HASH LIST
  hash_file="000_sha256sum_recurs_${target##*/}_d_$(date +%d-%m-20%y)_t_$(date +%H.%M).txt"
  if [ $(s_readonly "$current") -gt 0 -o $(w_denied "$current") -gt 0 ]; then if [ "$(whoami)" != root ]; then dest="/home/$(whoami)";echo -e "Output File    : $dest/$hash_file\n\n";echo "Seems you're currently in either a Read-Only system or a root owned directory as a regular user. You can find the hash results in your home folder."; else dest="/home/$USER";echo -e "Output File    : $dest/$hash_file\n\n";echo "Seems you're currently a Read-Only system. You can find the hash results in $USER's home folder.";fi; else dest="$current";echo -e "Output File    : $dest/$hash_file\n\n";echo "Results will be saved here.";fi



  #CAN REGULAR USER ACCESS TARGET DIR? ARE ALL IT'S SUBDIRS READABLE?
  if [ $(r_denied "$target") -gt 0 ]; then sudo=sudo; echo "Some folder were not read-able as a regular user. User elevation will be required.";fi

  #PERFORM RECURSIVE HASHING
  command=$($sudo find . -type f -not -type l -not -path "$arg1"  -not -path "$2"  -not -path "$3" -not -path "$4"  -not -path "$5"  -not -path "$6" -not -path "$7"  -not -path "$8"  -not -path "$9" |grep -v "\./000_sha"|sort|awk "{print \"$sudo sha256sum \\\"\"\$0}"|awk '{print $0"\""}'|tr '\n' ';')
  eval $command > "$dest/$hash_file"

  sha256sum "$dest/$hash_file"
  echo "Operation Length: $(stot) Seconds."
  echo -e  "======================================================================="



  if [ "$target" != "$current" ]; then cd "$current";fi


  exit
  #||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
  #||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
  #||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
  #||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
  1. When you exit from nano, be sure to exit the elevated status by entering:

    exit
    

FINAL THOUGHTS

  1. This will only work if you have bash installed. I've used some synthax for substring manipulation that does not work with sh, dash, ksh, or zsh. You can still use any of the other shells as your daily drivers but bash needs to be installed.

  2. Outputted lists can be compared with a variety tools such as: (in the terminal) diff, sdiff (and graphical) diffuse, kdiff, winmerge.

  3. My file sorts the output based on path, to make it easier to read by humans. I've noticed the sort command working differently across different distros. For example, in one distro CAPITAL letters took priority over non-caps and in the other they did not. This affects the line order of output files and could make files difficult to compare. This should not present any issues if you're always using the script in the same distro but may if hashes lists were generated in two different environments. This is easily remedied by sorting hash files an additional time so that the lines become ordered by hash rather than path:

     cat 000_sha256sum_oldhashlist|sort> ./old
     cat 000_sha256sum_newhashlist|sort> ./new
     sha256sum ./old ./new; diff ./old ./new
    
4
  • A more robust shebang line would be #!/usr/bin/env bash — it will find Bash in other directories as well, because the latter may be installed in /usr/bin rather than /bin, for example, meanwhile env tends to be in /usr/bin at all times as far as I noticed. Also worth to note is that, since you require Bash, you may use [[ blah-blah ]] double-bracket conditional expression instead of more generic [ blah-blah ] single-bracket variant. Apr 22 '18 at 12:26
  • Thanks for the pointers. I just finished looking up the [[ conditionals. They look really useful. Apr 23 '18 at 0:29
  • 1
    The concern about SHA1 being compromised is not really applicable in the case of comparing files after copying to verify integrity. The chances of a file being corrupted in transit but still having the same SHA1 are practically nil. If you suspect that an attacker might have had enough time to generate a different file with a colliding SHA1, then use SHA256, but for the typical case of copying files, that's overkill and slower than SHA1 or MD5. Jan 6 '19 at 9:59
  • 1
    You're own argument can be used against itself. If you're concerned about normal (non attack related) corruption, then sha1 itself is overkill. You can get faster results using md5/crc32. In either situation (tamper detection or corruption) sha1 is not a good fit. Personally, I use these hash lists for both scenarios and have not noticed any perceivable performance hit since I upgraded to sha256 but I'm not running a mega server either. Like I said in the answer, you are free to use any hash you want by replacing my sha256sum command with the one you want: sha1sum, md5sum, b2sum, crc32... Jan 6 '19 at 11:55
5

UPDATE: It's been a few years since I've posted this reply and in the meantime I've rewritten and improved the script I've presented here several times. I've decided to repost the new script as a brand new answer. I would highly recommend it over this one.

INTRODUCTION

I've observed that the order in which the find command outputs the found elements within a directory varies within identical directories on different partitions. If you're comparing the hashes of the same directory, you don't have to worry about that but if you're getting the hashes to ensure that no files were missed or corrupted during a copy, you need to include an additional line for sorting the content of the directory and it's elements. For example, Matthew Bohnsack's answer is quite elegant:

find ./path/to/directory/ -type f -print0  | xargs -0 sha1sum

But if you're using it to compare a copied directory to it's original, you would send the output to a txt file which you would compare to the outputted list from the other directory using Kompare or WinMerge or by simply getting the hashes of each lis. The thing is, as the order in which the find tool will output the content may vary from one directory to another, Kompare will signal many differences because the hashes weren't calculted in the same order. Not a big deal for small directories but quite annoying if you're dealing with 30000 files. Therefore, you have do the extra steps of sorting the output to make it easier to compare the hash lists between the two directories.

find ./path/to/directory/ -type f -print0  | xargs -0 sha1sum > sha1sum_list_unsorted.txt
sort sha1sum_list_unsorted.txt > sha1sum_list_sorted.txt

This would sort the output so that files with same hash are going to be on the same lines when running the differencing program (provided that no files are missing the new directory).

AND ONTO THE SCRIPT...

Here's a script that I wrote. It does what the same thing that the find/xarg answer does but it will sort the files before getting the sha1sum (keeping them in the same directory). The first line of the script finds all the files within the directory recursively. The next one sorts the results alphabetically. The following two, takes the sorted content and appends a sha1sum and quotation marks to the files in the sorted list, making a big shell script that calculates each files hash, one at a time and outputs it to content_sha1sum.txt.

#!/bin/bash
find . -type f > content.txt
sort content.txt > content_sorted.txt
awk '{print "sha1sum \""$0}' content_sorted.txt > temp.txt
awk '{print $0"\""}' temp.txt > get_sha1.sh
chmod +x get_sha1.sh
./get_sha1.sh > content_sha1sum.txt
rm content.txt
rm content_sorted.txt
rm temp.txt
rm get_sha1.sh
xdg-open content_sha1sum.txt

Hope this helps.

3
  • When the total length of all file names fits within command line, piping through sort -z (--zero-terminated) is easier than messing with a bunch of files. Feb 9 '18 at 10:41
  • @AntonSamsonov This is a really old script, I was just learning scripting at the time. I've since rewritten it a whole bunch of times. With regards to your comment what does zero terminating do when sorting: I read the man page of sort. They say zero terminating sticks a zero byte at the end of the line instead of a newline. What does that accomplish? Apr 22 '18 at 5:33
  • I've posted an update to this script as a seperate answer here: superuser.com/questions/458326/… Apr 22 '18 at 7:08
4

Another trick might be to use tar to hash the file contents & metadata:

tar -cf - ./path/to/directory | sha1sum
3
  • too bad that I only have one vote
    – 166_MMX
    Sep 22 '17 at 7:28
  • 1
    This does not work. tar includes a timestamp for some OSs (like OSX) and the sha1sum will be different each run.
    – srossross
    Sep 28 '17 at 20:43
  • What @srossross said. Plus, if you have different versions of tar on the two hosts, the outputs will be different. Jan 6 '19 at 10:00
2

Fast, robust and portable solution

Unlike some of the other solutions involving tar, the solution below works on any machine that has the standard Unix utilities, and is faster than all other solutions by parallelizing the checksumming:

find . -type f | xargs -d'\n' -P0 -n1 md5sum | sort -k 2 | md5sum

Since it uses a sort at the end, there is no real-time progress, so just let the command run.

Here's what the arguments do:

  • find . -type f finds all the files in the current directory and its subdirectories
  • xargs -d'\n' splits the output of find into lines (if you do expect to have files with newlines in them, then do the usual find -print0 | xargs -0)
  • -P0 n1 runs md5sum in parallel processes, using the maximum number of processes supported by the machine (multi-core!)
  • sort -k 2 sorts by the second field of the md5sum output, which is the full path to each file (the first is the MD5)
  • the final md5sum calculates a checksum of the list of file checksums, so you get a checksum of the entire directory on one line, which you can easily compare visually across terminal windows

Before you say that "MD5 has been compromised", keep in mind what your threat model is. Are you trying to make sure that the files you copied from some other host or disk arrived intact? Then MD5 is more than sufficient, because the chances of a file being corrupted in transit but having the same MD5 are zero. But if you are afraid of an attacker having the time to replace a file with a different one with a colliding checksum, then use sha256sum. The downside is that SHA functions are slower than MD5.

Real-time verbose progress

Finally, if you do want to see real-time progress, modify the pipeline to use a temporary file for the checksums:

find . -type f | xargs -d\\n -P0 -n1 md5sum | tee /tmp/sums && sort -k 2 /tmp/sums | md5sum

(Note that moving the sort right after find wouldn't work, because xargs -P0 parallelizes md5sum, and results can arrive out of order.)

This version of the command also lets you diff the two /tmp/sums files (make sure to rename the second one if it's on the same machine) and see which files differ.

7
  • 1
    MD5 has been broken 15 years ago, SHA1 is considered weakened and because the first attacks have been suggested for SHA2 we have Keccak/SHA3 now. And you keep promoting the use of MD5? Dec 9 '20 at 20:38
  • @0xC0000022L: I tried to pre-empt your question by explaining in my answer why I used md5sum for this scenario. Have you read that section? Jan 7 at 11:20
  • Yes, I had read the whole answer. And I find it reckless to use that as a justification for promoting MD5. If all you care about is error detection then CRC32/Adler etc should be fine, too. They're faster than MD5, btw. But if you care about the integrity of the data instead of merely detecting bit errors during transmission, MD5 is severely outdated and wrong. I'll wager someone will use your answer as authoritative argument without understanding even the difference between the purpose of cryptographic hashes and error correction data. Jan 7 at 11:35
  • As a side-note: why did you not link this answer on the same question instead? SHA-1 is weakened and no longer recommended, but to my knowledge it's not officially broken. For MD5 preimaging attacks have become very much feasible, to the extent that malware has been found with signatures "proving" (by way of an MD5 digest) that they had been legitimately signed, when in truth the malware had been changed to achieve getting the same digest which was found in the original signature. So signatures could literally be transplanted. Jan 7 at 11:39
  • That was years ago! And even SHA-1 has been deprecated for signing purposes in all but fringe cases, because MD5 showed how quickly a weakened hash algo can devolve into a broken one. Yet, your answer perpetuates the myth that it's fine to use MD5 and justifies it with speed as the argument. Tell you what, if you cleverly combine the transmission and the hashing you can keep the overhead low and in any case the slowdown from computing a hash compared to the I/O operation or the havoc malicious tampering could wreak is negligible. Jan 7 at 11:43
1

This seems to work for me:

find . \( -not -name . \) -type f -exec cat {} + | sha1sum

EDIT: this will only sha1sum all of the files contained in the directory tree. If a directory's name was changed, this wouldn't catch it. Maybe something like:

find . -exec sha1sum {} + 2>&1 | sha1sum

Would do it. About the same answer as the other one though

1

sha1deep -r . should help getting the sha1sum for each of the files, recursively within a given folder.

If you need an overall checksum, you can pipe the previous result into a new sha1sum. This is convenient to check whether 2 folders are strictly identical (including all their content).

So the solution would be looking like this if you want to work in the current folder:

sha1deep -r . | sha1sum
0

Rather than have ONE enormous file containing all of the hashed information I was looking for a way to make a file in each folder of a tree. I took some inspiration from the comments here. Mine is a bit more complex than what is posted here. I use file rotation but this is the least complex for new players. This version will have it overwrite the old check sums with new ones. It may be good to keep 2-3 versions depending on how frequently you run it and your need for 'depth'.

[user@host bin]$ cat mkshaindir 
#!/bin/dash
cd $1
sha512sum * >.sha512sum

[user@host bin]$ find /var/tmp -type d -print0 | xargs -0 -i  mkshaindir  {}

Note that mkshaindir, for my purposes, is a separate component because there may be a need for me to make a hash of files in a new folder, or of one that was recently changed. This can all be combined in to one script if needed.

The rest is left as an exercise for the reader.

0
0

based on previous answer:

find ./path/to/directory -print0 | LC_ALL=C sort --zero-terminated | tar --create --no-recursion --null --files-from /dev/stdin --file /dev/stdout --verbose --numeric-owner | sha1sum

  • stable sort
  • numeric owner and group id
  • verbose progress
  • filename safe
1
  • This didn't work on a copied directory containing just one file, and I suspect that was because I was running a slightly older version of tar (1.28) on the remote host, vs. 1.29 on the local host. Unfortunately, tar 1.29 hasn't been backported on Xenial. Jan 6 '19 at 8:44
0

The following did it for me when comparing two directories after an adb pull from an android device, without the "$PWD"/:

 find . -type f \( -exec sha1sum {} \; \) | awk '{print $1}' | sort | sha1sum
-1

@allquixotic's answer does not generate same hashes on the different machines that will not help us to verify and have consistent hashes for the folder.

Reason:

The output contains <hash> along with <file path>; which file path may differ on different machines and will cause to generate different hash values on different machines.

Example:

hello_mars.txt contains mars text and hello_world.txt contains world text.

$ pwd
/Users/alper/example
$ ls
hello_mars.txt  hello_world.txt
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \)
9591818c07e900db7e1e0bc4b884c945e6a61b24  /Users/alper/example/./hello_world.txt
02c13f3652546c74a71ae352e8ff0f87a90101b3  /Users/alper/example/./hello_mars.txt
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \) | sha1sum
a540b4f1b74e1fe19f25c366a44088bec266d68b  -

Now I rename the folder from example to example_1:

$ pwd
/Users/alper/example_1
$ ls
hello_mars.txt  hello_world.txt
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \)
9591818c07e900db7e1e0bc4b884c945e6a61b24  /Users/alper/example_1/./hello_world.txt
02c13f3652546c74a71ae352e8ff0f87a90101b3  /Users/alper/example_1/./hello_mars.txt
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \) | sha1sum
b3248c8fd2283788d57c5d774baf895e853ddce4  -

As you can see now generated hash is different from the previous one.


Correct command should be as follows, which will generate same output indepented from the folder's path:

find . -type f \( -exec sha1sum "$PWD"/{} \; \) | awk '{print $1}' | sort | sha1sum

please note that you can also use md5sum instead of sha1sum.


$ pwd
/Users/alper/example
$ ls
hello_mars.txt  hello_world.txt
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \) | awk '{print $1}' | sort
02c13f3652546c74a71ae352e8ff0f87a90101b3
9591818c07e900db7e1e0bc4b884c945e6a61b24
$ find . -type f \( -exec sha1sum "$PWD"/{} \; \) | awk '{print $1}' | sort | sha1sum
e0dcb7c2fbdbd83ece5223cf48796fbdfb0ad68d  -

Hence the path would be different on different machines. awk '{print $1}' will help us the obtain the first column, which has only the hash of the files. Later we need to sort those hashes, where the order might be different on different machines, which may also cause us to have different hashes if there are more than two files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.