1

I'm creating a script for copy/checksum of files....running latest Mac OS X/FreeBSD with possibility to port into CentOS, Debian or OpenBSD

More about script:

  1. check whether source path contains files/subdirs
  2. create file(s) checksum for each dir/subdir
  3. tar/compress into target path
  4. check file integrity in target path

Of course it is paranoid because file integrity checks are done on HW/HDD level and can be easily checked by S.M.A.R.T. but ten years later I'm unable to check integrity of original. Checksum is created on CF/XD card and it is original...you can copy as much as you like and never worry about so called rotten-bits, HW errors and so on.

Of course rsync can be used as well but I don't like idea of obsolete MD5/SHA1 checksums with possible collisions. It takes hours of work, "luck" and sweat to be on right place at right time and take one unique photo...if you lose original RAW...it is gone forever, only memory remains.

"Only paranoid survive" -- Andy Groove

I have a simple working script for step 1. in script:

today=`date +%Y-%m-%d`
CHK='shasum -a512'
CHK_OUTPUT=($today)-checksum.txt
find . -type f ! -name  ".*" -maxdepth 1 -exec $CHK {} \; > "$CHK_OUTPUT"

I get checksum file as expected but question is "Can we make it better?"

...cf83e1357eef47417a81a538327af927da3e  ./(2017-07-19)-checksum.txt

I want to get rid off annoying ./ so I coded following...

find ./ -type f ! -name  ".*" -maxdepth 1 -exec bash -c '$CHK $(basename {}) > $CHK_OUTPUT' \;

unfortunately, I get following error

bash: ${CHK_OUTPUT}: ambiguous redirect

another attempt

    find ./ -type f ! -name  ".*" -maxdepth 1 -exec bash -c '$CHK $(basename {})' \; > $CHK_OUTPUT

it somehow works but with weird results

I failed by UTFM & RTFM and I have no idea how to even ask Google :-D

Can anyone suggest how to do it, please?

Regards

David

  • You should be aware that some of the operating systems you mention do not have bash installed by default. – fd0 Jul 19 '17 at 19:02
2

How to pass arguments to a subshell in -exec

With find's -exec, you can pass complex commands using a subshell, as you correctly identified. However, there are some problems with your approach.

Your outside variable $CHK will not be expanded since it is within single quotes. What you can do to make the subshell know this variable is to export it before:

$ foo=bar
$ find . -type f -exec sh -c 'echo "$foo"' \;
(returns an empty line for every file found)

$ export foo=bar
$ find . -type f -exec sh -c 'echo "$foo"' \;
(returns "bar" for every file found)

Exporting a variable makes it part of your environment, which subshells can read. Or you pass it as a separate argument to the subshell, which is the go-to way here:

$ foo=bar
$ find . -type f -exec sh -c 'echo "$0"' "$foo" \;
(returns "bar" for every file found)

Of course, you can go on with $1, $2 etc, and use {} as an argument for your subshell, too, to use the actual filename. And don't forget to quote your variables.

Your specific case

You can actually simply rewrite your command into:

shasum -a512 * > "$CHK_OUTPUT"

because shasum is clever enough to do the job in one command, reading multiple files, without a loop or find. By default, * does not include files starting with a dot (but you can change it with shopt -s dotglob), so your find options are unnecessary, especially when the maxdepth is 1.

But let's pretend shasum wasn't that clever, so I can give you some more options. If you want to use find, this is how you'd handle multiple arguments:

CHK='shasum -a512'
find ./ -type f ! -name  ".*" -maxdepth 1 -exec \
sh -c '$0 "$(basename "$1")"' "$CHK" {} \; > "$CHK_OUTPUT"

But even then, this all can also be rewritten as a more readable:

today=$(date +%Y-%m-%d)
for f in *; do shasum -a512 "$f" > "($today)-checksum.txt"; done

It's often easier to loop through files than using find, although there is a limit on how many files you can process that way due to the expansion of the * — at some point it will be too long for your command line (the specific limit depends on your OS and shell).

And you could of course do it recursively with shopt -s globstar in Bash ≥ 4.0:

shopt -s globstar
for f in **/*; do …; done

But this again is the same as:

shasum -a512 **/* > "$CHK_OUTPUT"
  • thanks for reply slhck I forgot to mention that above by mentioned command is already inside the loop, which reads a single file. the whole idea behind the script is "secure" copy of thousands pictures in many subdirectories (I'm photographer alpinist) the whole script works weird...it looks like I have somewhere missin {}"" in variables because it sometimes works and then next time not. there may be also some problems with special characters in file names such as (2017-07-16)_Hohewand.jpg – David Hajes Jul 20 '17 at 19:06
  • @DavidHajes I assumed it would be part of a larger script. If you have specific problems with other parts of it, feel free to ask a question about it. – slhck Jul 21 '17 at 7:19
  • your solution work with just one problem - it keeps adding unfinished checksum into checksum.txt____tried following but it doesn't make any difference___ find ./ -type f ! -name ".*" -maxdepth 1 -exec sh -c '$0 "$(basename "$1")"' "$CHK" {} > "$CHK_OUTPUT" \; – David Hajes Jul 22 '17 at 11:47
  • What do you mean by "unfinished" checksum? Note that there is a difference between piping to $CHK_OUTPUT before or after \;. When you do it before, find will redirect to that file every time a file is found. You should do it after the find command to gather its entire output. – slhck Jul 23 '17 at 14:41
  • my first version of script did checksum of all files and excluded checksum.txt itself...now...checksum.txt is included on first line and it obviously fails next checksum because it was modified once all files inside a folder have been checksummed.....right now, I'm working on fix to exclude checksum.txt itself – David Hajes Jul 23 '17 at 16:57
0

UPDATE

Unfortunately, below mentioned command still creates empty checksum.txt in subdirectories where are no file(s).

find ./ -type f \( ! -iname ".*" ! -iname "\(????-??-??\)-checksum.txt" \) -maxdepth 1 -exec sh -c '$0 "$(basename "$1")"' "$CHK" {} \; > $CHK_OUTPUT

I don't know why but there is kludge

# remove empty checksum.txt
    if [[ -f $CHK_OUTPUT ]] && [[ ! -s $CHK_OUTPUT ]]; then
        rm $CHK_OUTPUT
    fi

=====================================================

final solution...seems to work so far**

find ./ -type f \( ! -iname ".*" ! -iname "\(????-??-??\)-checksum.txt" \) -maxdepth 1 -exec sh -c '$0 "$(basename "$1")"' "$CHK" {} \; > $CHK_OUTPUT
  • Thanks for closing the loop on your question. This would be a lot more helpful to others with a similar problem if you could expand the answer to explain what this code does. – fixer1234 Jul 23 '17 at 19:47
  • simply read my original question where is all described ;-) – David Hajes Jul 24 '17 at 8:55
  • Guess I shouldn't have been so ambiguous. :-) Obviously, what it does is solve the question. What I was really asking was how and why it solves the question. Can you explain the parameters of the command line; how the command works, which is more educational than simply a long line of unexplained code. "Teach a man to fish" and all that. – fixer1234 Jul 24 '17 at 11:47
  • sadly, if man itself learns to fish...it is hard to teach others to fish :-D – David Hajes Jul 25 '17 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.