0

I'm trying to build a script to make a mirror-backup of a free ESXi 6.5 to another free ESXi 6.5 host. I'm almost there but this issue is driving me crazy. This is a part of the script; I am using Bash for the script:

#!/bin/sh
find /vmfs/volumes/datastore1/ -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' ! -name *-flat.vmdk | while read line; do
    dir1=$(dirname "${line}"| sed 's/ /\\ /g')
    dir2=$(dirname "${line}"| sed 's/ /\\\\ /g')
    ssh -n [email protected] "mkdir -p $dir1"
    cmd=$(echo $line "XX.XX.XX.XX:\""$dir2"/\"")
    echo $cmd
    scp -pr $cmd
done

output is:

  • for every VM that have no spaces in name, succeded.
  • for every VM with spaces in name, (last word in VM name): No such file or directory

I tried everything to make this SCP get the full path and it ignores everything: Put single quotes, double quotes, escape char to space, double, triple escape chars. Put args directly in SCP, put all the args of SCP in a variable and pass it after.

When running outside script, command runs flawless. When running in script, it's giving error and takes only last part after space.

3
  • I created 2 vars (dir1 and dir2) because when creating remote dirs, dir1 works good, but when copying with scp i read here that i need to double-escape or triple-escape the spaces
    – Costin
    Sep 23, 2019 at 13:43
  • I have a hunch that you should use array...
    – Tom Yan
    Sep 23, 2019 at 13:46
  • How would that save me from "space nightmare" ? I would still have to pass the variable to scp....
    – Costin
    Sep 23, 2019 at 14:03

4 Answers 4

1

Your code is flawed in many aspects.

-name *-flat.vmdk is prone to globbing; what it expands to depends on files in the current working directory. * should be quoted (e.g. -name '*-flat.vmdk').

This is not the only time your code lacks quotes. echo $line is flawed because of this (and this in general).

read line should be at least IFS= read -r line. It would still fail if any path (returned by find) contained the newline character (which is a valid character in file names). For this reason find … -exec … \; is better. You can go like this:

find … -exec sh -c '…' sh {} \;

which introduces another level of quoting; or like this:

find … -exec helper_script {} \;

which makes quoting in the helper_script easier. The latter approach is advocated by this answer, still the answer doesn't fix other issues.

Your variables dir1 and dir2 seem to inject some cumbersome escaping to deal with spaces. You should not rely on escaping like this. Even if you managed to make it work with spaces, there are other characters you would need to escape in general. The right way is to quote properly.

There are at least three levels of quoting:

  1. in the original shell where find is invoked;
  2. in a shell spawned by -exec sh or in a shell interpreting the helper_script;
  3. in a shell spawned on the remote side by ssh … "whatever command" (similarly for paths processed by scp).

Introducing a helper_script makes the first level not interfere with the rest. The main command would be:

find /vmfs/volumes/datastore1/ -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' ! -name '*-flat.vmdk' -exec /path/to/helper_script {} \;

And the helper_script:

#!/bin/sh
# no need for bash

addrs=XX.XX.XX.XX

pth="$1"
drctry="${pth%/*}"
# no need for dirname (separate executable)

ssh "root@$addrs" "mkdir -p '$drctry'"
scp -pr "$pth" "$addrs:'$drctry/'"

Now the important thing is ssh gets mkdir -p 'whatever/the var{a,b}e/expand$t*' as a string. This is passed to the remote shell and interpreted. Without the inner single quotes it could be interpreted in a way you don't want; my example exaggerates this. You could try to escape every troublesome character, it would be hard; so quote.

But if the variable contains any single-quote then some substring can be unquoted on the remote side. This opens a code injection vulnerability. E.g. this path:

…/foo/'$(nasty command)'bar/baz/…

will be very dangerous when embedded in single-quotes and interpreted. You should sanitize $drctry beforehand:

drctry="$(printf '%s' "${pth%/*}" | sed "s/'/'\"'\"'/g")"

The example dangerous path will now look like this:

…/foo/'"'"'$(nasty command)'"'"'bar/baz/…

This is somewhat similar to your usage of sed, but since the single-quote character is now the only troublesome character, it should be better.

scp needs similar quoting in the remote path for basically the same reason. Again, proper escaping with backslashes is more hassle (if possible at all).


A slight improvement is to allow the helper script to process more than one object. This will run less shell processes:

find /vmfs/volumes/datastore1/ -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' ! -name '*-flat.vmdk' -exec /path/to/helper_script_2 {} +

And the helper_script_2:

#!/bin/sh

addrs=XX.XX.XX.XX

for pth; do
   drctry="$(printf '%s' "${pth%/*}" | sed "s/'/'\"'\"'/g")"
   ssh "root@$addrs" "mkdir -p '$drctry'"
   scp -pr "$pth" "$addrs:'$drctry/'"
done

It's possible to build a standalone command (not referring to any helper script) with -exec sh -c '…' (or -exec sh -c "…"). Because of the most outer quotes, this would tun into a quoting and/or escaping frenzy. The following trick with command substitution and here document is useful to avoid this:

find /vmfs/volumes/datastore1/ \
   -type f \
   -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' \
 ! -name '*-flat.vmdk' \
   -exec sh -c "$(cat << 'EOF'

addrs=XX.XX.XX.XX

for pth; do
   drctry="$(printf '%s' "${pth%/*}" | sed "s/'/'\"'\"'/g")"
   ssh "root@$addrs" "mkdir -p '$drctry'" \
   && scp -pr "$pth" "$addrs:'$drctry/'"
done

EOF
   )" sh {} +

To fully understand this (and some fragments in previous snippets) in the context of variable expansion you need to know about quotes within quotes and why EOF is quoted (the linked answer cites man bash but this is more general POSIX behavior). Also note I added -type f to rule out possible directories matching the regex; and I wrote ssh … && scp …, so if the former fails (which includes when mkdir -p fails), the latter will not run.

9
  • THANX a LOT ! Your advices and info are VERY precious ! Some of them i know (like why i need and what it is EOF, some cases of why i need to quote). I'm not a heavy scripting user, i don't do this often, i usualy build scripts to automate tasks and focus on get the job done and use as low resurces as i can (exclude redundant steps, data). I use a lot of forums but this forum by far gave me most of the answers i needed. This is the first time in ~10 years i neded help to make a script working.. I'm getting old... :(
    – Costin
    Sep 24, 2019 at 7:56
  • my (almost) final script looks like that (the testing is still pending as i cannot simulate the copying of the disks without snapshot of VM, and i cannot skip first VMs that works anyway):
    – Costin
    Sep 24, 2019 at 8:41
  • #!/bin/sh echo "--------- Backup config files -------------" find /vmfs/volumes/datastore1/ \ -type f \ -regex '.*\.(vmx\|nvram\|vmsd\|vmdk)$' \ ! -name '-flat.vmdk' \ -exec sh -c "$(cat << 'EOF' addrs=XX.XX.XX.XX for pth; do drctry="$(printf '%s' "${pth%/}" | sed "s/'/'\"'\"'/g")" ssh "root@$addrs" "mkdir -p '$drctry'" && scp -pr "$pth" "$addrs:'$drctry/'" done EOF )" sh {} +```
    – Costin
    Sep 24, 2019 at 8:46
  • 1
    @Costin Comments are not for posting code. Please make an edit to the question, add the current script (without removing the original content all the answers refer to), explain what improvements you made so far and what the outcome is. If you can identify the exact path that makes the script fail, include it in the question as well. Sep 24, 2019 at 9:36
  • 1
    @Costin One more thing. This is a Q&A site. If you believe your original problem is not solved yet, remove the acceptance from my answer to indicate the problem stands. Sep 24, 2019 at 9:38
0

Move the stuff on the right of the pipe (|) to a shell script, then do something like

find /vmfs/volumes/datastore1/ -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' ! -name *-flat.vmdk -exec /path/to/shell/script {} \;

The {} will properly escape each and every file name it successfully finds and then call your script passing the escaped/quoted file name as the first argument. Simply access it with $1 in your script.

0

You say this:

“When running outside script, command runs flawless. When running in script, it's giving error and takes only last part after space.”

What shell script are you using? What is your shebang (#!)? This all sounds like your command works in Bash but your script is running a generic shell.

So I would recommend that you make the first line of your script this:

#!/bin/bash -l

That will make the script run via Bash and the -l ensures it runs with your user’s environment variables. So your script will look something like this:

#!/bin/bash -l

find /vmfs/volumes/datastore1/ -regex '.*\.\(vmx\|nvram\|vmsd\|vmdk\)$' ! -name *-flat.vmdk | while read line; do
    dir1=$(dirname "${line}"| sed 's/ /\\ /g')
    dir2=$(dirname "${line}"| sed 's/ /\\\\ /g')
    ssh -n [email protected] "mkdir -p $dir1"
    cmd=$(echo $line "XX.XX.XX.XX:\""$dir2"/\"")
    echo $cmd
    scp -pr $cmd
done
7
  • 1
    I use bash, forgot to mention (it's not in paste)
    – Costin
    Sep 23, 2019 at 14:33
  • 1
    Well, a hundred appologies.... It seems your question indicated the problem... I always used bash and i wasn't paying attention on the shebang of the script (which i copied with a part of the script from vmware forums). It seems on ESXi hosts (VMware) there is no bash. Only a type of bourne shell (busybox), therefore all the strange behaviour. I'll start over. All your info is VERY precious and helped a lot. When i'll finish, i'll post the final version of my script.
    – Costin
    Sep 24, 2019 at 7:37
  • @Costin Happy to help! And welcome to the world of… BusyBox! If you found my answer to be helpful, please be sure to upvote it and if it’s the answer that solved your problem, please remember to check it off as such. Sep 24, 2019 at 13:37
  • Thank you. I'm new to this platform and i don't know much about it. Is there a way to mark an answer as helpful ? Because another user gave me the complete solution (the whole script) that worked flawless and i already gave him the mark as answer. But you were also very helpful, and i would like to mark that...
    – Costin
    Sep 24, 2019 at 13:57
  • @Costin You upvote answers that are helpful and check off answers as answers if they are the answer that solved the problem. Sep 24, 2019 at 13:58
0

Witness the magic of array:

$ line="meh bleh"
$ dir="hello\ world"
$ cmd=$(echo "$line" "$dir")
$ for i in $cmd; do echo "$i"; done
meh
bleh
hello\
world
$ for i in "$cmd"; do echo "$i"; done
meh bleh hello\ world
$ cmd=("$line" "$dir")
$ for i in "${cmd[@]}"; do echo "$i"; done
meh bleh
hello\ world
$

The problem with putting everything in a simple variable is that no one can tell what each argument is anymore.

3
  • thank you, i'll use arrays. I'll post the final version here when it will be done. :)
    – Costin
    Sep 24, 2019 at 7:39
  • If you are not using bash, array may not be available btw. Actually I'm not sure you need it anyway. I don't see why you would need to put the two arguments in a variable/array before you passed it to echo / scp. You merely seem to have "over-quoted".
    – Tom Yan
    Sep 24, 2019 at 8:18
  • yes, i just found out that. I put the arguments in a variable to check/control what is passed to scp, as i suspected a problem with quotes/spaces/etc
    – Costin
    Sep 24, 2019 at 8:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .