rsync: exclude not working when using a variable

Question

Could someone please explain to me this behaviour I noticed lately when I was sync'ing a directory to another server via rsync? I wanted to exclude the sub-directory "done" in this case.

When I put the options to use with rsync into a variable, that directory was not excluded. But it was when I put the options directly behind the rsync call. Changes to "-av" made a difference, but the exclude didn't work.

rsync 3.0.9-18, on bash, CentOS 7.4

Not excluded:

$ RSYNC_OPTIONS='-av --exclude "done"'
$ touch done/test.ignore && rsync ${RSYNC_OPTIONS} ${SOURCEDIR} ${TARGET}
sending incremental file list
done/test.ignore

sent 132 bytes  received 32 bytes  109.33 bytes/sec
total size is 0  speedup is 0.00

Excluded:

$ touch done/test.ignore && rsync -av --exclude "done" ${SOURCEDIR} ${TARGET}
sending incremental file list

sent 41 bytes  received 12 bytes  106.00 bytes/sec
total size is 0  speedup is 0.00

Answer 1

I have replicated the problem and used set -x to see what the both commands really look like. It turned out this command

rsync ${RSYNC_OPTIONS} ${SOURCEDIR} ${TARGET}

is in fact equivalent to this

rsync -av --exclude '"done"' ${SOURCEDIR} ${TARGET}

Note the quotes in quotes. Your pattern is not done; it's "done", as if the directory you want to exclude had double quotes in its actual name.

---

To almost fix this you can declare the variable without these troublesome quotes:

RSYNC_OPTIONS='-av --exclude done'                           # poor fix, don't
rsync ${RSYNC_OPTIONS} "${SOURCEDIR}" "${TARGET}"

But this will backfire if the pattern contains spaces etc. Another approach may be with eval:

RSYNC_OPTIONS='-av --exclude "name  with  double  spaces"'
eval rsync "${RSYNC_OPTIONS}" '"${SOURCEDIR}" "${TARGET}"'   # not recommended

eval will parse the line for the second time. It's very hard to use it right and safely. I double quoted ${RSYNC_OPTIONS} so "name with double spaces" doesn't lose double spaces. I single quoted "${SOURCEDIR}" "${TARGET}", so these variables are not expanded right away (otherwise their content would undergo expansion). This is tricky!

Besides, what about name "with' quotes? To get this exact string as an option-argument to rsync --exclude you need some obscure quoting and escaping in RSYNC_OPTIONS declaration. There are more reasons to avoid eval.

---

The real solution is to use an array in Bash. Note arrays are not portable.

RSYNC_OPTIONS=(-av --exclude 'name with  spaces and $u(h')
rsync "${RSYNC_OPTIONS[@]}" "${SOURCEDIR}" "${TARGET}"

I understand why you didn't quote ${RSYNC_OPTIONS} in your original approach. You should have quoted ${SOURCEDIR} and (separately) ${TARGET} though. The above command quotes each variable properly.

Or maybe ${SOURCEDIR} was meant to specify multiple sources? This would be the reason not to quote it, but then it could bring similar issues as ${RSYNC_OPTIONS}. In this case you should definitely use an array variable here as well.

Also consider variable names in lower case.

Answer 2

What about placing your quotes like this ?

RSYNC_OPTIONS='-av --exclude '"done"''

---

I only tested it on a simplified command:

SEP='-F ";"'
echo "1;2;3" | awk ${SEP} '{print $2}'

(not working)

SEP='-F '";"''
echo "1;2;3" | awk ${SEP} '{print $2}'

(working)

---

Edit:

With @Kamil technique, array works well too:

SEP=(-F ";")
echo "1;2;3" | awk ${SEP[@]} '{print $2}'