Command executed via ssh does not return proper return code

Question

Why this does not work?

$ ssh vm "bullshitcommand; echo $?"
bash: bullshitcommand: command not found
0

I expected non-zero return code.

Answer 1

Reason:

echo $?, you're actually sending a local variable $? and asking to echo it out in ssh shell, it seems. You can prove that by using not existing command before your ssh statement. For example:

notexistingcommand; ssh vm "echo $?" outputs

bash: nonexistentcommand: command not found

127

or even

export var=123; ssh vm "echo $var" outputs

123

Solution:

What you can do to use the remote variable is escaping and that will spit out -1 (or 127):

ssh vm "nonexistentcmd; echo \$?"

Or you can use single quotes (') as they do not expand variables and take $ symbol literally

ssh vm 'nonexistentcmd; echo $?'

Answer 2

Instead of saying:

ssh vm "bullshitcommand; echo $?"

say:

ssh vm "bullshitcommand"
echo $?

The return code of ssh would be the same as that of the remote command.

Answer 3

You sayd, this doesn't work:

$ ssh vm "bullshitcommand; echo $?"
bash: bullshitcommand: command not found
0

At first, let's analyze characters:

"not important $? not important"

is interpreted locally on client side, not on server This will be interpreted as:

"not important 0 not important"

because client side (local) special variable $? usually is 0. For preventing it you should make escaping, because you want put text $? as is without substitution:

"not important \$? not important"

Another problem is what you do. You simply lost exit code you interested in. Let look at example below:

$ ssh vm "bullshitcommand; echo external ret: \$?"; echo internal ret: $?
bash: bullshitcommand: command not found
external ret: 127
internal ret: 0

what happened? external $? intercepts exit code, then one step after this you have exitcode from echo command. This is allways 0 because echo has no possibility to crash :) And finally ssh server get exit code from last command, it is got from echo and is 0. After that 0 code is given by ssh client. Finally this cause ssh command exits with 0.

You can workaround this by temporary saving $? value, for example like that:

$ ssh vm "bullshitcommand; exitcode=\$? echo external ret: \$exitcode; exit \$exitcode"; echo internal ret: $?
bash: bullshitcommand: command not found
external ret: 127
internal ret: 127

When you will receive that result, you're at home :). Problem is, exitstatus is impermament, and above you've got workaround for this. Remember about escaping :)

But pay attention for historical broken ssh servers and clients. Sometimes server and/or client ignores exit status and permanently give back 0 return code. Test it if you must be sure. If you have that situation you should upgrade software, or do some workaround with transferring exit code by stdout stream. It is additional work for script and programmer that make it.