What is the difference between ssh host cmd and running ssh host and then running the command inside a shell?

I tried started a server running the command:

ssh user@host "cd some/directory; ./run-server.sh"

The server starts and then abruptly dies. However, when I start the command by running:

ssh user@host
cd some/directory

from inside the shell, the server starts without issue. How is the possible? My understanding of ssh was that these two invocations were indistinguishable.

  • Is your question general? or do you want to troubleshoot the issue with ./run-server.sh? – Kamil Maciorowski Feb 14 at 15:59
  • General: how can there be a difference between these two? I thought they were identical. – user1140191 Feb 14 at 16:03
  • OK. Then I expect you will accept an exhaustive answer rather than a partial answer that by chance names that one difference (unknown yet) that matters for ./run-server.sh. – Kamil Maciorowski Feb 14 at 16:06
  • Sure, are there a lot of differences for an "exhaustive" answer? – user1140191 Feb 14 at 16:08
  • Great, thank you! – user1140191 Feb 14 at 17:54

Based on my own experience, there is a difference between those two methods:

  • When using a one-liner command such as ssh user@host "cd some/directory; ./run-server.sh" you don't actually trigger a captive (or "interactive") login shell: your command is basically executed and then the SSH tunnel is closed as soon as the last instruction has been executed and hands down control back to the user
  • When using the ssh user@host command, you create a captive login shell, so every command thrown from there can execute until the moment you send the Ctrl-C or exit command

If you wish to keep your run-server.sh script alive using the one-liner command, you'll need to investigate how to create a daemon out of your script such as this answer about Debian-based distributions.

  • I don't understand what you mean: When you execute ssh host 'sleep 5', the shell stays open for 5 seconds. If you execute a command and set it in the background, it keeps executing even after you exit the shell. – user1140191 Feb 14 at 15:56

Preliminary notes

  • There are few differences. Which one is relevant, it depends on ./run-server.sh and possibly on your login shell, startup scripts, OS configuration. Further paragraphs elaborate on this.

  • You have revealed nothing about ./run-server.sh. I don't know if it interacts with the user, if it forks, what interpreter (shell) it really uses. This answer is meant to be general.

  • In this answer the word "server" means "the software (run by run-server.sh) that actually serves something" not "the machine".

Pseudo-terminal allocation

ssh user@host will request a pseudo terminal if the client has one. I assume in your case the client has one. ssh user@host ./run-server.sh will not, unless you use -t in the command line (or RequestTTY yes or similar in ssh_config).

I would expect neither run-server.sh nor the server itself to require a tty. Alternatively, if any of them does require and doesn't get a tty, it should print an error message. But in general they may fail silently.

If run-server.sh and the server, when they work (i.e. in your latter case), do their job totally without interacting with the user then they probably don't require a tty. On the other hand if any of them prints an interactive (text) menu then it probably requires one. If any of them asks for a password then it may require a tty.

Startup scripts

ssh user@host runs an interactive login shell. It's the shell specified in /etc/passwd for the user. ssh user@host ./run-server.sh uses the same shell in a non-interactive non-login way.

It depends on the shell how it behaves being interactive or not, login or not. Login shells should source /etc/profile and ~/.profile. Interactive shells may source other file(s) instead (or in addition). Non-interactive non-login shells may source nothing. Some shells may follow more complex rules than others.

It's sane to assume ssh user@host makes the remote shell source something while ssh user@host ./run-server.sh makes the remote shell source nothing. So some startup script(s) are sourced or not.

Depending on what these script(s) do, you may experience the following differences:

  1. Different environments.

    ./run-server.sh or the server itself may rely on some environment variable(s) defined in profile. In particular it's common to define PATH this way. More: it's the right way to define PATH. Without sourcing startup files you're left with the default PATH. In your case the server may have tried to run some component not available via the default PATH.

  2. Different working directories.

    In a startup script one can cd to another directory. In effect the two invocations of ssh will run the same code starting from different directories. Your code invokes cd some/directory where the path is not absolute, then ./run-server.sh is also relative. If this was the problem, you would probably get an error saying some/directory and/or ./run-server.sh doesn't exist. But in general some/directory/run-server.sh could exist in two different directories. You could run two different run-server.sh scripts and it's possible one of them succeeded and the other failed quickly.

  3. Resources available or not.

    In a startup script one can mount/decrypt/prepare some resources. In general it's not the right way to do it (the right way with systemd here) but I've seen various contraptions. If run-server.sh or the server itself depends on such resource then it may fail in case the startup script was not parsed.

  4. Different shells.

    In a startup script one can run (exec to) a different shell. The right way to change your login shell is with chsh; but it will fail if the desired shell is not in /etc/shells. You can try to kinda circumvent it by running the desired shell from your undesired login shell via a startup script (with some logic to avoid recursion, if the new shell is going to parse the same script). In effect the two invocations of ssh may run ./run-server.sh from different shells. If there's a proper shebang in run-server.sh then there should be no difference in interpreting the script. Without a shebang the interpreter depends on the invoking shell. The script may misbehave while being interpreted by a wrong shell.

  5. Something else? (meaning: the list may not be exhaustive).

Sequence of operations

I imagine run-server.sh starts the actual server. The server detaches itself from its original stdin, stdout and stderr; maybe it forks again, maybe as a different user (if allowed).

Possible scenarios:

  1. ssh user@host "cd some/directory; ./run-server.sh" will exit if run-server.sh exits and there's no file descriptor by any process open on the writing end of the pipe. If this happens and the user is no longer logged in, and the remaining process still belongs to the user (i.e. the server is only about to change its user ID, or not going to change it at all), and the setting in logind.conf is KillUserProcesses=yes, then the server will be killed.

    When you run ./run-server.sh manually, you stay logged in. The server is not killed until you log out, or it manages to change its user ID before you log out and therefore it is not affected by KillUserProcesses=yes at all.

  2. Another possibility is the server allowed ssh to exit, survived and only then requested some essential resource governed by a systemd unit depending on the user being logged in. When you run ./run-server.sh manually, you stay logged in and the server manages to get the resource on time.

  3. Other (meaning: the list may not be exhaustive).

Local or remote expansion, quoting and escaping

The command in question is ssh user@host "cd some/directory; ./run-server.sh". The crucial string is double-quoted and there's nothing to be expanded. In general there could be something; and then it could be the culprit. Let's suppose ./run-server.sh takes an argument and you want to pass (expanded) $HOME. Then this

# local shell
ssh user@host "cd some/directory; ./run-server.sh '$HOME'"

will expand $HOME locally (see Variable expansion and quotes within quotes). But this snippet in a remote shell

# remote interactive shell
cd some/directory; ./run-server.sh '$HOME'

will not expand $HOME at all. On the other hand this

# local shell
ssh user@host 'cd some/directory; ./run-server.sh "$HOME"'

will expand $HOME in the remote shell; and this

# remote interactive shell
cd some/directory; ./run-server.sh "$HOME"

will do the same. Or you may try to do this:

# local shell
ssh user@host "cd some/directory; ./run-server.sh $HOME"

(which is flawed because there is no quoting in the context of the remote shell), it will expand $HOME locally; or this:

#local shell
ssh user@host "cd some/directory; ./run-server.sh "$HOME""

(which is even more flawed because the variable is unquoted in the context of the local shell and in the context of the remote shell), it will expand $HOME locally; and compare to this:

# remote interactive shell
cd some/directory; ./run-server.sh $HOME    # also flawed
# or 
cd some/directory; ./run-server.sh "$HOME"  # not flawed

where $HOME is expanded in the remote shell.

You can escape locally:

# local shell
ssh user@host "cd some/directory; ./run-server.sh \"\$HOME\""

and it will properly quote $HOME in the remote shell and expand it there. But this

# remote interactive shell
cd some/directory; ./run-server.sh \"\$HOME\"

will neither expand $HOME nor remove quotes (they will stay as parts of the argument of ./run-server.sh).

The point is $HOME in both systems may be different; various ways of quoting or escaping may get you one or the other (or literal $HOME) in the right or a flawed way. Remember $HOME is just an example here. Any variable/parameter and even other types of expansion (e.g. globbing) bring the issue. It can be that only one of the two shells (local, remote) can expand the syntax you choose. So it's important to know at what moment (if ever) the expansion will take place.

With not enough experience or thoroughness you may think you used equivalent commands, while maybe you didn't. Different input can generate a different result then.

Final note

  • One would expect the server failing for whatever reason to print a proper error message. You may have observed no error messages. You may believe the server is good enough piece of software, so it would die loudly rather than silently; and the silence disturbs you. But after the server daemonizes itself correctly (if it indeed does), it's no longer attached to the stdin/stdout/stderr of the run-server.sh script. For this reason you shouldn't expect the most crucial error message to appear on your screen, check the server's log.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.