2

I'm making deployment scripts for my project, and I'm making a function that should be able to run a command both locally and remotely (on a server through ssh) and return the text output.

When running a local command, I can easily know if the command has finished by waiting for its PID. When I send a command over an open SSH connection, I get output back, but I don't know if the command exited or if the command is still running and will generate more output later.

Currently I've "solved" this by opening a new connection to the server for each command, which needless to say is excruciatingly slow.

This seems to be like it might be a common issue and might have a simple solution, perhaps even something that's built into SSH, so I'm asking here: How can I, over an open SSH connection, know whether the commands I've sent are finished. If I could somehow collect the exit code too that would be great.


To give a more concrete example, this basically demonstrates my problem in Ruby:

io = IO.popen(["ssh", "-q", "my-server"], "r+")

io.write("some command\n")

# Sleep for some arbitrary amount of time, because I don't know when the
# command has finished :(
sleep 1

output = io.readpartial(1_000_000)

io.write("some command\n")

# Sleep for some arbitrary amount of time, because I don't know when the
# command has finished :(
sleep 1

output = io.readpartial(1_000_000)
3
  • 1
    What about to execute a script and deal as was run locally or to give a long command ( something like ) Mycommnad; InterestingPID=$? ; ... ;wait $InterestingPID as multi-line command of ssh instruction? BTW can you use wait?
    – Hastur
    Jun 23, 2015 at 16:19
  • @Hastur I'm afraid I don't understand what you mean...
    – Hubro
    Jun 23, 2015 at 16:25
  • I'm sorry I was not enough clear. I imagine you want to do something like this, catching the PID of a command e.g. ( my_command -myoptions AndParameters) with LetsStoreInterestingPID=$? just after the command, and after some other commands wait $LetsStoreInterestingPID...
    – Hastur
    Jun 23, 2015 at 16:37

3 Answers 3

2

Wrap your command in a script that will notify you when it's done.

If you have an SMTP server available that will accept outgoing emails from your server, you could use that.

The below example will execute your-command, capture its output and stderr to a file, then use malix to mail the results. There's certainly a much better way of making sure the file the output is captured is unique than using the $$ pseudovariable (maybe use a routine that generates a random string).

#/bin/bash
your-command > /tmp/$$.$0.output 2>&1
RESULT=$?
echo >> /tmp/$$.$0.output
echo "Exit code $RESULT" >> /tmp/$$.$0.output
cat /tmp/$$.$0.output | mailx -s "your-command has finished with exit code $RESULT." [email protected]
rm /tmp/$$.$0.output

If you want something that lets you specify a queue of commands, look into Task Spooler (apt-get install tsp on Debian) which lets you add and delete commands from a queue and I believe can even email you the output of them.

Reading your question more carefully ... it seems you are treating what the SSH connection is inputting/outputting as a raw I/O stream.

SSH provides an encrypted pipe and has some features for port forwarding, and not much else. The typical thing connected to that pipe is a shell which usually expects an interactive user on the other end.

I think single-purpose keys can somewhat achieve what you are trying to do somewhat easily, but the easiest thing will be to create a small wrapper script that does this:

#/bin/bash
$@ 
echo "==== Command Output Finished ===="

and then look for the string ==== Command Output Finished ==== in your I/O routines to determine where the boundary between command outputs are. Of course you have to choose something that could not possibly be part of a valid output for a command. A more elaborate scheme is possible.

SMTP does something similar with it's boundary headers.

4
  • My issue is that I'm executing quick commands like find -type f -exec chmod 664 which takes 0.001 seconds or so. When I'm opening an SSH connection for every such command, they end up taking multiple seconds each, which makes my deploy script slow. Involving SMTP servers and mail probably won't fix that...
    – Hubro
    Jun 23, 2015 at 16:43
  • @Hubro Create the list of command to execute and use this solution can help? (the one with cat list_of_command | ssh ...)
    – Hastur
    Jun 23, 2015 at 16:45
  • I've updated my answer.
    – LawrenceC
    Jun 23, 2015 at 19:34
  • This looks like what I planned as my backup solution. I thought I'd generate a random SHA1 string and append ; echo "$? c8m2js7a9... after each command I send to the server, and use that to know when the command is finished and read the exit status. I was just hoping perhaps there was a more elegant solution, like running commands like usual which are transparently actually being executed on the remote server.
    – Hubro
    Jun 24, 2015 at 7:41
0

If you keep your solution of having each command run in a separate SSH session, you can speed up the time required to start each session by using an already-open connection. This means you only have to pay the connection overhead once, making subsequent sessions quick to start.

The simplest way to set it up is by opening the first connection with this:

ssh -M -S /tmp/ssh_mux_%h_%p_%r myserver

Then open each subsequent connection with:

ssh -S /tmp/ssh_mux_%h_%p_%r myserver "command"

There is more info in the ssh_config manpage under ControlMaster and ControlPath, or online here and here.

0

This answer is similar to one of the other ones above. But I will solve this by wrapping the output of the command in the special SOT and EOT characters using printf. So instead of sending the command, I send (with COMMAND already substituted):

export COMMAND="find ."
printf "%b%s%b" "\002" "`${COMMAND}`" "\003"

and then search for and strip off the SOT and EOT characters when they have both arrived. Essentially, this is the same as the echo "==== Command Output Finished ====" solution, but may be a little bit more robust.

1
  • I believe the canonical abbreviations in ASCII are STX and ETX. EOT is "End Of Transmission", \004. I admit in some sources regarding other character sets I have seen STX, SOT, ETX, EOT where ASCII has SOH, STX, EXT, EOT respectively. Still it seems ASCII nomenclature should be used in Linux. Dec 2, 2020 at 11:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .