Execute script remotely over SSH and disconnect from running process

Question

Although there are couple of similar questions over here I did not manage to find the one that answers my needs.

So what I need is to run a script that will deploy my application on the remote server. So when my CI runs all tests successfully it should run deployment script.

So I obviously wanted for my CI to run that script on the remote machine.

What I found is that I have couple of ways for my CI to run script on remote server:

ssh user@remote_server 'SOME_ENV_VARS=staging_vars bash -s' < deploy.sh

This works almost perfectly.

My deploy script is something like this:

cd project
killall node;
bash start.sh -s & <- THIS IS IMPORTANT
exit;

I also tried:

nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;

and

screen -dmS important_server bash start.sh -s 

The start.sh script is a never ending process (it starts server). It works normally but it never disconnects from ssh so my CI cannot finish the build.

The the best I actually managed to do is with Gaurav's proposal. The process did remain in background after termination of ssh connection. But I still needed to terminate it manually because my CI job was hanging :(

I must have done something wrong.

Answer 1

So basically after 40 or so hours of trials and failures I did manage to find solution. Shadowcoder's and Gaurav's answers were helpful indeed but to no avail.

So basically I have no idea why would this answer work, but it does. The problem lies somewhere in the fact that I was running a bash script that existed on one machine, via SSH, on other machine. That script was calling another script and had to exit SSH while leaving the remote script running. It sounds complicated but actually it looks reasonable to me. At least it sounded reasonable when I started writing it.

Let's further explain things (didn't seam important when I was writing the question).

My CI/CD software was running builds when detecting a push to master branch. On successful build it should call deploy.sh script automatically in order to post changes to staging server. Fairly standard. That script tells remote server to pull changes, stop server and restart it again. Obviously server can be started in 3 modes (staging, production and development) with single script that is called start.sh.

So the deployment script's main goal is to stop server, pull changes and start the starting script again.

So at first I tried to use nohup.

My deploy.sh script was made like this

cd project
killall node;
nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;

Looking at my CI logs I saw that no matter what I do, server actually runs smoothly, and I can see my changes on the staging server, however this script never ends! SSH connection remains open forever. Now strange things happened. When closing it manually in CI the server remained active, which was the desired behavior - but I did not want to have to manually intervene whenever build is triggered. I did notice one peculiar thing when looking at my CI logs. It was as if the script was waiting for ANOTHER command after exit. So in desperation I added one more exit. My script was now

cd project
killall node;
nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;
exit; 

This did work in fact but for some reason the start.sh process was killed when SSH was exited. Back to the square one.

Couple of (more like ten) of hours later. I noticed that screen solution (although not working at start) that Shadowcoder proposed might be the way to go though.

My start.sh script was doing boot-up of my server. That included database migrations, seeding of initial data, building the UI, doing setup of in-memory cache on tmpfs and eventually it was calling node index.js. Since I was trying and failing to reproduce the issue so I could update the question for Shadowcoder's information I tried to screen only the last command (node index.js).

That is how I came up to solution.

deploy.sh script looks like this

./start.sh -s;

exit;
exit; #I have no idea why the hell I need 2 exits but it works

start.sh script looks like hell but the important thing is that

screen -dmS server_name node index.js &

Please notice that I still had to end the command with & It just would not work without it, have literally no idea why.

So the solution seams hackish, and I am still sure I missed something and that things do not have to be this way, but this does work. Not sure why it does work or why it did not work before, but if this can save someone else's time - the solution is here. I would also like to know the explanation for this behavior.

Answer 2

Screen

Install it with apt install screen or from the site

After connecting to the SSH server:

# screen -q
...
# bash ./start.sh -s
...

You can detach from screen with Ctrl+A+D

Or, to have a solution as close as possible to the original, edit deploy.sh to use screen:

cd project
killall node;
screen -dmS important_server bash start.sh -s # No need to put screen in the background.
exit;

And your ssh command you are using should work with it.

ssh user@remote_server 'SOME_ENV_VARS=staging_vars bash -s' < deploy.sh

Answer 3

To execute program in background and exit please use nohup. nohup will make sure program is running after ssh session terminates.

nohup ./start.sh < /dev/null >/dev/null 2>&1 & 

Receiving and Sending Input/output from /dev/null will help if your program need any input or print output to console.