0

Although there are couple of similar questions over here I did not manage to find the one that answers my needs.

So what I need is to run a script that will deploy my application on the remote server. So when my CI runs all tests successfully it should run deployment script.

So I obviously wanted for my CI to run that script on the remote machine.

What I found is that I have couple of ways for my CI to run script on remote server:

ssh user@remote_server 'SOME_ENV_VARS=staging_vars bash -s' < deploy.sh

This works almost perfectly.

My deploy script is something like this:

cd project
killall node;
bash start.sh -s & <- THIS IS IMPORTANT
exit;

I also tried:

nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;

and

screen -dmS important_server bash start.sh -s 

The start.sh script is a never ending process (it starts server). It works normally but it never disconnects from ssh so my CI cannot finish the build.

The the best I actually managed to do is with Gaurav's proposal. The process did remain in background after termination of ssh connection. But I still needed to terminate it manually because my CI job was hanging :(

I must have done something wrong.

  • You should really deploy with Ansible and run the thing as a service. Why the -s in bash start.sh -s? – xenoid Apr 15 '19 at 20:36
  • I cannot use Ansible. – Rouz Apr 15 '19 at 22:58
  • Or to make things even more clear - I do not want to use Ansible. I want to deploy like this. This must be possible, and I would like a solution. Of course, there is a question of whether I should use it like this but that is not the topic here. the -s flag means that the server is started in staging mode. Not sure why that makes any difference though @xenoid – Rouz Apr 15 '19 at 23:09
1

So basically after 40 or so hours of trials and failures I did manage to find solution. Shadowcoder's and Gaurav's answers were helpful indeed but to no avail.

So basically I have no idea why would this answer work, but it does. The problem lies somewhere in the fact that I was running a bash script that existed on one machine, via SSH, on other machine. That script was calling another script and had to exit SSH while leaving the remote script running. It sounds complicated but actually it looks reasonable to me. At least it sounded reasonable when I started writing it.

Let's further explain things (didn't seam important when I was writing the question).

My CI/CD software was running builds when detecting a push to master branch. On successful build it should call deploy.sh script automatically in order to post changes to staging server. Fairly standard. That script tells remote server to pull changes, stop server and restart it again. Obviously server can be started in 3 modes (staging, production and development) with single script that is called start.sh.

So the deployment script's main goal is to stop server, pull changes and start the starting script again.

So at first I tried to use nohup.

My deploy.sh script was made like this

cd project
killall node;
nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;

Looking at my CI logs I saw that no matter what I do, server actually runs smoothly, and I can see my changes on the staging server, however this script never ends! SSH connection remains open forever. Now strange things happened. When closing it manually in CI the server remained active, which was the desired behavior - but I did not want to have to manually intervene whenever build is triggered. I did notice one peculiar thing when looking at my CI logs. It was as if the script was waiting for ANOTHER command after exit. So in desperation I added one more exit. My script was now

cd project
killall node;
nohup ./start.sh -s < /dev/null >/dev/null 2>&1 &
exit;
exit; 

This did work in fact but for some reason the start.sh process was killed when SSH was exited. Back to the square one.

Couple of (more like ten) of hours later. I noticed that screen solution (although not working at start) that Shadowcoder proposed might be the way to go though.

My start.sh script was doing boot-up of my server. That included database migrations, seeding of initial data, building the UI, doing setup of in-memory cache on tmpfs and eventually it was calling node index.js. Since I was trying and failing to reproduce the issue so I could update the question for Shadowcoder's information I tried to screen only the last command (node index.js).

That is how I came up to solution.

deploy.sh script looks like this

./start.sh -s;

exit;
exit; #I have no idea why the hell I need 2 exits but it works

start.sh script looks like hell but the important thing is that

screen -dmS server_name node index.js &

Please notice that I still had to end the command with & It just would not work without it, have literally no idea why.

So the solution seams hackish, and I am still sure I missed something and that things do not have to be this way, but this does work. Not sure why it does work or why it did not work before, but if this can save someone else's time - the solution is here. I would also like to know the explanation for this behavior.

0

Screen

Install it with apt install screen or from the site

After connecting to the SSH server:

# screen -q
...
# bash ./start.sh -s
...

You can detach from screen with Ctrl+A+D

Or, to have a solution as close as possible to the original, edit deploy.sh to use screen:

cd project
killall node;
screen -dmS important_server bash start.sh -s # No need to put screen in the background.
exit;

And your ssh command you are using should work with it.

ssh user@remote_server 'SOME_ENV_VARS=staging_vars bash -s' < deploy.sh

  • Well no. I cannot detach from screen This does not work. -dmS flag is throwing the error. Use: screen [-opts] [cmd [args]] or: screen -r [host.tty] – Rouz Apr 15 '19 at 22:59
  • What version of screen are you using? I am using 4.05.00 – Shadowcoder Apr 15 '19 at 23:22
  • Screen version 4.06.02 (GNU) 23-Oct-17 – Rouz Apr 16 '19 at 8:23
  • What exactly are you doing that leads to screen throwing an error? I just compiled that version (just in case something changed; however unlikely) but screen -dmS important_server bash $A_LONG_RUNNING_BASH_SCRIPT -s works perfectly fine... – Shadowcoder Apr 16 '19 at 15:38
  • the throwing of this error was not reproducible for me every time. And that made things even stranger. The actual solution is quite ridiculous and feels hackish, so I will post it so someone can maybe post wth is actually going on. – Rouz Apr 16 '19 at 20:23
0

To execute program in background and exit please use nohup. nohup will make sure program is running after ssh session terminates.

nohup ./start.sh < /dev/null >/dev/null 2>&1 & 

Receiving and Sending Input/output from /dev/null will help if your program need any input or print output to console.

  • this works in the sense that it will continue working after ssh session terminates of course. However this is not my issue. the issue actually is that ssh is not terminated. That leaves build task to hang. If i manually kill ssh the process will remain running, but this makes no sense from the CI/CD perspective – Rouz Apr 16 '19 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.