1

I have a series of server machines which I want to run the same command on. Each command takes hours and (even though I am running the commands using nohup and setting them to run in the background) I have to wait for each to finish before the next starts. Here is roughly how I have set it up:

On the host machines:

for i in {1..9}; do ssh RemoteMachine${i} ./RunJobs.sh; done

Where RunJobs.sh on each remote machine is:

 source ~/.bash_profile
cd AriadneMatching
for file in FileDirectory/Input_*;
do
    nohup ./Executable ${file} &
done
exit

Does anyone know of a way such that I dont have to wait for each job to finish before the next starts? Or alternatively a better way of doing this, I have a feeling what I am do is fairly sub-optimal. Cheers, Jack

4
  • maybe try sticking an ampersand somewhere.. superuser.com/questions/177218/… the term might be background process Some more ideas here that may be relevant stackoverflow.com/questions/29142/…
    – barlop
    Oct 20, 2013 at 21:46
  • I already have that with the actual job executions but you think maybe the './RunJobs.sh' lines needs to have an ampersand after it too?
    – JMzance
    Oct 20, 2013 at 21:50
  • i'm very ignorant on this and I didn't notice you'd already done the ampersand.
    – barlop
    Oct 20, 2013 at 22:07
  • for i in {1..9}; do nohup ssh RemoteMachine${i} ./RunJobs.sh& done. It's not very robust, but it's a quick-and-dirty if that works for you. Oct 20, 2013 at 23:58

4 Answers 4

0

Solution for the local machine

Based on parallel-ssh

# pssh -P --par 2 --hosts RemoteMachines /opt/RunJobs.sh

or:

# pssh -i --par 2 --hosts RemoteMachines /opt/RunJobs.sh

Explanation of the parameters:

-P
--print
    Display output as it arrives.  This option is of limited usefulness
    because output from different hosts are interleaved.

-i
--inline
    Display standard output and standard error as each host completes.

-p parallelism
--par parallelism
    Use the given number as the maximum number of concurrent connections.

-h host_file
--hosts host_file
    Read hosts from the given host_file.

Based on ansible

# ansible --forks 2 -i RemoteMachines '*' -m command -a /opt/RunJobs.sh

Explanation of the parameters:

-f NUM, --forks=NUM
    Level of parallelism.  NUM is specified as an integer, the default is 5.

-i PATH, --inventory=PATH
    The PATH to the inventory hosts file, which defaults to /etc/ansible/hosts.

-m NAME, --module-name=NAME
    Execute the module called NAME.

-a 'ARGUMENTS', --args='ARGUMENTS'
    The ARGUMENTS to pass to the module.

The command module takes the command name followed by a list of space-delimited arguments. The given command will be executed on all selected nodes. It will not be processed through the shell, so variables like $HOME and operations like "<", ">", "|", and "&" will not work.

You can read more in the Introduction To Ad-Hoc Commands.

N.B. ansible will not switch to the next group of hosts until all the current hosts ("forks") are done, so its parallelism is lower than of pssh (there might be a way to increase it, but I don't know it).

The RemoteMachines file looks something like this for both cases:

[email protected]
[email protected]
[email protected]
[email protected]

Solution for the remote machines

Rewrite RunJobs.sh into something like this:

find FileDirectory -name 'Input_*' -print0 | xargs -0 -P 2 -n 1 ./Executable

Explanation:

-0, --null
       Input items are terminated by a null character instead of by
       whitespace, and the quotes and backslash are not special (every
       character is taken literally).  Disables the end of file string,
       which is treated like any other argument.  Useful when input items
       might contain white space, quote marks, or backslashes.  The GNU find
       -print0 option produces input suitable for this mode.

-P max-procs, --max-procs=max-procs
       Run  up to max-procs  processes at a time; the default is 1.  If
       max-procs is 0, xargs will run as many processes as possible at a
       time.  Use the -n option or the -L option with -P; otherwise chances
       are  that  only  one  exec will be done.

 -n max-args, --max-args=max-args
       Use at most max-args arguments per command line.  Fewer than
       max-args arguments will be used if the size (see the -s option) is
       exceeded, unless the -x option is given, in which case xargs will
       exit.

nitro2k01's solution based on GNU Parallel is more powerful, but as you can see, GNU xargs isn't too bad either.

2

Firstly, you may want to look at using screen instead of nohup for making the session survive a disconnection. screen gives you a virtual terminal you can return to later. The basic usage is screen yourcommand to execute yourcommand and screen -DR to automagicalliy connect to an existing session, or create a new one if none exists. Just running screen without an argument also gives you a prompt inside a "screen" that you can use.

Secondly, you may want to look into GNU Parallel for parallel jobs. You may not want to have more processes running than you have CPU cores, because of diminishing returns and disk overload. A suitable command for you might be as follows, which, again, you would have to run inside a screen in order to make it survive a disconnect.

ls FileDirectory/Input_* | parallel -j 8 --workdir $PWD ./Executable {}

The files you want to process are listed and piped into parallel which is instructed to run 8 parallel jobs and work in the current directory.

Since screen doesn't understand pipes and other things in the commands given as an argument - that's the shell's job - you would need to either put the command in shell script or give a sh -c command to execute the command.

screen sh -c 'ls FileDirectory/Input_* | parallel -j 8 --workdir $PWD ./Executable {}'

or simply

screen ./runjob

where runjob contains:

#!/usr/bin/env bash
ls FileDirectory/Input_* | parallel -j 8 --workdir $PWD ./Executable {}
3
  • Thanks but parallel isnt installed on any of the machines (and I dont have sudo rights!)
    – JMzance
    Oct 20, 2013 at 22:40
  • Minor suggestion: find FileDirectory -name 'Input_*' -print0 | parallel -0 -j 8 --workdir $PWD ./Executable {} would be a bit safer than the ls. Oct 21, 2013 at 6:52
  • 1
    @JackMedley Please elaborate if you reason is not covered by oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html
    – Ole Tange
    Oct 21, 2013 at 8:16
1

Use screen on each machine. screen starts a command (generally a shell) whch can be detached from your ssh session, so that it continues executing the job just started inside it; in other words, the job, though still running, will not be interrupted by any interrupt (not just kill -9) as you log out.

When you feel like it, you can reconnect via ssh, and re-join the detached session via the command

screen -r

and this will seamlessly put you back into the shell you abandoned, showing you all standard output which you missed in the meantime, if any.

Like this, you don't have to wait for anything, nor do you have dangerously hanging ssh sessions which, if disconnected accidentally, would kill your jobs.

screen is in the repos.

0

for i in {1..9}; do ssh RemoteMachine${i} ./RunJobs.sh; done

try this

for i in {1..9}; do echo RemoteMachine${i}; done |xargs -I % -n 1 -P 10 ssh
% "./RunJobs.sh"

what that does is build a list of machine names and pipes the whole list to xargs. The arguments to xargs mean:

-I % replace the occurrence of % with the first argument

-n 1 use 1 argument per command line

-P 10 run 10 processes at a time

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .