0

I have an python script which gets 1 command line argument

python arg

Now I want a bash-script which:

  • calls python arg & with the argument arg=000
  • make this x-times

so therefore:

python 000 &
python 001 &
...
python 00x &

and if one of these processes finishes the script should fork a new process

python 00x+1 &

until a number N.


Footnotes:

  • I am pretty aware that this sounds like an ugly solution what I am trying to do - but I am forced to do it that way by a various number of reansons, which are out of my domain
  • I know i leave all(!) of the work to community, but all my googling and binging went to zero lines of script and therefore I have to leave it as it is.

EDIT: I need a solution that does not use parallel.

3

It sounds like you want to run multiple instances of the python script in parallel, but only run so many at once and when one finishes, start another. There is a handy utility for that called "parallel", which can be combined with the command "seq" to generate the list of numbers.

 seq -f '%03.0f' 0 N | parallel -j x "python {}"

In the above, replace N with the upper limit. Replace x with the maximum number of scripts to run simultaneously.

seq generates the list of numbers.

$ seq 5
1
2
3
4
5

$ seq 0 5
0
1
2
3
4
5

$ seq -f '%03.0f' 0 5
000
001
002
003
004
005

The last part looks like the numbers sequence that you are wanting to use... 5 being an arbitrary endpoint that you can change.

The parallel -j x "command {}" runs x copies of command simultaneously each time substituting the {} with one of the values from the list generated by seq. So if you have a sequence of 000-100 and you supply -j 3 it will run:

 python 000
 python 001
 python 002

all at the same time. When one of these exits, say "python 001" gets finished first, it will then start "python 003". Suppose "python 000" finishes next, it will substitute "python 004" next and so on until it finishses all 101 jobs.

Obviously you don't want to make -j too high or it will overload your system.

seq is a very standard linux command. parallel is offered as an optional package installable via the package manager for your system. Oddly RedHat based systems don't include it, but you can get source and binaries from the official site:

https://www.gnu.org/software/parallel/

  • Thanks. Out of curiosity, what could be an attempt without parallel. I am not sure, that this is installed on every machine and I do not have the permissions to install it. – user69453 Jun 14 '15 at 19:53
  • Ok as I feared - I need a solution without parallel. – user69453 Jun 15 '15 at 7:39
  • Understood. Before you dismiss parallel entirely however, you should know that once compiled, it is a single executable file that you could store in and run from your project directory. It need not be installed system wide and does not require root permission to install in your own directory. – wojtow Jun 15 '15 at 18:47
  • This post appears to contain a bure bash implementation of what you are wanting: prll.sourceforge.net/shell_parallel.html – wojtow Jun 15 '15 at 18:55
  • 1
    @user69453 GNU Parallel is a perl script. See if your excuse for not installing GNU Parallel is already covered oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html – Ole Tange Jan 28 '16 at 9:37
1

Building on the previous answer, but instead of using GNU parallel, using the standard command "xargs" with option -P option:

seq -f '%03.0f' 0 999 | xargs -n 1 -P 4 python

Try this to see how the processes run in parallel:

seq 0 10 | shuf | xargs -n 1 -P 3 -I % sh -c "echo start %; sleep %; echo done %"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.