2

So in Linux, I want to run a script 4 times, then do stuff to the output.

I can do:

script 1 > output1.txt
script 2 > output2.txt
script 3 > output3.txt
script 4 > output4.txt
cat output1.txt output2.txt output3.txt output4.txt > output5.txt
sort output5.txt | uniq -u 

But I'd like the first four to run in parallel (faster). If I put them in the background, then it'll jump to the fifth line, which will not be successful until the first four finish.

EDIT: Thanks for responses. I tried a few of these, but I'm not sure if they will work, because I need (a) 4 different scripts to run in parallel, not the same one four times, and (b) the outputs of each script need to be saved to a .txt file (this is very important). I've updated the above to reflect this.

People in my office love using xargs, so that would be my preference (though I'm open to anything).

Thanks!

2
3

The wait bash's builtin will do the trick...

script 1 > output1.txt &
script 2 > output2.txt &
script 3 > output3.txt &
script 4 > output4.txt &
wait
cat output1.txt output2.txt output3.txt output4.txt > output5.txt
sort output5.txt | uniq -u 
1
  • HA! Simple enough, but all I needed. Thanks! – Alex Oct 29 '13 at 23:35
2

With moreutils' parallel:

parallel -- script-1 script-2
cat out1 out2 out3 out4

The second line will be run once all jobs from parallel have exited. If your script contains arguments, you need to quote it, e.g.

parallel -- 'script-1 argument1' script-2

This can also be done with GNU Parallel but I don't have it on my system due to obvious naming conflicts. I think you only need to make a list of scripts and then:

cat list-of-scripts > parallel -j4
cat out1 out2 out3 out4

The -j4 means parallel will execute on 4 cores if possible.

1

I recommend using GNU parallel. There is a convenient tutorial for it here. It could either allow you to run all four scripts in parallel, and then run the fifth command, or you could set it up to pipe the out from the first four into the fourth, running all five in parallel. Here is the command to run all four in parallel:

parallel ::: script-1 script-2 script-3 script-4

You can pipe the output directly into the fifth command. By default, parallel will group output from each command, e.g. if script-2 completes first the output stream will begin with the entirety of script-2's output, followed by that of the next command to complete. Use --line-buffer if you don't care about output lines from different commands mixing. Use --keep-order if you need the the output from the commands kept in order, i.e. the output from script-1, then script-2, etc.

Another option is to just use xargs, which should already be installed on your system. xargs has a -P option that allows you to specify the number of concurrent processes. Note that it doesn't handle grouping the output, so you can't pipe directly into the fifth command. Usually you need to set -n as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.