Execute two or more commands simultaneously in shell script

Question

It is often very helpful to run commands in parallel in a shell script, but I cannot find ways to do it. Is this possible? If so, how would I achieve that? I'm mainly interested in doing that for fish.

It is useful when I'm running commands that take a long time to complete but can or need to be run at the same time to decrease the time it takes for everything to finish.

Answer 1

The most basic approach is with &, this other answer is right. It also advocates nohup but nohup redirects stdin, stdout and stderr, something you may or may not want. Read Difference between nohup, disown and & and make an educated decision.

Another approach is parallel. It will be useful if commands you want to parallelize are similar to one another and you can craft a pattern.

Basic variant of the tool (from moreutils, at least in Debian) allows you to limit the number of jobs that are run at the same time. GNU parallel is more advanced. If jobs you want to run generate output then the following options will be particularly useful:

--group
Group output. Output from each job is grouped together and is only printed when the command is finished. Stdout (standard output) first followed by stderr (standard error). […]

^(source)

(--group is enabled by default, so usually you don't need to use it explicitly.)

--keep-order
-k
Keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes. […] -k only affects the order in which the output is printed - not the order in which jobs are run.

^(source)

With them the output from multiple jobs will be organized, something you cannot get from &. Sometimes you may not care about output but still care about sequence; like in this answer of mine where GNU parallel is used to parallelize multiple curl processes and get the exit status from each, retaining the sequence.

In Debian GNU parallel is in a package named parallel. As a separate executable parallel can be run from any shell.

Answer 2

The best way to do this is to combine nohup prefacing the command with & appended to the end of the command

Unclear what your core goal is, but I hope the helps!

To run any command as a background process on a Linux machine, you should prepend the command with nohup and append & to the end.

So the final command would be:

nohup [your command] &

The nohup means the command should ignore “hang ups” and the ampersand & appended to it is a shell command that tells the system to run the command as background process. More information on it’s usage can be found here.

When you run a command like this, the process will run, you will be sent back to the command prompt and you can exit the terminal session or even go and do something else unrelated to that command during that terminal session.