gnu parallel: forward all of stdin to all processes

Question

I'm trying to process input in different ways in parallel, writing out the resulting lines as they come in. My current approach is this:

# process_parallel.sh
read input
parallel --colsep ' ' --linebuffer 'echo $input | {}' \
 ::: 'python process1.py' ./process2.sh ./process3

The context boils down to something like this:

(sleep 1; echo "short input arriving late") | ./process_parallel.sh | ./collate_results.sh

The above works but has one important drawback: The processes are not started until the input has been read. I want to start them immediately since that may take a while, then have parallel pipe its full stdin to each of them.

How do I achieve that?

Answer 1

Quite a bit of work has been put into GNU Parallel to make it not start a new job unless there is data to run on. This is due to some programs failing hard if they do not getting any input. So you will be working against GNU Parallel's design.

To duplicate the input you can use tee and process substitution:

cat namedpipe_or_file | tee >(process1) >(process2) >(process3) >/dev/null

It will start process1, process2, and process3 immediately. The output can, however, be mixed, so if the output must be used, you should redirect it to different files:

cat namedpipe_or_file | tee >(process1 > out1) >(process2 > out2) >(process3 > out3) >/dev/null

Answer 2

Have two wrapper scripts: first, modify process_parallel.sh so that each of the processes is instructed to read its input, from three files, say file1, file2, file3.

Now write a second wrapper script which sends in parallel its standard input to the three files, something like:

  #!/bins/bash
  #parallelise input
  # let's call this file parallelise
  parallel -j 3 -- "echo $1 > file1" "echo $1 > file2" "echo $1 file3"

Now you start your stuff with:

  ./process_parallel_sh; cat InputFile | parallelise; 

Answer 3

Parallel has a --tee option, which can be used like this:

(sleep 1; echo "short input arriving late") |\
parallel --colsep ' ' --linebuffer --pipe --tee ::: 'python process1.py' ./process2.sh ./process3 |\
./collate_results.sh

See GNU "parallel" how to achieve tee behaviour