Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I sometimes sit with a bunch of output fifos from programs that run in parallel. I would like to merge these fifos. The naïve solution is:

cat fifo* > output

But this requires the first fifo to complete before reading the first byte from the second fifo, and this will block the parallel running programs.

Another way is:

(cat fifo1 & cat fifo2 & ... ) > output

But this may mix the output thus getting half-lines in output.

When reading from multiple fifos, there must be some rules for merging the files. Typically doing it on a line by line basis is enough for me, so I am looking for something that does:

parallel_non_blocking_cat fifo* > output

which will read from all fifos in parallel and merge the output on with a full line at a time.

I can see it is not hard to write that program. All you need to do is:

  1. open all fifos
  2. do a blocking select on all of them
  3. read nonblocking from the fifo which has data into the buffer for that fifo
  4. if the buffer contains a full line (or record) then print out the line
  5. if all fifos are closed/eof: exit
  6. goto 2

So my question is not: can it be done?

My question is: Is it done already and can I just install a tool that does this?

share|improve this question
add comment

1 Answer 1

up vote 0 down vote accepted

This solution will only work if the number of fifos are fewer than the number of jobs that GNU parallel can run in parallel (which is limited by file handles and number of processes):

parallel -j0 --line-buffer cat ::: fifo*

It seems to be able to move up to 500 MB/s:

window1$ mkfifo {1..100}
window1$ parallel -j0 --line-buffer cat ::: {1..100} | pv >/dev/null

window2$ parallel -j0 'cat bigfile > ' ::: *
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.