Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there already a program that reads multiple pipes or file descriptors and writes to the standard output (not splitting lines).

Like cat, but reading all files simultaneously and preserving lines.

Something like:

until(all fds are EOF) {
    select(multiple fds);
    read a line from triggered fd;
    write the line to the stdout;

It is needed to avoid coding of select/epoll loops or using multithreading in simple programs. Like "select loop for bash".

Update: I've also created a similar question on unix.SE.

share|improve this question
Please describe what you're trying to accomplish. This almost sounds like an XY problem. – Dennis Williamson Feb 23 '11 at 18:33
@Dennis Williamson, Every time I want to write a oneliner that shares output the problem arises. Program with even loop is sometimes too big to simple one-shot tasks. – Vi. Feb 23 '11 at 18:40
Please be more specific. Can you post an example? – Dennis Williamson Feb 23 '11 at 18:41
@Dennis Williamson, I have a stateful program that reads data from /dev/input/whatever and issues HTTP requests and starts other programs. And now I want it to handle multiple input devices (but don't want to think about FD loops or threading). I want solution that can be applied to any of such problems. Like pipe demultiplexor/multiplexor. – Vi. Feb 23 '11 at 18:46

A partial solution could be to use:

tail -fq file1 file2 file3 | grep -v ^$ | yourprogram

I name this a partial solution because the grep removes the empty line that tail adds every times it changes file. If your input contains empty lines, they are removed.

share|improve this answer
Clever, a +1 from me. – Nifle Feb 23 '11 at 20:14
Fails: mkfifo pipe1 pipe2 && tail -fq pipe1 pipe2; then writing to pipe2 hangs. – Vi. Feb 24 '11 at 10:38
tail -f on multiple pipes works in general bad unless it had read each of them at least once. – Vi. Feb 24 '11 at 10:45
Also it is slow. – Vi. Mar 1 '11 at 15:56

A alternative using xargs(1) and tail(1):

Create the data sources:

$>touch datafile.txt
$>mkfifo pipe1
$>mkfifo pipe2

Start tail processes with xargs:

$>echo datafile.txt pipe1 pipe2 | xargs -n 1 -P 0 tail -f >> datasink.txt

Push data into sources:

$>echo dataset1 > pipe1
$>echo dataset2 > datafile.txt
$>echo dataset3 > pipe1
$>echo dataset4 > pipe2

Show data from sink:

$>cat datasink.txt 
share|improve this answer
up vote 0 down vote accepted

Created a program to do just it: fdlinecombine. It reads specified file descriptors and outputs the data linewise.

Checked some corner cases, tested a bit. Hoping it will be useful for somebody too.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .