3

Is there already a program that reads multiple pipes or file descriptors and writes to the standard output (not splitting lines).

Like cat, but reading all files simultaneously and preserving lines.

Something like:

until(all fds are EOF) {
    select(multiple fds);
    read a line from triggered fd;
    write the line to the stdout;
}

It is needed to avoid coding of select/epoll loops or using multithreading in simple programs. Like "select loop for bash".

Update: I've also created a similar question on unix.SE.

4
  • Please describe what you're trying to accomplish. This almost sounds like an XY problem. Feb 23, 2011 at 18:33
  • @Dennis Williamson, Every time I want to write a oneliner that shares output the problem arises. Program with even loop is sometimes too big to simple one-shot tasks.
    – Vi.
    Feb 23, 2011 at 18:40
  • 1
    Please be more specific. Can you post an example? Feb 23, 2011 at 18:41
  • @Dennis Williamson, I have a stateful program that reads data from /dev/input/whatever and issues HTTP requests and starts other programs. And now I want it to handle multiple input devices (but don't want to think about FD loops or threading). I want solution that can be applied to any of such problems. Like pipe demultiplexor/multiplexor.
    – Vi.
    Feb 23, 2011 at 18:46

3 Answers 3

2

A partial solution could be to use:

tail -fq file1 file2 file3 | grep -v ^$ | yourprogram

I name this a partial solution because the grep removes the empty line that tail adds every times it changes file. If your input contains empty lines, they are removed.

4
  • Clever, a +1 from me.
    – Nifle
    Feb 23, 2011 at 20:14
  • Fails: mkfifo pipe1 pipe2 && tail -fq pipe1 pipe2; then writing to pipe2 hangs.
    – Vi.
    Feb 24, 2011 at 10:38
  • tail -f on multiple pipes works in general bad unless it had read each of them at least once.
    – Vi.
    Feb 24, 2011 at 10:45
  • Also it is slow.
    – Vi.
    Mar 1, 2011 at 15:56
1

Created a program to do just it: fdlinecombine. It reads specified file descriptors and outputs the data linewise.

Checked some corner cases, tested a bit. Hoping it will be useful for somebody too.

1

A alternative using xargs(1) and tail(1):

Create the data sources:

$>touch datafile.txt
$>mkfifo pipe1
$>mkfifo pipe2

Start tail processes with xargs:

$>echo datafile.txt pipe1 pipe2 | xargs -n 1 -P 0 tail -f >> datasink.txt

Push data into sources:

$>echo dataset1 > pipe1
$>echo dataset2 > datafile.txt
$>echo dataset3 > pipe1
$>echo dataset4 > pipe2

Show data from sink:

$>cat datasink.txt 
dataset1
dataset2
dataset3
dataset4

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .