1

I'm trying to understand piping and redirection more in depth. In every guide it says that in cmd1 | cmd2 cmd1 will execute and stdout will be shared with stdin of cmd2. But that doesn't sit quite good in my head so I tested it:

So it looks like both ./test.sh and grep are running in parallel!

Here is the source code:

#!/bin/bash
a=0
while :
do
        ((a=a+1))
        echo "Line number $a"
        sleep 1
done

Trying it with different commends shows similar results. Does pipe run in parallel?

How pipe and redirection work "under the hood" ?

  • “In every guide it says that in cmd1 | cmd2 cmd1 will execute and stdout will be shared with stdin of cmd2.” But that doesn’t make any statement about whether the commands will run at the same time or consecutively...? – Daniel B Jul 2 '19 at 19:38
  • @DanielB Exactly, I found it hard to imagine, therefore my question... – Adam Katav Jul 2 '19 at 19:42
3

Yes all the commands run at the same time. In your case the grep starts at the same time as test.sh, but since it reads its stdin, it is blocked until test.sh puts something in it, which will not occur at once. The two processes pace each other, if the second is slow, the first will fill up it stdout output buffer and will be made to wait until the second has read some. If the first is slow, the second is blocked while it reads input if there is none available.

In real life, the stdout of the first process is written in rather large chunks (around 4K)(*), so the second process has nothing to read until the first process has written at lest 4K worth of data. Your test.sh using echo could be a bit different and do line-buffered output.

(*) The rule being that if the output is to a "tty" (ie, terminal) it is line-buffered: a LF in the output causes the actual output to occur. Otherwise, it is buffered by chunks of 4K. The first process can also explicitly suppress all buffering but this is bad for performance.

| improve this answer | |
  • How can the second command run with input of less than 4K? I'm pretty sure that "ls -l /dev" for example doesn't write 4K to HDD and only then "sort" will run... Also what happens when both commands try to modify the same file? – Adam Katav Jul 2 '19 at 19:50
  • 1
    When the first command has nothing more to write, it flushes and closes the stdout handle, so everything is written then. So yes, you can have less than 4K of data, but in that case most of the processing in the second process occurs after the first has ended. In most cases you cannot have two processes opening the same file for writing, the default for "write" open is to get exclusive write access. – xenoid Jul 2 '19 at 19:55
  • A pipe isn't a random-access file... If more a FIFO queue. One process can only add things at one end, while the other can only read at the other end. The only edge cases are queue full or empty. – xenoid Jul 2 '19 at 20:03
1

You're really looking at what's termed flow control, and/or (re)direction -- again; flow control, or stdin, stdout (standard in / standard out).

I hope my above statement helps you better understand what's intended to occur, and what you can better expect.

pipe, or "piping" stdout, merely directs stdout in most cases.

Parallel is really another name for tandem. You can attempt to run 2 different/same commands/outputs at the same time. But expecting a synchronized run would be unwise, as it's unlikely to occur.

To your example; Here you introduce what's called a conditional. Much the same as a road sign, or a traffic light. These conditional(s) direct flow based on a condition. In your example, the while "condition" -- while this, do that. Conditions generally redirect stdin, stdout.

In direct answer to your 2 direct questions

  1. Does pipe run in parallel

No. not necessarily. At least not in the normal sense of "parallel" (tandem).

But pipe can fire (initiate) 2 jobs at the same time. Even the same job twice.

  1. How pipe and redirection work "under the hood" ?

I hope I've better illustrated that for you above. :)

or

cat ./ASCII | grep somename >./output

The above cat ./ASCII is output (stdout). Which becomes input (stdin) by way of pipe (|) for grep somename. Which is (re)directed (>) to its final output ./output.

| improve this answer | |
  • If pipe initiate 2 jobs at the same time, is it not "in parallel"? And do both commands share the same file only one calls it stdin and the other stdout? – Adam Katav Jul 2 '19 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.