Does piping executes commands in parallel?

Question

I'm trying to understand piping and redirection more in depth. In every guide it says that in cmd1 | cmd2 cmd1 will execute and stdout will be shared with stdin of cmd2. But that doesn't sit quite good in my head so I tested it:

So it looks like both ./test.sh and grep are running in parallel!

Here is the source code:

#!/bin/bash
a=0
while :
do
        ((a=a+1))
        echo "Line number $a"
        sleep 1
done

Trying it with different commends shows similar results. Does pipe run in parallel?

How pipe and redirection work "under the hood" ?

Answer 1

Yes all the commands run at the same time. In your case the grep starts at the same time as test.sh, but since it reads its stdin, it is blocked until test.sh puts something in it, which will not occur at once. The two processes pace each other, if the second is slow, the first will fill up it stdout output buffer and will be made to wait until the second has read some. If the first is slow, the second is blocked while it reads input if there is none available.

In real life, the stdout of the first process is written in rather large chunks (around 4K)(*), so the second process has nothing to read until the first process has written at lest 4K worth of data. Your test.sh using echo could be a bit different and do line-buffered output.

(*) The rule being that if the output is to a "tty" (ie, terminal) it is line-buffered: a LF in the output causes the actual output to occur. Otherwise, it is buffered by chunks of 4K. The first process can also explicitly suppress all buffering but this is bad for performance.

Answer 2

You're really looking at what's termed flow control, and/or (re)direction -- again; flow control, or stdin, stdout (standard in / standard out).

I hope my above statement helps you better understand what's intended to occur, and what you can better expect.

pipe, or "piping" stdout, merely directs stdout in most cases.

Parallel is really another name for tandem. You can attempt to run 2 different/same commands/outputs at the same time. But expecting a synchronized run would be unwise, as it's unlikely to occur.

To your example; Here you introduce what's called a conditional. Much the same as a road sign, or a traffic light. These conditional(s) direct flow based on a condition. In your example, the while "condition" -- while this, do that. Conditions generally redirect stdin, stdout.

In direct answer to your 2 direct questions

1. Does pipe run in parallel

No. not necessarily. At least not in the normal sense of "parallel" (tandem).

But pipe can fire (initiate) 2 jobs at the same time. Even the same job twice.

2. How pipe and redirection work "under the hood" ?

I hope I've better illustrated that for you above. :)

or

cat ./ASCII | grep somename >./output

The above cat ./ASCII is output (stdout). Which becomes input (stdin) by way of pipe (|) for grep somename. Which is (re)directed (>) to its final output ./output.