23

Here is a command that sorts files in a folder in reverse order

ls | sort -r

What does the | symbol in that command do?

What I'm really looking for here is high level (easy to understand) explanation of pipes for Linux beginners. I see other questions about pipes here on Superuser, but nothing that elicits an answer that explains in simple terms what they do and how they differ from redirection (the > or < symbol).

4
  • 7
    This has nothing to do with Linux (which is a kernel). Pipes in general are a means of re-directing input/output, in a shell like bash it is no different. The only thing special about a | is that it does not use a name, output from the l-hand command is passed directly to the input for the command on the r-hand side of the pipe. May 19, 2014 at 19:25
  • For a history-lesson on the subject, read linfo.org/pipe.html May 20, 2014 at 7:56
  • ls -1r (Note the number one argument) should produce similar result to ls | sort -r.
    – Ivan Chau
    May 20, 2014 at 8:44
  • I like to explain it that way: A pipe takes the output of one command and makeing it usable for a following command. For example, you can do cat /somefile | grep cool. This will take the output of somefile and make it available for grep, and then grep will print all lines with the word cool in it.
    – user529857
    Apr 22, 2017 at 9:30

5 Answers 5

29

The following is simplified a bit to help new users.

Well, first, it's necessary to understand the concept of standard input and standard output.

In Linux and other UNIX-like operating systems, each process has a standard input (stdin) and a standard output (stdout). The usual situation is that stdin is your keyboard and stdout is your screen or terminal window.

So when you run ls, it will throw it's output to stdout. If you do nothing else, it will go to your screen or terminal window, and you will view it.

Now, some Linux commands interact with the user, and use stdin to do that, your text editor being one of those. It reads from stdin to accept your keystrokes, do things, and then writes stuff to stdout.

However, there are also non-interactive or "filter" commands that do NOT work interactively, but want a bunch of data. These commands will take everything stdin has, do something to it, and then throw it to stdout

Let's look at another command called du - stands for disk usage. du /usr, for example, will print out (to stdout like any other Linux command) a list of every file in that directory and it's size:

# du /usr
2312    /usr/games
124     /usr/lib/tc
692     /usr/lib/rygel-1.0
400     /usr/lib/apt/methods
40      /usr/lib/apt/solvers
444     /usr/lib/apt
6772    /usr/lib/gnash

As you can tell right off the bat, it isn't sorted, and you probably want it sorted in order of size.

sort is one of those "filter" commands that will take a bunch of stuff from stdin and sort it.

So, if we do this:

# du /usr | sort -nr

we get this, which is a bit better:

4213348 /usr
2070308 /usr/lib
1747764 /usr/share
583668  /usr/lib/vmware
501700  /usr/share/locale
366476  /usr/lib/x86_64-linux-gnu
318660  /usr/lib/libreoffice
295388  /usr/lib/vmware/modules
290376  /usr/lib/vmware/modules/binary
279056  /usr/lib/libreoffice/program
216980  /usr/share/icons

And you can now see that the "pipe" connects the stdout of one command to the stdin of another. Typically you will use it in situations like this where you want to filter, sort or otherwise manipulate the output of a command. They can be cascaded if you want to process output through multiple filter-type commands.

If you type sort by itself, it will still try to read from stdin. Since stdin is connected to your keyboard, it will be waiting for you to type, and process things until you press Control-D. It won't prompt you since it's not really meant to be used interactively.

It's possible for a program to tell whether stdin is interactive or not, so some programs may act differently if you issue them by themselves or at the end of a pipe.

Also, piping a program that only works interactively, like vi, will result in you having a bad time.

Pipes are different from redirection in that the data shuffled from one command to the next without being stored anywhere. So, In the above example, du's output is not stored anywhere. The majority of the time you don't want this with pipes because the reason to use pipes is to process the output of a command in some way - but, there is a command tee that lets you have your cake and eat it too, it will copy what it receives from stdin to both stdout and a file of your choosing. You can also likely do this from bash with some arcane syntax involving ampersands and brackets that I don't know about.

6
  • Note that this isn't unique to Linux, or even POSIX. Most (all?) shells on Windows do this too. And probably other OSes.
    – Bob
    May 20, 2014 at 6:51
  • I know the concept of stdin and stdout is different under Windows than it is under Linux, though probably not very much from the point of view of the Windows cmd.exe or Powershell.
    – LawrenceC
    May 20, 2014 at 14:29
  • I'm pretty curious on how it's different - would you mind explaining? Maybe in chat if comments aren't a good place for it.
    – Bob
    May 20, 2014 at 14:50
  • 1
    Win32 programs most definitely do have a standard input and output. For example, you can retrieve your process' standard input, output or error handle with the Win32 function GetStdHandle(). It's also trivial to redirect a spawned [child] process' standard streams with .NET, which I believe maps to Win32 functions (but I'm not 100% sure of that - I'm not a Win32 dev).
    – Bob
    Jul 16, 2014 at 11:56
  • 1
    Ah, here's the Win32 equivalent, by setting the appropriate parameters in the STARTUPINFO struct for CreateProcess().
    – Bob
    Jul 16, 2014 at 11:58
30

If you're comfortable with output and input redirection, the explanation is really quite easy.

Command1 | Command2

does the same as

Command1 > tempfile
Command2 < tempfile

but without tempfile. The output of Command1 is directly connected to the input of Command2 and the transfer happens in-memory.

4
  • I could be wrong but I think the tempfile exists even in the pipe syntax. It just doesn't have a name.
    – Taemyr
    May 20, 2014 at 11:14
  • 3
    No it does not. No file system operations are involved when piping output from one command to another command’s input.
    – Daniel B
    May 20, 2014 at 11:17
  • 1
    Though, under DOS (and most likely Windows), a temp file is created by the pipe. Not *nix, but worth nothing the difference. May 20, 2014 at 14:20
  • I’m pretty sure this is not correct. Process Monitor reports no CreateFile or WriteFile calls to back your claim. /edit: That’s for the Windows part, of course.
    – Daniel B
    May 20, 2014 at 15:03
4

The pipe character (|) connects the output of one program to the input of another.

In this example echo prints the word hello, and wc -c does a character count of its input (including the newline):

echo hello | wc -c
6
2
  • I think you should say at once that echo will output "hello\n". You're not telling the OP to study, you're telling this to everybody who reads your answer. Why multiply this waste of time?
    – Rodrigo
    Mar 16, 2018 at 22:42
  • Thanks for the feedback, I shortened my answer. Mar 16, 2018 at 23:52
3

Really if you want to know what pipes do and the difference between > and |, then go to a directory with a lot of files, and

from a terminal ls vs ls | more (or doing that from Windows with DIR and DIR | MORE)

If you used > more you'll see it creates a file called 'more' rather than sending the output of ls to the 'more' command. So if somebody did >more it'd probably a mistake, one wouldn't do >more you'd do >file1. More is a well known command.

The < like the > is also for linking a command and a file, rather than a command to a command. But while > sends the output of a command to a file, the < sends a file as input to a command. I rarely use < as I usually use cat file1 | to send the output of a file to a command.

$ grep a < file1 abc

$ cat file1 | grep a abc

grep with 2 parameters is of the form grep pattern file. grep with one parameter is grep pattern. And you can send it the file by piping the contents of the file to it, or by using <. If using <, you write the command name first, then the filename after so command < file. If using | to pipe the contents of a file, you use cat file1 | command.

Also many commands take a file as input anyway so grep a file1 will work, just as cat file1 | grep a, and grep a < file1.

I was doing pipes (|) and > on DOS even 15 years ago.

To summarize how | differs from < and > - The pipe sits between 2 commands The < and > sit between a command and a file. The > is output to a file. The < is input from a file.

2

In order to understand this, try it yourself:

sort -r

Now you're hanging with a cursor, and it's not doing anything. What happens if you type in some data?

1
2
3
5
4

Still nothing, right? Now press ctrl+D

5
4
3
2
1

So what sort does is, it takes input (what you typed), does something with it (sorts) and gives it back as output. The ls command does not take input, it only generates output. The pipe symbol takes the output from ls, and feeds it as input to the sort command.

> does not feed output to a program, but stores the output as a file. < uses a file as input.

Not the answer you're looking for? Browse other questions tagged .