45

From this commandlinefu post:

$ diff <(wget -q -O - URL1) <(wget -q -O - URL2)

Seems like multiple standard inputs but---if I remember my Linux Porgramming course correctly---this can't be it. I thought that, by definition, standard input was one stream.

Perhaps this has zilch to do with stdin? Can someone explain this to me; maybe provide links to documentation.

Bonus: How would a script handle these multiple streams? If someone could provide a Python or Perl example, that would be very helpful.

Improve this question
2

3 Answers 3

Reset to default
37

This isn't multiple standard input. This is a bash'ism that called 'Process Substitution' http://tldp.org/LDP/abs/html/process-sub.html

It creates a pseudo file (/dev/fd/something) for each substitution. It's pretty useful. The command can only read as a stream, meaning it can not go back and forth with fseek. It needs to read it as a stream of bytes, like a pipe.

BONUS Answer

You don't need to do too much to use this. As far as your script is concerned, it gets a valid filename on the command line, that can be open()ed like anything else. As others have said, you'd see diff /dev/fd/XX /dev/fd/YY. If you do a stat() on any of these pseudo-files, you'll see it's a named pipe, and you should treat it with pipe semantics - namely no fseek() or ftell(). If you do a stat() test to explicitly see if it's a file (e.g. [ -f $1 ]) this will break - this is implemented as a named pipe after all.

Improve this answer
3
  • Another good reference for bash process substitution - wiki.bash-hackers.org/syntax/expansion/proc_subst
    – studgeek
    Apr 28, 2017 at 3:00
  • And what is the recommended usage of this feature? Support it whenever your program could reasonably work with a stream instead of requiring a file with random access?
    – masterxilo
    Apr 15, 2019 at 18:40
  • @masterxilo You don't always want to store data on disk -- and although nothing, on Unix or other POSIX-like systems we may have today, mandates a file is necessarily created "on disk" (persistent storage) -- for most people most files typically end up on disk, while you want to avoid to use it for temporary or outright sensitive data. Working with streams through pipes gives you that possibility of containing everything through e.g. the kernel (which does the streaming for you, through file descriptors but no stored filesystem data). In any case, it's not just one feature, it's a paradigm.
    – Armen Michaeli
    Apr 27 at 6:51
9

<(...) does process substitution in bash. The output of the process in the parens is sent to an additional file descriptor beyond the normal 3, and a filename is returned corresponding to that file descriptor. In this way the output of a command can be treated as a filename to be passed to another command.

Improve this answer
1
  • However, the resulting filename/file handle doesn't have all the features of a normal file on disk. In particular, random access is not possible, so if the program requires that, you cannot use process substitution.
    – masterxilo
    Apr 28 at 8:37
8

There is one stdin and one stdout for each process. They are usually connected to the terminal, but they can be redirected separately from one another.

In the example, there are two wget processes involved, each of which gets its own stdin and stdout. Each wget process writes to -, which is its stdout. Then bash's process substitution <(...) connects the stdout of the process to a unique pseudo-file, from which diff can read. Note that the two process substitutions yield two different pseudo-files! Thus, diff sees something like:

diff /dev/fd/XX /dev/fd/YY

where the stdout of wget -q -O - URL1 is connected to /dev/fd/XX, and the stdout of wget -q -O - URL2 to /dev/fd/YY.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.