3

As many of us are surely aware, it's always a good idea to make your program accept stdin input. Very many programs do allow that *nix environments. This lets us do cool things like piping echo "foo" | less. Quite often one can find that cat barfile | baz is logically equivalent to baz barfile as behind the scenes it's just reading of strings anyway.

Nowadays, there are a lot more programs that one can not pipe to by default. Some programs have a flag that still allows for the behaviour mentioned above, but many don't.

Now, my question is, does a temporary file pipe exist?

Now, with my literally non-existent ability to write Bash and about 5 minutes on Google, I came up with this

#!/bin/bash
if [ $# -ne 2 ]; then exit 1; fi
f=$(mktemp)
($1) > $f
if [ $? -eq 0 ]; then ($2 $f); else exit $?; fi
rm $f

Calling this fpipe, we can do things like fpipe 'wget -O- www.example.com' baz where baz is a program that we can't pipe into but can do baz file.

My question is how can we do better. I suspect that with more Bash knowledge, re-writing the above script to take any amount of arguments would be fairly trivial (so that we can do things like fpipe 'foo x' bar baz where with piping we could do something like foo x | bar | baz. Bah, we could probably mix the two and end up with things such as fpipe 'wget -O- www.example.org | rev' baz.

Is there an existing construct that achieves this? I believe I saw a construct of sort foo x > bar < baz or something along these lines. I would have thought that this is a fairly common issue but my searches aren't bringing anything up. This means that I'm either not searching hard enough or that I'm missing something fairly obvious.

If there isn't a Proper Way(TM) to achieve this, is it possible to define a convenience syntax? Say, foo x |> baz | rev where |> pretty much directly translates to my script.

PS: I'm aware that my /script/ is very fragile and naive (such as quitting on non 0); feel free to post a better one.

1
  • Are you looking for solutions that are specifically bash or zsh? Between the two you'll get possibly double the results. Even with just one shell interpreter, you're likely to get a number of different methods or tactics. This question seems ill-fitted to the medium in which you've asked it.
    – killermist
    Jan 22, 2015 at 3:44

3 Answers 3

4

This will work (in principle) in any shell, but it requires a “recent enough version” of the operating system:

wget -O- www.example.com | baz /dev/stdin

When /dev/stdin exists, it is typically a symbolic link to /proc/self/fd/0, so if your box doesn’t have /dev/stdin, check for /proc/self/fd/0.


Of course the invention of pipes was one of the great innovations of Unix.1  Temporary intermediate files are fine in principle, but may have performance costs, and in the case of

program_that_produces_gobs_of_output | grep regular_expression_that_matches_very_few_lines

you potentially have the issue of not being able to fit the file (all the output from the first command) on the disk.  So, the best of both worlds is to use a named pipe.  This lets you capture the standard output from some program2 and provide it as input to another program as a pathname parameter:

f=$(mktemp)  &&  rm -f "$f"  &&  mkfifo "$f"
$1 >> "$f" &
$2 "$f"
rm -f "$f"

The first rm -f "$f" is necessary because mktemp not only generates a filename; it creates the file.  If we don’t remove it, the mkfifo will fail.  (Alternatively, we could use mktemp -u, which generates a filename without creating a file.)
___________
1 Whether Unix “invented” pipes is somewhat debatable.  According to The Evolution of the Unix Timesharing System (by Dennis M. Ritchie) (PDF), the concept existed in other forms in other places before pipes appeared in Unix in 1972.

2 or, in general, any output stream; e.g., from cp somefile "$f" or dd … of="$f"

5

With a recent enough version of Bash:

$ baz <(wget -O- www.example.com)

It's called process substitution.

2
  • +1 Process substitution has been available since at least version 2.0, released around 1997.
    – chepner
    Jan 21, 2013 at 1:27
  • @chepner Wow. I think I need to find a "What's new in bash 4.0" page then. Jan 21, 2013 at 12:04
-1

Here's two methods to have your script act identically if file arguments are given, or if it should read lines from stdin:

1) cat the files, use the special filename "-" if there are no files specified on the command line

#!/bin/bash
[[ $# -eq 0 ]] && set -- "-"
n=0
while read line; do
    echo "$((++n)) $line"
done < <(cat "$@")

2) if files are specified, cat their contents to the script's stdin stream

#!/bin/bash
[[ $# -gt 0 ]] && exec 0< <(cat "$@")
n=0
while read line; do
    echo "$((++n)) $line"
done

Not the answer you're looking for? Browse other questions tagged .