37

We all know mkfifo and pipelines. The first one creates a named pipe, thus one has to select a name, most likely with mktemp and later remember to unlink. The other creates an anonymous pipe, no hassle with names and removal, but the ends of the pipe get tied to the commands in the pipeline, it isn't really convenient to somehow get a grip of the file descriptors and use them in the rest of the script. In a compiled program, I would just do ret=pipe(filedes); in Bash there is exec 5<>file so one would expect something like "exec 5<> -" or "pipe <5 >6" -is there something like that in Bash?

42

You can unlink a named pipe immediately after attaching it to the current process, which practically results in an anonymous pipe:

# create a temporary named pipe
PIPE=$(mktemp -u)
mkfifo $PIPE
# attach it to file descriptor 3
exec 3<>$PIPE
# unlink the named pipe
rm $PIPE
...
# anything we write to fd 3 can be read back from it
echo 'Hello world!' >&3
head -n1 <&3
...
# close the file descriptor when we are finished (optional)
exec 3>&-

If you really want to avoid named pipes (e.g. the filesystem is read-only), your "get a grip of the file descriptors" idea also works. Note that this is Linux-specific due to the use of procfs.

# start a background pipeline with two processes running forever
tail -f /dev/null | tail -f /dev/null &
# save the process ids
PID2=$!
PID1=$(jobs -p %+)
# hijack the pipe's file descriptors using procfs
exec 3>/proc/$PID1/fd/1 4</proc/$PID2/fd/0
# kill the background processes we no longer need
# (using disown suppresses the 'Terminated' message)
disown $PID2
kill $PID1 $PID2
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptors when we are finished (optional)
exec 3>&- 4<&-
23

While none of the shells I know can make pipes without forking, some do have better than the basic shell pipeline.

In bash, ksh and zsh, assuming your system supports /dev/fd (most do nowadays), you can tie the input or the output of a command to a file name: <(command) expands to a file name that designates a pipe connected to the output from command, and >(command) expands to a file name that designates a pipe connected to the input of command. This feature is called process substitution. Its primary purpose is to pipe more than one command into or out of another, e.g.,

diff <(transform <file1) <(transform <file2)
tee >(transform1 >out1) >(transform2 >out2)

This is also useful to combat some of the shortcomings of basic shell pipes. For example, command2 < <(command1) is equivalent to command1 | command2, except that its status is that of command2. Another use case is exec > >(postprocessing), which is equivalent to, but more readable than, putting the whole rest of the script inside { ... } | postprocessing.

  • I tried this with diff and it worked but with kdiff3 or with emacs, it didn't work. My guess is that the temporary /dev/fd file is being removed before kdiff3 gets to read it. Or perhaps kdiff3 is trying to read the file twice and the pipe is only sending it once? – Eyal Aug 17 '15 at 9:15
  • @Eyal With process sustitution, the file name is a “magic” reference to a pipe (or a temporary file on Unix variants that don't support these magic variants). How the magic is implemented depends on the OS. Linux implements them as “magic” symbolic links whose target is not a valid file name (it's something like pipe:[123456]). Emacs sees that the symlink's target is not an existing file name and that confuses it enough that it doesn't read the file (there may be an option to get it to read it anyway, though Emacs doesn't like opening a pipe as a file anyway). – Gilles Aug 17 '15 at 10:14
10

Bash 4 has coprocesses.

A coprocess is executed asynchronously in a subshell, as if the command had been terminated with the ‘&’ control operator, with a two-way pipe established between the executing shell and the coprocess.

The format for a coprocess is:

coproc [NAME] command [redirections] 
3

As of October 2012 this functionality still doesn't seem to exist in Bash, but coproc can be used if all you need unnamed/anonymous pipes for is to talk to a child process. Problem with coproc at this point is that apparently only one is supported at a time. I can't figure out why coproc got this limitation. They should have been an enhancement of the existing task backgrounding code (the & op), but that's a question for the authors of bash.

  • Not only one coprocess is supported. You can name them, as long as you don't provide a simple command. Give it a command list instead: coproc THING { dothing; } Now your FDs are in ${THING[*]} and you can run coproc OTHERTHING { dothing; } and send and receive things to and from both. – clacke Mar 9 at 13:57
  • 2
    @clacke in man bash, under the BUGS title, they say this: There may be only one active coprocess at a time. And you get a warning if you start a second coproc. It appears to work, but I don't know what explodes in the background. – Radu C Apr 17 at 13:07
  • Ok, so it currently only works by luck, not because it was intentional. Fair warning, thanks. :-) – clacke Apr 17 at 13:53
2

While @DavidAnderson's answer covers all the bases and offers some nice safeguards, the most important thing it reveals is that getting your hands on an anonymous pipe is as easy as <(:), as long as you stay on Linux.

So the shortest and simplest answer to your question is:

exec 5<> <(:)

On macOS it won't work, then you'll need to create a temporary directory to house the named fifo in until you've redirected to it. I don't know about other BSDs.

  • You do realize your answer only works because of a bug in linux. This bug does not exist in macOS, thus requiring the more complex solution. The final version I posted will work in linux even if the bug in linux is fixed. – David Anderson Mar 9 at 16:19
  • @DavidAnderson Sounds like you have deeper knowledge of this than I do. Why is the Linux behavior a bug? – clacke Mar 9 at 19:17
  • 1
    If exec is passed and anonymous fifo that is opened only for reading, then exec should not allow this anonymous fifo to be opened for read and writing using a custom file descriptor. You should expect to get a -bash: /dev/fd/5: Permission denied message, which is what macOS issues. I believe the bug is that Ubuntu does not produce the same message. I would be willing to change my mind if someone could produce documentation saying exec 5<> <(:) is explicated permitted. – David Anderson Mar 9 at 21:36
  • @DavidAnderson Wow, that's fascinating. I assumed bash was doing something internally, but it turns out it's Linux that allows simply doing open(..., O_RDWR) on one unidirectional pipe end provided by the substitution and that turns it into a bidirectional pipe in one FD. You're probably right that one shouldn't rely on this. :-D Output from using execline's piperw to create the pipe, then repurposing it with bash <>: libranet.de/display/0b6b25a8-195c-84af-6ac7-ee6696661765 – clacke Mar 10 at 6:33
  • Not that it matters, but if you want to see under Ubuntu what is passed to exec 5<> , then enter fun() { ls -l $1; ls -lH $1; }; fun <(:). – David Anderson Mar 10 at 16:46
1

The following function was tested using GNU bash, version 4.4.19(1)-release (x86_64-pc-linux-gnu). The operating system was Ubuntu 18. This function take a single parameter which is the desired file descriptor for the anonymous FIFO.

MakeFIFO() {
    local "MakeFIFO_upper=$(ulimit -n)" 
    if [[ $# -ne 1 || ${#1} -gt ${#MakeFIFO_upper} || -n ${1%%[0-9]*} || 10#$1 -le 2
        || 10#$1 -ge MakeFIFO_upper ]] || eval ! exec "$1<> " <(:) 2>"/dev/null"; then
        echo "$FUNCNAME: $1: Could not create FIFO" >&2
        return "1"
    fi
}

The following function was tested using GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17). The operating system was macOS High Sierra. This function starts out by created a named FIFO in an temporary directory known only to the process that created it. Next, the file descriptor is redirected to the FIFO. Finally, the FIFO is unlinked from from the filename by deleting the temporary directory. This makes the FIFO anonymous.

MakeFIFO() {
    MakeFIFO.SetStatus() {
        return "${1:-$?}"
    }
    MakeFIFO.CleanUp() {
        local "MakeFIFO_status=$?"
        rm -rf "${MakeFIFO_directory:-}"    
        unset "MakeFIFO_directory"
        MakeFIFO.SetStatus "$MakeFIFO_status" && true
        eval eval "${MakeFIFO_handler:-:}'; true'" 
    }
    local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)" "MakeFIFO_file=" 
    MakeFIFO_handler="$(trap -p EXIT)"
    MakeFIFO_handler="${MakeFIFO_handler#trap -- }"
    MakeFIFO_handler="${MakeFIFO_handler% *}"
    trap -- 'MakeFIFO.CleanUp' EXIT
    until "$MakeFIFO_success"; do
        [[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
        && 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
        MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
        MakeFIFO_file="$MakeFIFO_directory/pipe"
        mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
        ! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
        MakeFIFO_success="true"
    done
    rm -rf "${MakeFIFO_directory:-}"
    unset  "MakeFIFO_directory"
    eval trap -- "$MakeFIFO_handler" EXIT
    unset  "MakeFIFO_handler"
    "$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}

The above functions can be combined to a single function that will work on both operating systems. Below is an example of such a function. Here, an attempt is made to create a truly anonymous FIFO. If unsuccessfully, then a named FIFO is created and converted to an anonymous FIFO.

MakeFIFO() {
    MakeFIFO.SetStatus() {
        return "${1:-$?}"
    }
    MakeFIFO.CleanUp() {
        local "MakeFIFO_status=$?"
        rm -rf "${MakeFIFO_directory:-}"    
        unset "MakeFIFO_directory"
        MakeFIFO.SetStatus "$MakeFIFO_status" && true
        eval eval "${MakeFIFO_handler:-:}'; true'" 
    }
    local "MakeFIFO_success=false" "MakeFIFO_upper=$(ulimit -n)" "MakeFIFO_file=" 
    MakeFIFO_handler="$(trap -p EXIT)"
    MakeFIFO_handler="${MakeFIFO_handler#trap -- }"
    MakeFIFO_handler="${MakeFIFO_handler% *}"
    trap -- 'MakeFIFO.CleanUp' EXIT
    until "$MakeFIFO_success"; do
        [[ $# -eq 1 && ${#1} -le ${#MakeFIFO_upper} && -z ${1%%[0-9]*}
        && 10#$1 -gt 2 && 10#$1 -lt MakeFIFO_upper ]] || break
        if eval ! exec "$1<> " <(:) 2>"/dev/null"; then
            MakeFIFO_directory=$(mktemp -d) 2>"/dev/null" || break
            MakeFIFO_file="$MakeFIFO_directory/pipe"
            mkfifo -m 600 $MakeFIFO_file 2>"/dev/null" || break
            ! eval ! exec "$1<> $MakeFIFO_file" 2>"/dev/null" || break
        fi
        MakeFIFO_success="true"
    done
    rm -rf "${MakeFIFO_directory:-}"
    unset  "MakeFIFO_directory"
    eval trap -- "$MakeFIFO_handler" EXIT
    unset  "MakeFIFO_handler"
    "$MakeFIFO_success" || { echo "$FUNCNAME: $1: Could not create FIFO" >&2; return "1"; }
}

Here is an example of creating an anonymous FIFO, then writing some text to the same FIFO.

fd="6"
MakeFIFO "$fd"
echo "Now is the" >&"$fd"
echo "time for all" >&"$fd"
echo "good men" >&"$fd"

Below is an example of reading the entire contents of the anonymous FIFO.

echo "EOF" >&"$fd"
while read -u "$fd" message; do
    [[ $message != *EOF ]] || break
    echo "$message"
done

This produces the following output.

Now is the
time for all
good men

The command below closes the anonymous FIFO.

eval exec "$fd>&-"

References:
Creating an anonymous pipe for later use
Files in Publicly Writable Directories Are Dangerous
Shell Script Security

0

Using the great and bright answer from htamas, I modified it a little to use it in an one liner, here it is:

# create a temporary named pipe
PIPE=(`(exec 0</dev/null 1</dev/null; (( read -d \  e < /proc/self/stat ; echo $e >&2 ; exec tail -f /dev/null 2> /dev/null ) | ( read -d \  e < /proc/self/stat ; echo $e  >&2 ; exec tail -f /dev/null 2> /dev/null )) &) 2>&1 | for ((i=0; i<2; i++)); do read e; printf "$e "; done`)
# attach it to file descriptors 3 and 4
exec 3>/proc/${PIPE[0]}/fd/1 4</proc/${PIPE[1]}/fd/0
...
# kill the temporary pids
kill ${PIPE[@]}
...
# anything we write to fd 3 can be read back from fd 4
echo 'Hello world!' >&3
head -n1 <&4
...
# close the file descriptor when we are finished (optional)
exec 3>&- 4<&-
  • 7
    I can't help noticing that your one-liner has more than one line. – Dmitry Grigoryev Jul 6 '16 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.