How does piping/redirection in Linux work exactly?

Question

I have these three examples of redirection stdin/stdout, only one of them is working the way it is intended to. I would love if someone can explain that to me.

The goal is to sort the contents in file1 and save the changes to the same file.

1. sort file1 | tee file1 > /dev/null --------> It works

2. sort file1 | tee file1 --------> Contents of file1 will be erased

3. sort file1 | tee file1 > file2 --------> Contents of file1 will be erased

PS. tee copies standard input to each FILE, and also to standard output.

What makes the first example work?

Answer 1

In according to my tests on Debian Wheezy, all 3 scenarios may lead to both results (file1 gets sorted and written back to itself OR nothing gets sorted and nothing gets written to file1.

I believe that this is normal behavior and comes from the way Linux works with files. Think about the command - the sort command starts to read file1 and immediately sends it's output to tee. Tee reads the output, writes it back to file1 and prints it to /dev/null. In case sort is quick enough to read the whole file1, tee gets sorted output. But in case tee gets it's lock on the file, it erases it (tee always erases the output file, except when append option is used) and that's pretty much what's happening in all your 3 scenarios.

To make it shorter, lets say, that sometimes sort is not quick enough to read file1. In such case tee erases the file BEFORE sort can read it.

I would recommend the following procedure:

cat file1 | sort > /tmp/sorting.tmp; mv /tmp/sorting.tmp file1

In case you want to see the sorted output on stdout, do it like this:

cat file1 | sort | tee /tmp/sorting.tmp; mv /tmp/sorting.tmp file1

It's not a good idea to let 2 different commands working with 1 file on multiprocessor systems - you can never be sure which one gets executed first. On a single threaded system, the behavior would be different - sequential.

Answer 2

I doubt that that behavior is predictable (and surely would not depend on it). The tee command probably starts a new process to send its input to the 'other' destination. The operating system will 'buffer' the output till it reaches the point where it creates the destination file and writes its temporary buffer to the file. The exact moment this happens (and overwrites the source) probably depends on:

• The size of the file and available memory for the buffer
• The elapsed time
• If the input from the pipe to tee finishes

This goes deeper than bash: It's the way the programs work which bash starts. The shell just interprets the commands you type in, and starts the programs needed to execute the commands. The shell has no control on how each program works, and even less on how those programs interact. Asking a program (or a set of programs) to take data from an input file, and write the result over the same input file in the same sentence, is the responsibility of the user.

Don't forget that bash is just the interpreter of user commands: It's just a shell around the operating system to convert user intentions into system calls.

And it's documented, too! Or this mail, which addresses similar problems. Or this StackOverflow thread. Or this Serverfault thread.

Note that this can also happen with redirection of stdin: if you take inputs commands from a file: $ myprog < commandfile. If myprog writes to commandfile, there is no guarantee that all commandfile's commands will execute.

A really basic analogy would be something like this instruction list:

- Execute the instructions step by step
- Dip this instruction list in a bucket of black paint
- Type in the following commands:
  find /etc -type f -exec cat '{}' \; | tr -c '.[:digit:]' '\n' \
  | grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$'

I'd imagine you'd make a copy first? (command taken from Advanced Bash-Scripting Guide)

Answer 3

So you want the original contents of the file to remain, while appending the file with the changes?

tee over writes by default, try using the -a flag to append the file with the changes.

Answer 4

sort file1 | tee file1 > tmp && mv file1 original && mv tmp file1

You can write the file to a placeholder, rename the original to a backup, then move the placeholder to the original.