1

I have these three examples of redirection stdin/stdout, only one of them is working the way it is intended to. I would love if someone can explain that to me.

The goal is to sort the contents in file1 and save the changes to the same file.

  1. sort file1 | tee file1 > /dev/null --------> It works

  2. sort file1 | tee file1 --------> Contents of file1 will be erased

  3. sort file1 | tee file1 > file2 --------> Contents of file1 will be erased

PS. tee copies standard input to each FILE, and also to standard output.

What makes the first example work?

3
  • 4
    Just for the record: if you're using GNU sort you can use -o option: sort file1 -o file1. May 6, 2015 at 16:21
  • Thanks for this info, I didn't know it. But I gave the sort command as an example, the -o option doesn't come with other commands such as sed and cut. So in order to manipulate the contents of a file and write the changes to it will force me to use the piping/redirection methods which I don't know how they are conceptually operating. As you can see above, I can't figure the mechanism that made the first one to work but not the other two. May 6, 2015 at 17:43
  • The man pages for bash and tee provide extensive examples that are usually specific to the implementation installed, and will go into depth about the side effects of things like noclobber.....
    – Don Simon
    May 6, 2015 at 18:33

4 Answers 4

0

In according to my tests on Debian Wheezy, all 3 scenarios may lead to both results (file1 gets sorted and written back to itself OR nothing gets sorted and nothing gets written to file1.

I believe that this is normal behavior and comes from the way Linux works with files. Think about the command - the sort command starts to read file1 and immediately sends it's output to tee. Tee reads the output, writes it back to file1 and prints it to /dev/null. In case sort is quick enough to read the whole file1, tee gets sorted output. But in case tee gets it's lock on the file, it erases it (tee always erases the output file, except when append option is used) and that's pretty much what's happening in all your 3 scenarios.

To make it shorter, lets say, that sometimes sort is not quick enough to read file1. In such case tee erases the file BEFORE sort can read it.

I would recommend the following procedure:

cat file1 | sort > /tmp/sorting.tmp; mv /tmp/sorting.tmp file1

In case you want to see the sorted output on stdout, do it like this:

cat file1 | sort | tee /tmp/sorting.tmp; mv /tmp/sorting.tmp file1

It's not a good idea to let 2 different commands working with 1 file on multiprocessor systems - you can never be sure which one gets executed first. On a single threaded system, the behavior would be different - sequential.

1
  • Thanks Thomas, your answer was very clear. Now I understand that my problem wasn't with sort and tee but it was with how pipe and redirection work in Linux. Although @jcoppens almost answered the same but wasn't specific as you were. May 15, 2015 at 21:47
2

I doubt that that behavior is predictable (and surely would not depend on it). The tee command probably starts a new process to send its input to the 'other' destination. The operating system will 'buffer' the output till it reaches the point where it creates the destination file and writes its temporary buffer to the file. The exact moment this happens (and overwrites the source) probably depends on:

  • The size of the file and available memory for the buffer
  • The elapsed time
  • If the input from the pipe to tee finishes

This goes deeper than bash: It's the way the programs work which bash starts. The shell just interprets the commands you type in, and starts the programs needed to execute the commands. The shell has no control on how each program works, and even less on how those programs interact. Asking a program (or a set of programs) to take data from an input file, and write the result over the same input file in the same sentence, is the responsibility of the user.

Don't forget that bash is just the interpreter of user commands: It's just a shell around the operating system to convert user intentions into system calls.

And it's documented, too! Or this mail, which addresses similar problems. Or this StackOverflow thread. Or this Serverfault thread.

Note that this can also happen with redirection of stdin: if you take inputs commands from a file: $ myprog < commandfile. If myprog writes to commandfile, there is no guarantee that all commandfile's commands will execute.

A really basic analogy would be something like this instruction list:

- Execute the instructions step by step
- Dip this instruction list in a bucket of black paint
- Type in the following commands:
  find /etc -type f -exec cat '{}' \; | tr -c '.[:digit:]' '\n' \
  | grep '^[^.][^.]*\.[^.][^.]*\.[^.][^.]*\.[^.][^.]*$'

I'd imagine you'd make a copy first? (command taken from Advanced Bash-Scripting Guide)

7
  • OK, but why the first example always works and the other two don't? Especially when the first, theoretically, matches the third. ps:- All three examples use the same file, so the size is the same and so is the buffer. May 6, 2015 at 18:29
  • Check if it still works with a larger file - more than 64kB, or 128kB. The first and third are very different. The first redirects the screen output to /dev/null, which is very fast, while the third sends it to a real file, which involves a lot of work - opening, buffering, closing and other administrative work - and involves access to the disk.
    – jcoppens
    May 6, 2015 at 18:58
  • Still, there should be rules that govern the way the bash interprets the commands and in my three examples no one yet shed the light of how they are interpreted by the bash that made the first give a result that differs from the others in spite of being, technically, the same (the examples). May 7, 2015 at 9:49
  • Comment was too long, I've added it to the answer.
    – jcoppens
    May 7, 2015 at 14:18
  • By the way, did you test with larger files?
    – jcoppens
    May 7, 2015 at 14:31
0

So you want the original contents of the file to remain, while appending the file with the changes?

tee over writes by default, try using the -a flag to append the file with the changes.

3
  • 1
    Actually, I don't want to append, I want to overwrite. It works with the first example. I want to know the mechanism behind the piping/redirection that only leads the first example to work but not the other two. May 6, 2015 at 16:58
  • sort file1 | tee file1 makes changes to file1, and writes them over the file. Not sure what you're asking?
    – Azuma
    May 6, 2015 at 17:02
  • sort file1 | tee file1 will erase the contents of file1 May 6, 2015 at 17:09
0
sort file1 | tee file1 > tmp && mv file1 original && mv tmp file1

You can write the file to a placeholder, rename the original to a backup, then move the placeholder to the original.

1
  • 1
    changing file2 to tmp actually made it work, but going back to my examples you can see that I already have a solution. My question is how piping/redirection is working in my above stated examples so only one of them is working correctly. In other words I want to understand the behavior of it. May 6, 2015 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.