Is cat output modified by writing to the file

Question

With a piped command sequence like:

$ cat afile | somecommand | tee afile

afile is both read (by cat) and written (by tee). The question is, supposing afile is at least several megabytes (or large enough to not be fully buffered by the OS), will the later bytes in the file read by cat be affected when tee starts to write to the file?

In other words, is it ever possible for tee to overwrite the file before cat has finished with it?

Answer 1

In a pipelined set of processes it is almost always a terrible mistake to write to a file that is also being read from at the same time.

This is because the write operations occur concurrently with the read operations. Usually resulting in a prematurely truncated file. In the past this surprised people more used to DOS pipelines where the operating system serialised the processes by means of hidden temporary files (or their moral equivalent)

Solutions mostly involve using temporary files and renaming them on completion.

somecommand < infile | tee tempfile; mv tempfile infile

Obviously, this can introduce other problems.

Some utilities (awk, perl etc) handle this for you if you give them appropriate command line options.

perl -i -e 'somecommands' infile ... 

Note that the problem you experience is nothing to do with cat. In my example I avoided unnecessary use of cat, partly to make this clear, partly out of tradition.

Answer 2

A few experiments for my own pleasure (and for anyone else interested in the question). I generated a 100 MB afile using /dev/urandom and tried a few pipelines and redirects using the same file.

Most of the following commands don't make logical sense (as a "successful" result should keep the file unchanged) and it should also be noted that if different files are specified for input and output, all of the commands result in a complete copy of the input.

Here's a few of the results:

$ cat <afile >afile

Definition: cat stdin from afile and redirect to afile. Result: afile is truncated (0 bytes)

$ cat afile|cat >afile

Definition: cat afile and pipe stdout to another cat, redirected to afile. Result: afile is truncated (0 bytes)

$ tee afile <afile >/dev/null

Definition: tee stdin from afile and write to afile (and redirect stdout to /dev/null). Result: afile is truncated (0 bytes)

$ cat afile|tee afile >/dev/null

Definition: cat afile and pipe stdout to tee (and redirect tee stdout to /dev/null). Result: afile is reduced to 128 KB

The last entry shows the problem most clearly: cat only manages to buffer 128 KB and pipe it to tee before the file is gone. So if your file is small you might be lucky, but it's best to heed the answer and always separate input and output files.