Obtain both merged and separated stdout and stderr

I am looking for a way to run a program Prog and end up with 3 files:

1. The stdout of Prog
2. The stderr of Prog
3. Both stdout and stderr of Prog combined just as they would be on screen if no redirecting took place.

Is there a combination of redirecting, pipes etc. which could achieve this?

Note: I normally use bash.

-

As far as I know, there is no elegant way of doing this (for an inelegant way that works but ain't pretty, scroll down to the end of my answer). I doubt you can do any better than:

$command >stdout.txt 2>stderr.txt && cat stdout.txt stderr.txt > both.txt  There are various cool tricks you can use but none of them seems to succeed in producing the 3 files any better than the above. The main problem is that the file both.txt will not show the messages (STDERR and STDOUT) in the correct order. This is because (as explained here): When you redirect both standard output and standard error to the same file, you may get some unexpected results. This is due to the fact that STDOUT is a buffered stream while STDERR is always unbuffered. This means that every character of STDERR is written as soon as it is available while STDOUT writes stuff in batches. When both STDOUT and STDERR are going to the same file you may see error messages appear sooner than you would have expected them in relation to the actual output of your program or script. It isn’t anything to be alarmed about but is simply a side-effect of buffered vs. unbuffered streams, you just need to keep it in mind. The best alternative I could find was using bash subshells, is kind of complex and still does not display the output in the correct order. I made a simple Perl script, test.pl that prints "OUT" to STDOUT and "ERR" to STDERR, repeating the process 3 times: #/usr/bin/perl for($i=0; $i<=2;$i++){
print STDOUT "OUT\n";
print STDERR "ERR\n"
}


Its normal, un-redirected output is :

$./test.pl OUT ERR OUT ERR OUT ERR  To redirect output(s) I ran: (./test.pl 2> >(tee error.txt) > >(tee out.txt)) > both.txt  This uses tee, a program that will print its input to screen and to a file name. So, I am redirecting STDERR and passing it as input to tee, telling it to write it to the file error.txt. Similarly with STDOUT and the file out.txt. I am placing the whole thing in a subshell ((...)) so I can then capture all of its output and redirect to both.txt. Now, this works inasmuch as it creates 3 files, one with STDERR, one with STDOUT and one with both. However, as explained above, this results in the messages appearing in the incorrect order in both.txt: $ cat both.txt
ERR
ERR
ERR
OUT
OUT
OUT


The only way around this I could find was to append the time it was printed to each line of output and then sorting, but it is getting seriously convoluted and, in your place, I would ask myself if it is really worth it:

 $(./test.pl \ 2> >(while read n; do echo date +%N"$n"; echo "$n" >>error.txt; done) \ > >(while read n; do echo date +%N"$n"; echo "$n" >> out.txt; done )) \ | gawk '{print$2}'> both.txt

-
Thanks for the answer. I've tried both yours and Kevin's approach; accepting yours for idea of timestamps. The interesting thing is that the outputs end up in (almost) correct order in the combined file even without stamping and sorting. Could it be because Prog is actually a Python script doing Popen() and the outputs itself come from the process it spawns? And BTW, how does the terminal circumvent stdout buffering? –  Angew Dec 13 '12 at 9:20
Sorry, @Agnew, reached the limits of my STDOUT/STDERR foo :), no idea. –  terdon Dec 13 '12 at 14:52
I don't believe that writing timestamps will (always) help. Since Stdout can delay and accumulate data, action taken by the re-direct (including adding timestamps) can be delayed. The timestamps will show when the data was written to the output file, which is not necessarily the same as when the event occurred. –  Kevin Fegan Dec 13 '12 at 19:56
@Kevin, indeed but the relative time difference between STDOUT and STDERR should be conserved. The point is keeping the same order, not knowing exactly when each line was printed. –  terdon Dec 13 '12 at 21:26

1) create two scripts:

getout.sh
geterr.sh


Script will write data to f1.txt and also write the same data to file f12.txt

Script will write data to f2.txt and also write the same data to file f12.txt

I do not believe it's possible to reliably preserve the exact order of the data written to the file: f12.txt. This is because writes to Stdout are buffered by the operating system and writes to Stderr happen in near real-time. This can (will) delay the Stdout data from being seen by the script. The effect of this is:

a. For some data written to Stdout BEFORE data written to
Stderr, the data written to f12.txt from Stderr may
precede the data written by Stdout.
b. Some data written to Stdout will be accumulated with
other data written earlier or later. As the script sees it,
these data lines may all appear at the same time (together).
c. Because of (b), for some data written to Stderr in-between
multiple data writes to Stdout, the data written to f12.txt
from Stderr may precede the whole group of data written to Stdout.
d. Because of all this, including timestamps with the data
written to f12.txt will not preserve the order that events happened.


2) create 2 named pipes:

% mknod pout p
% mknod perr p


3) let scripts "listen" to the pipes:

% ./getout.sh < pout &
% ./geterr.sh < perr &


4) Start your the program (Prog) like this:

% Prog 1> pout 2> perr

-
Nice, hadn't thought of named pipes. Still, you can do the same thing without named pipes (and, so with fewer steps) by using sub-shells as I have done in my answer. Also, you still have the problem of the lines not appearing in the correct order in f12.txt but you can fix that by making the two scripts append date to their output and sorting. –  terdon Dec 11 '12 at 16:26

To have them separate into different files:

./Prog 1> stdout.log 2> stderr.log


To have them combine into same file:

./Prog 1>> combine.log 2>> combine.log


There is a 'shorthand' way to do the combine for bash, but the above is safer in case you use sh.

Depends on how accurate your combine need, the following may or may not work

touch stdout.log
touch stderr.log
tail -f stdout.log >> combine.log &
tail -f stderr.log >> combine.log &
./Prog 1>> stdout.log 2>> stderr.log ; pkill tail

-
Thanks; I know these. However, I'd need a way to "do both of these simulatneously." That is, have all of stdout.log, stderr.log and combine.log when the run finishes. –  Angew Dec 10 '12 at 19:30
Update my answer for combine.log –  John Siu Dec 10 '12 at 20:05
In some versions of the shell, 1> combine.log 2> combine.log will throw the standard output into the bit bucket and capture only the stderr in combine.log. –  Scott Dec 11 '12 at 1:08