5

I am looking for a way to run a program Prog and end up with 3 files:

  1. The stdout of Prog
  2. The stderr of Prog
  3. Both stdout and stderr of Prog combined just as they would be on screen if no redirecting took place.

Is there a combination of redirecting, pipes etc. which could achieve this?

Note: I normally use bash.

3 Answers 3

4

As far as I know, there is no elegant way of doing this (for an inelegant way that works but ain't pretty, scroll down to the end of my answer). I doubt you can do any better than:

$ command >stdout.txt 2>stderr.txt && cat stdout.txt stderr.txt > both.txt

There are various cool tricks you can use but none of them seems to succeed in producing the 3 files any better than the above. The main problem is that the file both.txt will not show the messages (STDERR and STDOUT) in the correct order. This is because (as explained here):

When you redirect both standard output and standard error to the same file, you may get some unexpected results. This is due to the fact that STDOUT is a buffered stream while STDERR is always unbuffered. This means that every character of STDERR is written as soon as it is available while STDOUT writes stuff in batches. When both STDOUT and STDERR are going to the same file you may see error messages appear sooner than you would have expected them in relation to the actual output of your program or script. It isn’t anything to be alarmed about but is simply a side-effect of buffered vs. unbuffered streams, you just need to keep it in mind.

The best alternative I could find was using bash subshells, is kind of complex and still does not display the output in the correct order. I made a simple Perl script, test.pl that prints "OUT" to STDOUT and "ERR" to STDERR, repeating the process 3 times:

#/usr/bin/perl 
for($i=0; $i<=2; $i++){
    print STDOUT "OUT\n"; 
    print STDERR "ERR\n"
}

Its normal, un-redirected output is :

$ ./test.pl
OUT
ERR
OUT
ERR
OUT
ERR

To redirect output(s) I ran:

(./test.pl 2> >(tee error.txt) > >(tee out.txt)) > both.txt 

This uses tee, a program that will print its input to screen and to a file name. So, I am redirecting STDERR and passing it as input to tee, telling it to write it to the file error.txt. Similarly with STDOUT and the file out.txt. I am placing the whole thing in a subshell ((...)) so I can then capture all of its output and redirect to both.txt.

Now, this works inasmuch as it creates 3 files, one with STDERR, one with STDOUT and one with both. However, as explained above, this results in the messages appearing in the incorrect order in both.txt:

$ cat both.txt 
ERR
ERR
ERR
OUT
OUT
OUT

The only way around this I could find was to append the time it was printed to each line of output and then sorting, but it is getting seriously convoluted and, in your place, I would ask myself if it is really worth it:

 $(./test.pl \
   2> >(while read n; do echo `date +%N`" $n"; echo "$n" >>error.txt; done) \
    > >(while read n; do echo `date +%N`" $n"; echo "$n" >> out.txt; done )) \
 | gawk '{print $2}'> both.txt 
4
  • Thanks for the answer. I've tried both yours and Kevin's approach; accepting yours for idea of timestamps. The interesting thing is that the outputs end up in (almost) correct order in the combined file even without stamping and sorting. Could it be because Prog is actually a Python script doing Popen() and the outputs itself come from the process it spawns? And BTW, how does the terminal circumvent stdout buffering? Dec 13, 2012 at 9:20
  • Sorry, @Agnew, reached the limits of my STDOUT/STDERR foo :), no idea.
    – terdon
    Dec 13, 2012 at 14:52
  • I don't believe that writing timestamps will (always) help. Since Stdout can delay and accumulate data, action taken by the re-direct (including adding timestamps) can be delayed. The timestamps will show when the data was written to the output file, which is not necessarily the same as when the event occurred. Dec 13, 2012 at 19:56
  • @Kevin, indeed but the relative time difference between STDOUT and STDERR should be conserved. The point is keeping the same order, not knowing exactly when each line was printed.
    – terdon
    Dec 13, 2012 at 21:26
2

1) create two scripts:

getout.sh
geterr.sh

getout.sh will receive Stdout.
Script will write data to f1.txt and also write the same data to file f12.txt

geterr.sh will receive Stderr.
Script will write data to f2.txt and also write the same data to file f12.txt

I do not believe it's possible to reliably preserve the exact order of the data written to the file: f12.txt. This is because writes to Stdout are buffered by the operating system and writes to Stderr happen in near real-time. This can (will) delay the Stdout data from being seen by the script. The effect of this is:

a. For some data written to Stdout BEFORE data written to 
   Stderr, the data written to f12.txt from Stderr may 
   precede the data written by Stdout.  
b. Some data written to Stdout will be accumulated with 
   other data written earlier or later. As the script sees it, 
   these data lines may all appear at the same time (together).  
c. Because of (b), for some data written to Stderr in-between 
   multiple data writes to Stdout, the data written to f12.txt 
   from Stderr may precede the whole group of data written to Stdout.  
d. Because of all this, including timestamps with the data 
   written to f12.txt will not preserve the order that events happened.  

2) create 2 named pipes:

% mknod pout p
% mknod perr p

3) let scripts "listen" to the pipes:

% ./getout.sh < pout &
% ./geterr.sh < perr &

4) Start your the program (Prog) like this:

% Prog 1> pout 2> perr
1
  • Nice, hadn't thought of named pipes. Still, you can do the same thing without named pipes (and, so with fewer steps) by using sub-shells as I have done in my answer. Also, you still have the problem of the lines not appearing in the correct order in f12.txt but you can fix that by making the two scripts append date to their output and sorting.
    – terdon
    Dec 11, 2012 at 16:26
0

To have them separate into different files:

./Prog 1> stdout.log 2> stderr.log

To have them combine into same file:

./Prog 1>> combine.log 2>> combine.log

There is a 'shorthand' way to do the combine for bash, but the above is safer in case you use sh.


Depends on how accurate your combine need, the following may or may not work

touch stdout.log
touch stderr.log
tail -f stdout.log >> combine.log &
tail -f stderr.log >> combine.log &
./Prog 1>> stdout.log 2>> stderr.log ; pkill tail
3
  • Thanks; I know these. However, I'd need a way to "do both of these simulatneously." That is, have all of stdout.log, stderr.log and combine.log when the run finishes. Dec 10, 2012 at 19:30
  • Update my answer for combine.log
    – John Siu
    Dec 10, 2012 at 20:05
  • In some versions of the shell, 1> combine.log 2> combine.log will throw the standard output into the bit bucket and capture only the stderr in combine.log. Dec 11, 2012 at 1:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .