Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

As a continuation from my last post where I have used grep & tail -f to find occurences of "rare" events. I would like to record this in another file.

I've tried turning

tail -f log.txt | egrep 'WARN|ERROR'

into

tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

The file gets created, but nothing is populated, is this a caching issue or otherwise? How would I get a real-time appending of my tail's output to a new file?

share|improve this question
up vote 26 down vote accepted

Buffering is the problem.

Do it this way,

tail -f log.txt | egrep --line-buffered 'WARN|ERROR' | tee filtered_output.txt
#                       ^^^^^^^^^^^^^^^

Confirmed to work on Cygwin too.

share|improve this answer
    
Thanks, can't try until Monday now sadly. Will update then. – Mike Oct 23 '09 at 17:43
    
Work's a treat! Thank-you very much indeed. – Mike Oct 26 '09 at 13:42

It's probably a buffering issue. See this SO post on disabling the auto-buffering when using pipes. You can use the unbuffer command from expect:

$ unbuffer tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

Edit: Since you have a longer pipeline, you probably need to unbuffer each command (except the last):

$ unbuffer tail -f log.txt | unbuffer egrep 'WARN|ERROR' | tee filtered_output.txt

Edit 2: unbuffer is available on Cygwin from the expect source package (eg expect-20030128-1-src.tar.bz2, found in the expect/examples folder), but it's a very short script. If you have the expect package already installed, simply put this into a script called unbuffer in your /usr/local/bin directory:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

eval spawn -noecho $argv
set timeout -1
expect

On Debian, the unbuffer command is provided in the expect-dev package and is installed as expect_unbuffer.

share|improve this answer
    
is there a way to accomplish this with cygwin? – Mike Oct 23 '09 at 15:53
    
added info on using in cygwin; you'll need the expect package. – quack quixote Oct 23 '09 at 16:15
    
Thanks, can't try until Monday now sadly. Will update then. – Mike Oct 23 '09 at 17:42

When using a command that does not really 'finish' (such as tail -f), this actually does not really work or that well (at all).

You should be able to redirect the output to a text file. Try this:

tail -f log.txt | egrep 'WARN|ERROR' > filtered_output.txt
share|improve this answer
    
This doesn't seem to work. – Mike Oct 23 '09 at 15:49

This is the version of unbuffer that I have:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

if {[string compare [lindex $argv 0] "-p"] == 0} {
    # pipeline
    set stty_init "-echo"
    eval spawn -noecho [lrange $argv 1 end]
    close_on_eof -i $user_spawn_id 0
    interact {
    eof {
        # flush remaining output from child
        expect -timeout 1 -re .+
        return
    }
    }
} else {
    set stty_init "-opost"
    set timeout -1
    eval spawn -noecho $argv
    expect
}
share|improve this answer
    
+1 thanks for the additional info. dunno if it's cygwin-friendly but it looks like a more intelligent script. – quack quixote Oct 24 '09 at 22:21

As others have pointed out, you can use the unbuffer utility from Expect.

Note, however, that depending on your system and available version of Expect, you may need to use the -p switch to unbuffer. Citing the man page:

   Normally, unbuffer does not read from stdin.  This simplifies use of unbuffer in some situations.  To use unbuffer in a  pipeline,  use
   the -p flag.  Example:

           process1 | unbuffer -p process2 | process3

So you might need this invocation:

unbuffer -p tail -f log.txt | unbuffer -p egrep 'WARN|ERROR' | tee filtered_output.txt

BTW, see this article for thorough explanation of the output buffering problem: http://www.pixelbeat.org/programming/stdio_buffering/

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .