Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

As a continuation from my last post where I have used grep & tail -f to find occurences of "rare" events. I would like to record this in another file.

I've tried turning

tail -f log.txt | egrep 'WARN|ERROR'

into

tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

The file gets created, but nothing is populated, is this a caching issue or otherwise? How would I get a real-time appending of my tail's output to a new file?

share|improve this question

5 Answers 5

up vote 15 down vote accepted

Buffering is the problem.

Do it this way,

tail -f log.txt | egrep --line-buffered 'WARN|ERROR' | tee filtered_output.txt
#                       ^^^^^^^^^^^^^^^

Confirmed to work on Cygwin too.

share|improve this answer
    
Thanks, can't try until Monday now sadly. Will update then. –  Mike Oct 23 '09 at 17:43
    
Work's a treat! Thank-you very much indeed. –  Mike Oct 26 '09 at 13:42

It's probably a buffering issue. See this SO post on disabling the auto-buffering when using pipes. You can use the unbuffer command from expect:

$ unbuffer tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

Edit: Since you have a longer pipeline, you probably need to unbuffer each command (except the last):

$ unbuffer tail -f log.txt | unbuffer egrep 'WARN|ERROR' | tee filtered_output.txt

Edit 2: unbuffer is available on Cygwin from the expect source package (eg expect-20030128-1-src.tar.bz2, found in the expect/examples folder), but it's a very short script. If you have the expect package already installed, simply put this into a script called unbuffer in your /usr/local/bin directory:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

eval spawn -noecho $argv
set timeout -1
expect

On Debian, the unbuffer command is provided in the expect-dev package and is installed as expect_unbuffer.

share|improve this answer
    
is there a way to accomplish this with cygwin? –  Mike Oct 23 '09 at 15:53
    
added info on using in cygwin; you'll need the expect package. –  quack quixote Oct 23 '09 at 16:15
    
Thanks, can't try until Monday now sadly. Will update then. –  Mike Oct 23 '09 at 17:42

When using a command that does not really 'finish' (such as tail -f), this actually does not really work or that well (at all).

You should be able to redirect the output to a text file. Try this:

tail -f log.txt | egrep 'WARN|ERROR' > filtered_output.txt
share|improve this answer
    
This doesn't seem to work. –  Mike Oct 23 '09 at 15:49

This is the version of unbuffer that I have:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

if {[string compare [lindex $argv 0] "-p"] == 0} {
    # pipeline
    set stty_init "-echo"
    eval spawn -noecho [lrange $argv 1 end]
    close_on_eof -i $user_spawn_id 0
    interact {
    eof {
        # flush remaining output from child
        expect -timeout 1 -re .+
        return
    }
    }
} else {
    set stty_init "-opost"
    set timeout -1
    eval spawn -noecho $argv
    expect
}
share|improve this answer
    
+1 thanks for the additional info. dunno if it's cygwin-friendly but it looks like a more intelligent script. –  quack quixote Oct 24 '09 at 22:21

As others have pointed out, you can use the unbuffer utility from Expect.

Note, however, that depending on your system and available version of Expect, you may need to use the -p switch to unbuffer. Citing the man page:

   Normally, unbuffer does not read from stdin.  This simplifies use of unbuffer in some situations.  To use unbuffer in a  pipeline,  use
   the -p flag.  Example:

           process1 | unbuffer -p process2 | process3

So you might need this invocation:

unbuffer -p tail -f log.txt | unbuffer -p egrep 'WARN|ERROR' | tee filtered_output.txt

BTW, see this article for thorough explanation of the output buffering problem: http://www.pixelbeat.org/programming/stdio_buffering/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.