As a continuation from my last post where I have used grep & tail -f to find occurences of "rare" events. I would like to record this in another file.

I've tried turning

tail -f log.txt | egrep 'WARN|ERROR'


tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

The file gets created, but nothing is populated, is this a caching issue or otherwise? How would I get a real-time appending of my tail's output to a new file?

5 Answers 5


Buffering is the problem.

Do it this way,

tail -f log.txt | egrep --line-buffered 'WARN|ERROR' | tee filtered_output.txt
#                       ^^^^^^^^^^^^^^^

Confirmed to work on Cygwin too.


It's probably a buffering issue. See this SO post on disabling the auto-buffering when using pipes. You can use the unbuffer command from expect:

$ unbuffer tail -f log.txt | egrep 'WARN|ERROR' | tee filtered_output.txt

Edit: Since you have a longer pipeline, you probably need to unbuffer each command (except the last):

$ unbuffer tail -f log.txt | unbuffer egrep 'WARN|ERROR' | tee filtered_output.txt

Edit 2: unbuffer is available on Cygwin from the expect source package (eg expect-20030128-1-src.tar.bz2, found in the expect/examples folder), but it's a very short script. If you have the expect package already installed, simply put this into a script called unbuffer in your /usr/local/bin directory:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

eval spawn -noecho $argv
set timeout -1

On Debian, the unbuffer command is provided in the expect-dev package and is installed as expect_unbuffer.

  • is there a way to accomplish this with cygwin?
    – Mike
    Oct 23, 2009 at 15:53
  • added info on using in cygwin; you'll need the expect package. Oct 23, 2009 at 16:15
  • Thanks, can't try until Monday now sadly. Will update then.
    – Mike
    Oct 23, 2009 at 17:42

When using a command that does not really 'finish' (such as tail -f), this actually does not really work or that well (at all).

You should be able to redirect the output to a text file. Try this:

tail -f log.txt | egrep 'WARN|ERROR' > filtered_output.txt
  • 1
    This doesn't seem to work.
    – Mike
    Oct 23, 2009 at 15:49

This is the version of unbuffer that I have:

#!/usr/bin/expect --
# Description: unbuffer stdout of a program
# Author: Don Libes, NIST

if {[string compare [lindex $argv 0] "-p"] == 0} {
    # pipeline
    set stty_init "-echo"
    eval spawn -noecho [lrange $argv 1 end]
    close_on_eof -i $user_spawn_id 0
    interact {
    eof {
        # flush remaining output from child
        expect -timeout 1 -re .+
} else {
    set stty_init "-opost"
    set timeout -1
    eval spawn -noecho $argv
  • +1 thanks for the additional info. dunno if it's cygwin-friendly but it looks like a more intelligent script. Oct 24, 2009 at 22:21

As others have pointed out, you can use the unbuffer utility from Expect.

Note, however, that depending on your system and available version of Expect, you may need to use the -p switch to unbuffer. Citing the man page:

   Normally, unbuffer does not read from stdin.  This simplifies use of unbuffer in some situations.  To use unbuffer in a  pipeline,  use
   the -p flag.  Example:

           process1 | unbuffer -p process2 | process3

So you might need this invocation:

unbuffer -p tail -f log.txt | unbuffer -p egrep 'WARN|ERROR' | tee filtered_output.txt

BTW, see this article for thorough explanation of the output buffering problem: http://www.pixelbeat.org/programming/stdio_buffering/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.