Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What I am trying to do is filter the output of one log file into separated log files based on a grep filter.

tail -f test.log | tee >(grep "Error" > error.log) >(grep "Warning"" > warning.log)

This way all "Error" entries are in one file, and all "Warning" entries are in a separate file.

I know this works in concept, because if I use cat instead of tail, I get the correct file output, but cannot track changes in real time (which I need because I am watching output logs from actively running tests)

Also, if I remove the '>' file re-directors from the grep commands it outputs the individual grep outputs to the console correctly; however, I want a recorded file as well.

Answer:

When writing to a file the output of grep was being buffered. Using egrep with a --line-buffer option fixed the behavior.

The new command looks like:

tail -f test.log | tee >(egrep --line-buffered "ERROR" > error.log) >(egrep --line-buffered "WARNING" > warning.log)

share|improve this question
1  
Could be a lineBuffer problem in tailf. Please check: superuser.com/questions/59497/… – cfreire Oct 1 '13 at 23:55
    
This fixed my problem, thank you! – Ryan Oct 7 '13 at 17:25

It might be easier if you use more than one line to do this. You could write a bash script:

#!/bin/bash

tail -f test.log | while read line; do
    if echo "$line" | grep -q "Error"; then
        echo "$line" >> error.log
    elif echo "$line" | grep -q "Warning"; then
        echo "$line" >> warning.log
    # The following is in case you want to print out lines that do not match 
    else
        echo "$line"
    fi
done
share|improve this answer
    
My next step was to create a Python program to do what I want, since I would get much more flexibility out of it that way. I wanted to understand why a command that looked like it should work, wasn't. Turns out it was buffering. – Ryan Oct 7 '13 at 17:28

@Ryan, thanks for posting an answer to your question.
another (universal) way to set per-line buffering is to use stdbuf command:

tail -f test.log | tee >(stdbuf -oL grep "ERROR" > error.log) >(stdbuf -oL grep "WARNING" > warning.log)

here is an excerpt from man stdbuf:

SYNOPSIS
   stdbuf OPTION... COMMAND
...
   -o, --output=MODE
          adjust standard output stream buffering
...
   If MODE is 'L' the corresponding stream will be line buffered.
....
EXAMPLES
   tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
   This will immedidately display unique entries from access.log
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .