Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

What I am trying to do is filter the output of one log file into separated log files based on a grep filter.

tail -f test.log | tee >(grep "Error" > error.log) >(grep "Warning"" > warning.log)

This way all "Error" entries are in one file, and all "Warning" entries are in a separate file.

I know this works in concept, because if I use cat instead of tail, I get the correct file output, but cannot track changes in real time (which I need because I am watching output logs from actively running tests)

Also, if I remove the '>' file re-directors from the grep commands it outputs the individual grep outputs to the console correctly; however, I want a recorded file as well.

Answer:

When writing to a file the output of grep was being buffered. Using egrep with a --line-buffer option fixed the behavior.

The new command looks like:

tail -f test.log | tee >(egrep --line-buffered "ERROR" > error.log) >(egrep --line-buffered "WARNING" > warning.log)

share|improve this question
1  
Could be a lineBuffer problem in tailf. Please check: superuser.com/questions/59497/… –  cfreire Oct 1 '13 at 23:55
    
This fixed my problem, thank you! –  Ryan Oct 7 '13 at 17:25

2 Answers 2

It might be easier if you use more than one line to do this. You could write a bash script:

#!/bin/bash

tail -f test.log | while read line; do
    if echo "$line" | grep -q "Error"; then
        echo "$line" >> error.log
    elif echo "$line" | grep -q "Warning"; then
        echo "$line" >> warning.log
    # The following is in case you want to print out lines that do not match 
    else
        echo "$line"
    fi
done
share|improve this answer
    
My next step was to create a Python program to do what I want, since I would get much more flexibility out of it that way. I wanted to understand why a command that looked like it should work, wasn't. Turns out it was buffering. –  Ryan Oct 7 '13 at 17:28

@Ryan, thanks for posting an answer to your question.
another (universal) way to set per-line buffering is to use stdbuf command:

tail -f test.log | tee >(stdbuf -oL grep "ERROR" > error.log) >(stdbuf -oL grep "WARNING" > warning.log)

here is an excerpt from man stdbuf:

SYNOPSIS
   stdbuf OPTION... COMMAND
...
   -o, --output=MODE
          adjust standard output stream buffering
...
   If MODE is 'L' the corresponding stream will be line buffered.
....
EXAMPLES
   tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq
   This will immedidately display unique entries from access.log
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.