12

I am trying to pipe tail -f into awk to monitor a logfile in realtime, but according to examples, there should be not problem but I can't get it to work.

here is the command I'm running

tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

But when I edit my file using nano add a line, it is not printed in real time, If I run the awk command directly, my new line appear in the result.

1
14

You don't see it in real time because, for purposes of efficiency, pipes are buffered. tail -f has to fill up the buffer, typically 4 kB, before the output is passed to awk.

A fix is to use the unbuffer command which is part of the expect package:

unbuffer tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'

This tricks tail into thinking it is writing to an interactive terminal. As a result, it doesn't buffer.

For more, see https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

Alternatively, if you have GNU coreutils 7.5 or better, you can disable output buffering with the stdbuf command:

stdbuf -o0 tail -f logfile.log | awk -F" " '{print $1, $2, $7, $8}'
6
  • Sadly the stdbuf solution does not seem to work either.
    – gimpycpu
    Apr 16 '14 at 3:22
  • But you are right that if I add ~10 rows, the buffer is flushed. So while the code works, the unbuffered part doesn't
    – gimpycpu
    Apr 16 '14 at 3:29
  • I just fixed a typo in the stdbuf command. Other than that, there might be an additional issue with your editor not updating the file in a way that triggers tail. Can you try just appending to the file without the editor. For example, run date >>logfile.log a couple times in one window and see if awk, running in another window, prints the updates.
    – John1024
    Apr 16 '14 at 3:52
  • Yes I did tried echo "My long string here" >> logfile.log from a different putty window. When I execute that line about 10 times the awk command will execute.
    – gimpycpu
    Apr 16 '14 at 3:59
  • @gimpycpu Using nano, I got various results depending on how I edited the file before saving. Using echo "My long string here" >> logfile.log, however, it works every time for me.
    – John1024
    Apr 16 '14 at 4:00
6

@John1024's alternative answer is not quite correct, as the stdbuf -o0 prefix is misplaced. It belongs as the prefix to the awk command, not the tail -f command, so the correct command should be:

tail -f logfile.log | stdbuf -o0 awk -F" " '{print $1, $2, $7, $8}'

This is what worked for me, BUT only after I located an awk version that would cooperate. So note that not all versions of awk allow this! I had to try nawk, gawk, mawk, etc until I found one that worked. If you need that particular command string to function as written, keep trying various awk/gawk/mawk versions by being explicit with the path of the awk binary and even downloading others until you hit one that works.

2
  • I used the comment link to do that, but it prevented me due to not enough reputation. What means should I have used?
    – kenneth558
    Sep 13 '16 at 0:04
  • Suggest the change to the post to which you had the suggestion ('improve this answer' option). Even anonymous users can do that.
    – kenorb
    Sep 13 '16 at 1:09
3

On Ubuntu 20.10 where awk links to mawk version 1.3.4,

Add -W interactive. E.g. tail -f $file | awk -W interactive '{print}'

(Thanks to the other folks whose answers here caused me to read the man page looking for buffer.)

2

You could un-buffer it by adding a while loop as given in the example below. I have tested this on a running log file and it worked for me.

tail -f input.log | while read a; do echo "$a" | awk -F" " '{print $1, $2, $7, $8}' >> output.log; done
3
  • You generally want to avoid piping into while read if you can . If you can't, putting the awk pipe after the done would somewhat improve performance. But anyway, the proper fix is to disable buffering, not work around it.
    – tripleee
    Aug 16 '17 at 9:21
  • This helped me view log messages on an embedded system that lacked unbuffer and stdbuf
    – user275717
    Aug 9 '18 at 23:11
  • This will create a new awk process per each line, instead of running it once. May 4 '20 at 20:37
1

Neither stdbuf nor unbuffer seem to work for me (Ubuntu 18.04), but using a named pipe does. I created a shell script containing:

mkfifo /tmp/pipe
awk -F" " '{print $1, $2, $7, $8}' < /tmp/pipe &
tail -f input.log > /tmp/pipe
1
  • The stock version of awk has not been good to me with stdbuf, like you've discovered. Eventually I chanced on a version of awk/gawk/nawk/mawk that would work, but the REAL question in my mind is whether the stdbuf capability of awk is destined to become less prevalent or more amongst the collection of awks. And by the way, I applaud your use of named piping. One-liners are more useful, however, in many situations. Do you think you can put all that into one line like cron for instance would need?
    – kenneth558
    Sep 8 '20 at 22:00
0

In my experience with this problem, it appears that buffering issues can affect both the read and write sides of each command. In my case, I had a command like the following:

$ parallel ... | ggrep ...

This worked fine, printing output as it was produced by parallel. When I tried to process further with awk, though:

$ parallel ... | ggrep ... | awk ...

I didn't see output until parallel was done. I naturally thought the issue was with awk, hence finding this post, so I tried this:

$ parallel ... | ggrep ... | stdbuf -o0 awk ...

But what actually fixed it was this:

$ parallel ... | stdbuf -o0 ggrep ... | awk ...

So it appears that ggrep can tell when it's printing vs. hooked up to another command, and buffers its output in the latter case. (Incidentally, ggrep actually has a --line-buffered flag which fixes my problem as well.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.