2

I have a scenario where I want to filter log files. To do this I am wanting to follow the last 10 lines or so of a log file and as new lines are read in, pipe them through an awk program and then redirect the output to a listening port on a remote host via netcat.

Currently my script has the following line in it:

echo "\`awk -f rules.awk <(tail -F $@)\`" | nc -u 127.0.0.1 5004

Where $@ is the list of arguments supplied in the initial call and rules.awk simply contains some desired patterns to match on.

Calling the script with ./script logfile.log appears to do nothing and the script hangs. I speculate that the pipe actually requires a process to terminate before its final output is actually piped to the new command.

So I tried a different approach running the following in my terminal:

nc -u 127.0.0.1 5004 <(awk -f rules.awk <(tail -F logfile.log))

This resulted in the following error:

Invalid port specification: /dev/fd/63

It appears that the redirect has actually passed the logical device of the stream rather than its 'value' but I'm just guessing again.

Could someone with a bit more BASH nous point me in the right direction with this?

|improve this question
  • I think I'm not clear on what you want to do. Since you're doing a tail -f, it will never end. It should pass output to nc on a regular basis (probably following 1k or 4k blocks, depending on the kind of buffering awk and nc are doing. When would you want this command to end? – Alan Shutko Oct 17 '12 at 15:06
  • It wouldn't need to. It could continue running in a shell and send the relevant lines in the log files to a server for indexing in real time. It doesn't NEED to do this but I thought it would be a nice to have so am keen to get it working. What you say about the buffering is interesting though. My test file is very small so I'll try with a bigger file (> 4K) – Troy Patrick Oct 17 '12 at 20:00
  • Some more background on what I'm doing: The company I work for is wanting to use a product called Splunk to index log files and expose the business intelligence they contain. The licensing for Splunk is based on the amount of data that is indexed per day which is why I'm writing this script to reduce the data sent to the Splunk server for indexing. – Troy Patrick Oct 17 '12 at 20:04
1

File descriptors are a bit confusing, they are certainly more confusing than pipes. So before trying to redirect the output of a subshell (<()) as standard input to a command, try getting what you want with pipes. Also, it may help to get the output-commands working correctly before sending the input to a target command (nc). 

cat <(tail -F /var/log/daemon.log ) | cat -n | awk 'NR>3 { print}'

this is equivalent to 

tail -F /var/log/daemon.log | cat -n | awk 'NR>3 {print}';

but "better" since it does not rely on bash-specific syntax `<()'

|improve this answer
0

Offhand, this seems like it should work to me

tail -F $@ | awk -f rules.awk | nc -u 127.0.0.1 5004
|improve this answer
  • I thought so too but looking at this I think not: smop.co.uk/blog/index.php/2006/06/26/tail-f-and-awk It's the follow flag that's causing the issues btw. If I remove it then what you have suggested will work. – Troy Patrick Oct 17 '12 at 2:33
  • This does work. It looked like it wasn't but I think that was actually an issue with buffering. If I tried this with a significantly large input file I got the behavior I wanted. – Troy Patrick Oct 17 '12 at 20:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.