Pre-pending to an output line of `tail -f file` if its contents match another file

Question

Trying to cram all of the following into the title was hard, so this is what I really mean:

I am doing a tail -F access.log to watch an Apache2 log in realtime. The first thing on each line is an IP address followed by a space. I also have an unchanging CSV file where each row is an IP address, comma, name. For example:

10.1.2.33,John Smith
10.1.2.190,Jane Doe

Deliverable: I want to follow my access log as before, but if the CSV happens to contain a row with the same IP address that the current logfile line starts with, then insert the person's name at the start of the line.

For example, a line like this:

10.1.2.33 - - [22/Aug/2013:13:41:24 +0000] "GET /index.php ...

Should instead render as:

John Smith 10.1.2.33 - - [22/Aug/2013:13:41:24 +0000] "GET /index.php ...

Ideally, I would like to do this entirely by piping some regex commands together. So far, my biggest issue is that grep doesn't accept a pattern on stdin, it expects the subject on stdin (unless I'm missing something). Also, I have no problem transforming the CSV into some better format if required. Thanks.

Answer 1

Here's a POSIX shell solution

tail -f access.log | while read -r line;do
    ip=${line%% *}
    name=$(grep -F "$ip" your_csv_file|cut -d, -f2)
    if [ -z "$name" ];then
        printf "%s\n" "$line"
    else
        printf "%s\n" "$name $line"
    fi
done

Edit

Two improvements made to the solution:

• -r switch added to read so that it doesn't evaluate escape sequences in the lines it reads
• -F switch added to grep so that it treats the IP as a fixed string rather than a regular expression.

Answer 2

I would do this with a little Perl script:

#!/usr/bin/env perl

## Open the first argument, the .csv file
 open($f,"<","$ARGV[0]"); 
## Collect the list of ips and names in the %ips hash
 while(<$f>){
    chomp;
    @a=split(/,/); 
    $ips{$a[0]}=$a[1]
 }
## Now open the second argument, the log file to be watched. The
## trick here is to open by piping through tail -F
 open($f,"-|", "tail -F $ARGV[1]");
 while(<$f>){
 ## Get the ip
  /^([\d\.]+)/; 
  $ip=$1; 
 ## Prepend the associated name from the .csv file if one is defined
  s/$ip/$ips{$ip} $ip/ if defined($ips{$ip});
 ## Print the line
  print;
}

Save that in your $PATH as ip2name.pl, make it executable (chmod +x ip2name.pl) and then run it like this:

ip2name.pl /home/foo/ips.csv /var/log/apache2/access.log

Answer 3

grep can get the patterns from stdin, but it cannot get the text from stdin at the same time. See the -f option. If your shell supports process substitution, you can simulate it with

produce_patterns | grep -f- <( produce_input )

For example:

( echo b; echo y ) | grep -f- <( for i in {a..z} ; do echo $i ; done )

Output:

b
y