Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'd like to monitor my vsftpd logs and pull information from the logs to process the files that have been uploaded. Up till now everything was great and worked fine:

tail -n0 -F /var/log/vsftpd.log | while read line; do
    if echo "$line" | grep -q 'OK UPLOAD:'; then
        #do a bunch of processing stuff - this takes some time
    fi
done

I started doing some scale testing and uploaded 500 files at once. For some reason the log lines are missing or they are cut off when you upload so many things. For example a typical line looks like:

Sun Apr 7 09:08:51 2013 [pid 25409] [cam02430] OK UPLOAD: Client "206.132.183.201", "/20130407/09/20130407_090842D.avi", 531792 bytes, 426.14Kbyte/sec

But sometimes the line looks like this (if not missing entirely):

:08:51 2013 [pid 25409] [cam02430] OK UPLOAD: Client "206.132.183.201", "/20130407/09/20130407_090842D.avi", 531792 bytes, 426.14Kbyte/sec

It is cutting off the first few characters of the log. It doesn't do it for every line but most. Also many lines are just missing. I guess I have a race condition of my processing not finishing in time. How I can get around this?

EDIT I guess I have 3 options:

  1. Don't use tail and try to use a combination of things like a cron job/grep to check the log for a new entry. I imagine this could be tricky. How do I tell what is new in that log?

  2. Monitor the file change with monit or equivalent. I don't believe this is an option. The files get stored into somewhat random directories with files names that are time stamped.

  3. logstash or equivalent. I don't think these programs fit my needs. But maybe someone knows otherwise.

Right now I'm focusing on #1 with no good leads. So any thoughts on any of this would be appreciated.

share|improve this question
    
I don't know anything about the logging behaviour of vsftpd, but you could (if no new subdirectories get created on a regular basis within the directory tree that you populate using the ftp process) use inotifywait with a for-loop to pick up on completed files (well, write/ colse events_. –  tink Apr 12 '13 at 19:25
    
Intensive log processing should not be done using shell scripts, I'd go for JSON log format + logstash/elasticsearch/kibana/statsd/graphite –  dawud Apr 12 '13 at 19:30
    
@tink I considered that when I started this project but was not clear on how well it would handle the close events (successful upload) and the files get stored in new subdirectories that are always changing according to date. That part is out of my control. –  Tom Apr 13 '13 at 0:05
    
@Tom When the character get cut off, can you verify that it only cut off characters of that particular line or you are actually missing multiple lines in between. –  John Siu Apr 15 '13 at 7:50
    
@JohnSiu Actually it is missing lines as well. –  Tom Apr 15 '13 at 17:36
show 2 more comments

4 Answers 4

You may start prevent forking a "grep" process for every line in the log, by rewriting your code like this:

tail -n0 -F /var/log/vsftpd.log | grep 'OK UPLOAD:' | while read line ; do
        #do a bunch of processing stuff - this takes some time
done

But I agree with other posters, shell is not really suitable for heavy text/string manipulation. You should consider using use another scripting engine (at least awk, or better perl or python)

share|improve this answer
    
Thanks, that sounded promising but that suffers the same problem. I'm exploring other options. –  Tom Apr 16 '13 at 18:01
add comment

I had good experience with Perl using the file open '<' options, if you are willing to relax your limit to shell scripts only.

e.g. http://stackoverflow.com/questions/1425223/how-do-i-read-a-file-which-is-constantly-updating

share|improve this answer
add comment

As @epoon suggested, Perl might be a good choice:

#!/usr/bin/env perl

open(FH,'<',$ARGV[0]) || die("Could not open file $ARGV[0]\n");
for (;;) {
    while (<FH>) {
    if (/OK UPLOAD/) { ## this is where you do your processing. As an example, 
                       ## I am collecting the file's info
        /^(.+?)\s*\[pid\s*(\d+).+?\"([\d\.]+).+?\"(.+?)\".+?(\d+).+?\s([^\s]+)/;
        my ($date,$pid,$client,$filename,$size,$rate) = ($1,$2,$3,$4,$5,$6);
        print "On $date, process $pid uploaded file \"$filename\" of $size bytes from $client at $rate\n"

    }
    }
    # eof reached on FH, but wait a second and maybe there will be more output
    sleep 1;
    seek FH, 0, 1;      # this clears the eof flag on FH
}

Running this script on /var/log/vsftpd.log should give output like

On Sat Apr 20 16:30:05 CEST 2013, process 25409 uploaded file "/20130407/09/20130407_090842D.avi" of 531792 bytes from 206.132.183.201 at 426.14Kbyte/sec

I would be happy to update this if you specify what exactly you want to do with the log info.

share|improve this answer
    
I think this might work, but at any rate I went down a different route. I've written a notification api into vsftpd. This guarantees completion. +1 for the details. Thx. –  Tom Apr 20 '13 at 15:18
add comment

Did you try using the --line-buffered option for grep?

tail -f /var/log/vsftpd.log | grep --line-buffered 'OK UPLOAD:' | while read line ; do
        #do a bunch of processing stuff - this takes some time
done
share|improve this answer
    
Yes I did, thanks. But no luck, same problem. –  Tom Apr 18 '13 at 17:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.