1

I'd like to monitor my vsftpd logs and pull information from the logs to process the files that have been uploaded. Up till now everything was great and worked fine:

tail -n0 -F /var/log/vsftpd.log | while read line; do
    if echo "$line" | grep -q 'OK UPLOAD:'; then
        #do a bunch of processing stuff - this takes some time
    fi
done

I started doing some scale testing and uploaded 500 files at once. For some reason the log lines are missing or they are cut off when you upload so many things. For example a typical line looks like:

Sun Apr 7 09:08:51 2013 [pid 25409] [cam02430] OK UPLOAD: Client "206.132.183.201", "/20130407/09/20130407_090842D.avi", 531792 bytes, 426.14Kbyte/sec

But sometimes the line looks like this (if not missing entirely):

:08:51 2013 [pid 25409] [cam02430] OK UPLOAD: Client "206.132.183.201", "/20130407/09/20130407_090842D.avi", 531792 bytes, 426.14Kbyte/sec

It is cutting off the first few characters of the log. It doesn't do it for every line but most. Also many lines are just missing. I guess I have a race condition of my processing not finishing in time. How I can get around this?

EDIT I guess I have 3 options:

  1. Don't use tail and try to use a combination of things like a cron job/grep to check the log for a new entry. I imagine this could be tricky. How do I tell what is new in that log?

  2. Monitor the file change with monit or equivalent. I don't believe this is an option. The files get stored into somewhat random directories with files names that are time stamped.

  3. logstash or equivalent. I don't think these programs fit my needs. But maybe someone knows otherwise.

Right now I'm focusing on #1 with no good leads. So any thoughts on any of this would be appreciated.

7
  • I don't know anything about the logging behaviour of vsftpd, but you could (if no new subdirectories get created on a regular basis within the directory tree that you populate using the ftp process) use inotifywait with a for-loop to pick up on completed files (well, write/ colse events_.
    – tink
    Apr 12, 2013 at 19:25
  • Intensive log processing should not be done using shell scripts, I'd go for JSON log format + logstash/elasticsearch/kibana/statsd/graphite
    – dawud
    Apr 12, 2013 at 19:30
  • @tink I considered that when I started this project but was not clear on how well it would handle the close events (successful upload) and the files get stored in new subdirectories that are always changing according to date. That part is out of my control.
    – Tom
    Apr 13, 2013 at 0:05
  • @Tom When the character get cut off, can you verify that it only cut off characters of that particular line or you are actually missing multiple lines in between.
    – John Siu
    Apr 15, 2013 at 7:50
  • @JohnSiu Actually it is missing lines as well.
    – Tom
    Apr 15, 2013 at 17:36

4 Answers 4

1

You may start prevent forking a "grep" process for every line in the log, by rewriting your code like this:

tail -n0 -F /var/log/vsftpd.log | grep 'OK UPLOAD:' | while read line ; do
        #do a bunch of processing stuff - this takes some time
done

But I agree with other posters, shell is not really suitable for heavy text/string manipulation. You should consider using use another scripting engine (at least awk, or better perl or python)

1
  • Thanks, that sounded promising but that suffers the same problem. I'm exploring other options.
    – Tom
    Apr 16, 2013 at 18:01
1

I had good experience with Perl using the file open '<' options, if you are willing to relax your limit to shell scripts only.

e.g. https://stackoverflow.com/questions/1425223/how-do-i-read-a-file-which-is-constantly-updating

1

As @epoon suggested, Perl might be a good choice:

#!/usr/bin/env perl

open(FH,'<',$ARGV[0]) || die("Could not open file $ARGV[0]\n");
for (;;) {
    while (<FH>) {
    if (/OK UPLOAD/) { ## this is where you do your processing. As an example, 
                       ## I am collecting the file's info
        /^(.+?)\s*\[pid\s*(\d+).+?\"([\d\.]+).+?\"(.+?)\".+?(\d+).+?\s([^\s]+)/;
        my ($date,$pid,$client,$filename,$size,$rate) = ($1,$2,$3,$4,$5,$6);
        print "On $date, process $pid uploaded file \"$filename\" of $size bytes from $client at $rate\n"

    }
    }
    # eof reached on FH, but wait a second and maybe there will be more output
    sleep 1;
    seek FH, 0, 1;      # this clears the eof flag on FH
}

Running this script on /var/log/vsftpd.log should give output like

On Sat Apr 20 16:30:05 CEST 2013, process 25409 uploaded file "/20130407/09/20130407_090842D.avi" of 531792 bytes from 206.132.183.201 at 426.14Kbyte/sec

I would be happy to update this if you specify what exactly you want to do with the log info.

1
  • I think this might work, but at any rate I went down a different route. I've written a notification api into vsftpd. This guarantees completion. +1 for the details. Thx.
    – Tom
    Apr 20, 2013 at 15:18
0

Did you try using the --line-buffered option for grep?

tail -f /var/log/vsftpd.log | grep --line-buffered 'OK UPLOAD:' | while read line ; do
        #do a bunch of processing stuff - this takes some time
done
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .