I have an application running remotely on a vendor *NIX server. This application writes out log entries to a file. This logging process is very much a typical append-only approach, so the file only ever increases in size, and once content is written - it can never be modified.

I would like to pull these log entries onto my local *NIX server within my own network in real(ish) time, so that the log file could be tailed on my own locals erver.

Given the append-only nature of the log file, one would assume it is fairly easy to track the delta as new lines (bytes) are added, and only pull across changes as the log file grows.

I was hoping for some inspiration regarding available tools to perform this task....?

I would be happy to pull the files over ssh, which I asusme will be a pre-req. SFTP is another alternative available, if that's of any use to applications.


Unless I misunderstand the question, this is what syslog was made for - general remote logging functionality, not just your specific desire to copy the new parts of one log file. Here is one whitepaper to get you started. Basically you configure the remote device to send logs to your system, and your local device to collect them. Here is a more recent article.

Since it's fairly standard, you can have one central syslog server collecting logs for all sorts of equipment (many of my wireless access points include the ability to send to syslog, and it's the only way one can look at logs from previous power-cycles on those devices, as the local log is non-persistant.)

  • This is a good suggestion. However, it's worth noting that it involves getting the vendor to modify their OS setup, which could be tricky. Also, I will need to look into how this will operate if the network connection is flakey. I have a very high uptime, but we're operating over a VPN - and I don't trust it fully. – jwa Oct 23 '13 at 6:48

Further to the suggestions from Cristian, rsync does seem to be a viable option ... but it does require scheduling (by cron?) to continuously evaluate and update as required.

The --apend flag seems to be our friend here. This will assume that the content of files previously copied, but subsequently updated, has not changed.

For example:

  1. Cron invokes, remote file 10MB, local file does not exist a. Copies entire 10MB from remote server b. rsync exists
  2. Cron invokes, remote file 12MB, local file 10MB a. rsync assumes first 10MB of each file is the same b. rsync downloads the 2mb delta c. rsync exits

Continually invoking the cron is a bit of a pain. Although adding it to the inittab to ensure it re-spawns whenever it exists might be a possibility. This would mean that the rsync process is always running, starting the next pull-down immediately after the previous one finishes.

Any other suggestions?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.