Is there a way to create special files under Linux that would keep only, say the 100 last written lines? I have a process filling a log file, and I'd like to regularly parse its 100 last lines.

I know I could use some kind of logrotate, but is there a way to create a special file that would fill up till it reaches 100 lines, then, adding a line removes the oldest one, so that the file only keeps 100 lines? (a kind of line-based FIFO)

Thanks a lot

  • 2
    tail does that, so you could write a cron job to fill the file on whatever refresh interval you want. just overwrite with tail <file> > <targetfile>. if your refresh rate is high enough, it will act like a FIFO buffer. – Frank Thomas Feb 15 '16 at 22:51
  • 1
    While it would be cool to have a custom virtual file system that you can cat to get derived content (like a ring buffer), these aren't very popular. They are called log-structured file systems. It would be perfect for that scenario. en.wikipedia.org/wiki/Log-structured_file_system . For now, I suggest you just write a script that does a tail -n 100 to itself. – David Betz Feb 16 '16 at 0:31
  • Thanks, I was not aware of such filesystems! The ideal would have been special files that act like such ring buffers (that would avoid needing to install and mount a whole FS, and I thought mainstream Linux distros already supported this). I'll definitely end up using tail, but I'm happy I have learnt something :) – Daladim Feb 16 '16 at 21:49
  • In general there is a way to create files that are just what you want them to be: custom FUSE (Filesystem in Userspace). It is not as easy as you probably hoped for, but it's possible. – Kamil Maciorowski Feb 22 '16 at 9:48

Logs are text(like) files and being that kind, appending new disk blocks to them when it's demanded by new lines is a quick action well supported by any file system.

However, constantly dropping out the first line when a new is coming would mean reorganizing at least some if not all the blocks of the file CONSTANTLY. That would mean big overhead dedicated to logging while one of the most important characteristics we want from logging is that it be lightweight .

File systems are not prepared for this (at least I haven't heard about this type), that's why logrotate/tail/database-backed logging are used where the last records are of importance.

  • Thanks for this low-level hint I had not thought about :) – Daladim Feb 16 '16 at 21:53

Presuming that you have a script that processes only the last 100 lines each time it is run, these lines are best captured with the tail command, as it does pretty much what you want to do. The key here is the -n switch which dictates how many lines it should capture, starting from the end.

You can incoprorate tail -n 100 somefile.log in your script directly, or you can periodically run tail -n 100 somefile.log > onlylast100lines.log to create a file with only the last 100 lines. The latter approach will rewrite the target file every time, so no need to delete it between each run.

  • I was aware of tail (and I'll finally end up using it), but out of curiosity, I wanted to know whether there were Linux features I did not know that would do he job. Thanks anyway. – Daladim Feb 16 '16 at 21:51

Elaborating on Jarmunds method, you could make a bash script like this:

while true; do 
    tail -n 100 somefile.log > onlylast100lines.log
    sleep 5

this would parse the last 100 lines of somefile.log into onlylast100lines.log every 5 seconds. The file would be overwritten each time, so it would be always only the last 100 lines. This could be added to your login script or to whatever runlevel you want this to be executed.

  • I was aware of tail (and I'll finally end up using it), but out of curiosity, I wanted to know whether there were Linux features I did not know that would do he job. Thanks anyway. – Daladim Feb 16 '16 at 21:51

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.