Is there a way to remove the first N lines from a log that is being actively appended by an application?


No, operating systems like Linux, and it's filesystems, don't make provision for removing data from the start of a file. In other words, the start point of storage for a file is fixed.

Removing lines from the start of a file is usually accomplished by writing the remaining data to a new file and deleting the old. If a program has the old file open for writing, that file's deletion is postponed until the application closes the file.

As commenters noted, for the reasons given in my previous sentence, you usually need to coordinate logfile pruning with the programs that are writing the logs. Exactly how you do this depends on the programs. Some programs will close and reopen their logfiles when you send them a signal (e.g. HUP) and this can be used to prevent log records being written to a 'deleted' logfile, without disrupting service.

There are many utilities available for managing the size of log files, for example logrotate

Some programs have their own utilities. For example, the Apache webserver includes a rotatelogs utility.

  • 3
    But you should not do this while something is still has the file open and still appending to it, because it would write to the now deleted file, and you would lose those log messages. – Tarnay Kálmán Mar 27 '12 at 11:37
  • True. Even if you used the same filename. – Hennes Dec 9 '15 at 11:35
  • too bad the OS's don't let you, that would sure be convenient for log rotaters to not have to reload processes after rotation :| – rogerdpack Feb 2 '16 at 17:33

I think this task can be achieved with sed

sed -i '1,10d' myfile

would remove the lines from 1st to the 10th line form the file.

I think everybody should at least have a look at this sed 1 liners.

Note that this does not work for logfiles that are being actively appended to by an application (as stated in the question).

sed -i will create a new file and 'delete' the file that is being written to. Most applications will continue to write log records to the deleted log file and will continue to fill disk space. The new, truncated, log file will not be appended to. This will only cease when the application is restarted or is otherwise signalled to close and reopen its log files. At which point there will be a gap (missing log records) in the new log file if there has been any loggable activity between the use of sed and the application restart.

A safe way to do this would be to halt the application, use sed to truncate the log, then restart the application. This approach can be unacceptable for some services (e.g. a web-server with high throughput and high service-continuity requirements)

  • Do you know what happens to the applications that are appending? – Adam Matan Mar 27 '12 at 14:38
  • depends on the application. what are you using? – l1zard Mar 27 '12 at 17:21
  • Let's assume a normal open file handler which appends lines and flushes every now and then. – Adam Matan Mar 27 '12 at 19:01
  • then you can use it without -i and with a different output file instead. sed '1,10d' myfile > myfile.new. just spend a few hours in understanding how sed works. it's worth it. :) – l1zard Mar 27 '12 at 19:06
  • 7
    No, this should not work. sed -i creates a new file with the edited content and the old one is removed so you are not editing the active file: $ ls -i --- 6823554 testfile --- $ sed -i 's/test/final/' testfile --- $ ls -i --- 6823560 testfile ------ Please check how does sed -i work. Why does this wrong answer have so many upvotes? – pabouk Dec 16 '13 at 9:57

No. A solution to this generic problem of log file growth is log rotation. This involves the regular (nightly or weekly, typically) moving of an existing log file to some other file name and starting fresh with an empty log file. After a period the old log files get thrown away.

See: http://www-uxsup.csx.cam.ac.uk/~jw35/courses/apache/html/x1670.htm


Maybe copy, truncate, tail the copy back on to the size=0 truncation, and delete the copy?

Better yet tail to tail-copy, truncate original, concat tail-copy onto original.

You get lines in the log at tail length so better then a byte length limit.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.