3

I got a huge text file (log file) in my CentOS which I would like to remove top part of, probably couple of thousand lines each day. (Or probably just split into two)

I have search this site and found that most using grep, sed to remove the lines but output to another file. Not sure if it is possible that using shell script (bash) that I can update the file in place? instead of:

sed current file > new file
cp new file > current file

Thanks!

2
  • 1
    Why are you not using logrotate? Mar 10, 2012 at 2:05
  • logrotate is probably my last option, but if possible I would love to have bash / shell to remove lines with certain pattern. e.g. lines with date 2012/03/08 Mar 10, 2012 at 2:18

5 Answers 5

5

sed --in-place $filter $file

0
5

There is no simple way to remove lines from the beginning of file !

Even by using sed -i, you create a new file as shown with the following commands (> is my prompt):

> echo "Helo World" > toto
> ls -i toto
147543 toto
> sed -i -e 's/Helo/Hello/' toto
> ls -i toto
147292 toto

Notice that the inode number is not the same. This means that you create a new file with the same name, not that you modify the file in place.

This is important if your log file is open by a program while you perform this operation. If it is, you will create a new file while the program holding the file will keep writing to the old file. To show this, let's try the following:

for f in $(seq 1 100); do date; echo $f; sleep 1; done > file1&
ln file1 file2
sleep 5
sed -i -e '1,10d' file1
ls -l file1 file2
sleep 5
ls -l file1 file2

The 2nd ls will show the same size for file1 and a growing size for file2. If I had not done a ln before executing sed, the original file would have keep growing without being accessible via the file system hierarchy. This would result in use space on disk as shown by df but not shown by du. More information can be found here and here.

Log rotation is your friend here, but it cannot be done without help from the logging program. There should be a way to tell the program to close and reopen the file, so the new file would be used, but the log written after the beginning of the sed and the end of reopening the file could be lost. If you do not want to loose logs, you can copy the file first, ask for the program to reopen the file, and then modify the copied file. This is what logrotate allows you to do with minimal scripting.

You can read more on this subject here (apache 1.3), here (apache 2.4) and here (bind 9).

1
  • Thanks for the elaborate answer! Very good pointers and give me a new direction to look at my problem. Thanks! Mar 12, 2012 at 5:52
2

You can use Vim in Ex mode:

ex -sc '1d2000|x' file
  1. 1 move to first line

  2. 2000 select 2000 lines

  3. d delete

  4. x save and close

0

I got a huge text file (log file) in my CentOS which I would like to remove top part of

you can use tail to generate an new file containing only the last N lines

tail -n logfile >newlogfile
zcat logfile > $(date +%Y%m%d)logfile.gz && mv -f newlogfile logfile 2>/dev/null

, probably couple of thousand lines each day. (Or probably just split into two)

You can get the number of lines in the file with:

NUMLINES=$(awk 'END{print NR}' logfile)
#do some integer math and split with head and tail

I have search this site and found that most using grep, sed to remove the lines but output to another file. Not sure if it is possible that using shell script (bash) that I can update the file in place? instead of:

yes, you can use sed to delete the first n lines

#remove the first 10 lines
sed -i '1,10d' logfile
0

Set up a cron job to rotate the log? Hmm?

http://linuxcommand.org/man_pages/logrotate8.html

3
  • log rotate is my last option, because I want to chop off lines base on some rules I have written in bash, so I am looking into doing it with sed Mar 29, 2012 at 1:17
  • I'm looking at the logrotate manpage on ubuntu. It seems to have lots of bells and whistles including "prerotate" and "postrotate" command hooks. I wonder if that could be exploited to do some custom action, like filter the log.
    – Kaz
    Mar 29, 2012 at 1:44
  • I will try to explore down that path when I got the chance to overhaul the whole concept, but at this point I just need something from bash / sed to do that. Mar 30, 2012 at 8:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .