Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I got a huge text file (log file) in my CentOS which I would like to remove top part of, probably couple of thousand lines each day. (Or probably just split into two)

I have search this site and found that most using grep, sed to remove the lines but output to another file. Not sure if it is possible that using shell script (bash) that I can update the file in place? instead of:

sed current file > new file
cp new file > current file

Thanks!

share|improve this question
    
Why are you not using logrotate? –  Ignacio Vazquez-Abrams Mar 10 '12 at 2:05
    
logrotate is probably my last option, but if possible I would love to have bash / shell to remove lines with certain pattern. e.g. lines with date 2012/03/08 –  forestclown Mar 10 '12 at 2:18

4 Answers 4

up vote 5 down vote accepted

sed --in-place $filter $file

share|improve this answer
    
Thanks, this is so simple... –  forestclown Mar 10 '12 at 6:55

There is no simple way to remove lines from the beginning of file !

Even by using sed -i, you create a new file as shown with the following commands (> is my prompt):

> echo "Helo World" > toto
> ls -i toto
147543 toto
> sed -i -e 's/Helo/Hello/' toto
> ls -i toto
147292 toto

Notice that the inode number is not the same. This means that you create a new file with the same name, not that you modify the file in place.

This is important if your log file is open by a program while you perform this operation. If it is, you will create a new file while the program holding the file will keep writing to the old file. To show this, let's try the following:

for f in $(seq 1 100); do date; echo $f; sleep 1; done > file1&
ln file1 file2
sleep 5
sed -i -e '1,10d' file1
ls -l file1 file2
sleep 5
ls -l file1 file2

The 2nd ls will show the same size for file1 and a growing size for file2. If I had not done a ln before executing sed, the original file would have keep growing without being accessible via the file system hierarchy. This would result in use space on disk as shown by df but not shown by du. More information can be found here and here.

Log rotation is your friend here, but it cannot be done without help from the logging program. There should be a way to tell the program to close and reopen the file, so the new file would be used, but the log written after the beginning of the sed and the end of reopening the file could be lost. If you do not want to loose logs, you can copy the file first, ask for the program to reopen the file, and then modify the copied file. This is what logrotate allows you to do with minimal scripting.

You can read more on this subject here (apache 1.3), here (apache 2.4) and here (bind 9).

share|improve this answer
    
Thanks for the elaborate answer! Very good pointers and give me a new direction to look at my problem. Thanks! –  forestclown Mar 12 '12 at 5:52

I got a huge text file (log file) in my CentOS which I would like to remove top part of

you can use tail to generate an new file containing only the last N lines

tail -n logfile >newlogfile
zcat logfile > $(date +%Y%m%d)logfile.gz && mv -f newlogfile logfile 2>/dev/null

, probably couple of thousand lines each day. (Or probably just split into two)

You can get the number of lines in the file with:

NUMLINES=$(awk 'END{print NR}' logfile)
#do some integer math and split with head and tail

I have search this site and found that most using grep, sed to remove the lines but output to another file. Not sure if it is possible that using shell script (bash) that I can update the file in place? instead of:

yes, you can use sed to delete the first n lines

#remove the first 10 lines
sed -i '1,10d' logfile
share|improve this answer

Set up a cron job to rotate the log? Hmm?

http://linuxcommand.org/man_pages/logrotate8.html

share|improve this answer
    
log rotate is my last option, because I want to chop off lines base on some rules I have written in bash, so I am looking into doing it with sed –  forestclown Mar 29 '12 at 1:17
    
I'm looking at the logrotate manpage on ubuntu. It seems to have lots of bells and whistles including "prerotate" and "postrotate" command hooks. I wonder if that could be exploited to do some custom action, like filter the log. –  Kaz Mar 29 '12 at 1:44
    
I will try to explore down that path when I got the chance to overhaul the whole concept, but at this point I just need something from bash / sed to do that. –  forestclown Mar 30 '12 at 8:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.