Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a log file, each line in the log is prepend with a date, like so:

2012-03-06 11:34:48,657 blah blah blah...

How do I grep this file and get only the lines from 8am to 11pm only?

My intention is I want to count the number of errors happening within 8am to 11pm.

share|improve this question
up vote 15 down vote accepted
egrep '^[^ ]+ (0[89]|1[0-9]|2[012]):'

Detailed explanation can be found in various regex (regular expression) tutorials; egrep uses "POSIX extended" syntax (man 7 regex).

  • The first ^ means "start of the line".

  • [^ ]+ just matches the date field, regardless of the actual date.

    • [...] means "any character between the brackets", so [89] will match either 8 or 9; [0-9] is any number, and [^ ] is anything except a space (because of the ^ inside brackets).

    • + means "one or more of the previous" (for example, a+ would match a, aaa, and aaaaaaaa).

    • So ^[^ ]+ will start with the beginning of line, and match as many non-space characters as it can.

  • (...|...|...) means "either of the given patterns", so (0[89]|1[0-9]|2[012]) means "either 0[89] or 1[0-9] or 2[012]". It will match all numbers from 08 to 22.


A somewhat better option is:

awk -F'[: ]' '$2 >= 8 && $2 <= 22 { print }'

The -F option splits every line into separate fields according to the [: ] regex (matching either : or a space), and the awk script checks the 2nd column (the hour).

share|improve this answer
    
Hi, it works.. but can you add a little explanation how it works? I dont get the ^[^ ]+ part.. – Rosdi Jun 21 '12 at 11:14

Why bother using grep? You can simply use sed.

example:

sed -n '/Jun 17 13:39:54/ , /Jun 18 10:50:28/p' kern.log

This will print all the logs between June 17 13:39:54 and June 18 10:50:28

share|improve this answer
1  
I think the OP wants all logs between 8am and 11pm, bot just the ones from a specific day. – Dennis Jun 21 '12 at 11:14
1  
He can do that too. Eg: sed -n '/2012-3-06 11:34:48/,/2012-3-06 16:34:48/p' logfile.name – Nima Jun 21 '12 at 11:16
1  
Nasty typo. I meant: not the ones from a specific day. – Dennis Jun 21 '12 at 11:29
1  
The problem with this approach is that there has to be a line in the log file with that timestamp. In the above example, if there is no line with timestamp 2012-03-06 11:34:48, then nothing will print out. The awk approach above would work for such cases (i.e. you want all logs with hour between 8 and 11, but you don't know if there is a log entry at time 2012-03-06 08:00:00 or anything for that hour at all). – user650654 Nov 19 '14 at 21:03

There's actually a much easier way to do this.

Download/Documentation: autodrgrep.kl.sh

Command:

./autodrgrep.kl.sh   notchef   /tmp/client.log   '2016-05-08_08:00:00,2016-05-08_23:00:00'   'INFO'   'a2ensite'   5  10  -show

Explanation:

  • autodrgrep.kl.sh is the tool name.

  • notchef is an option that is passed to the tool to tell it what to do. In this particular case, it is telling the tool what type of log file /tmp/client.log is.

  • /tmp/client.log is of course the log file.

  • 2016-05-08_19:12:00,2016-05-08_21:13:00 is the range of date from within the log that you wish to scan

  • "INFO" is one of the strings that is in the lines of logs that you're interested in.

  • "a2ensite" is another string on the same line that you expect to find the "INFO" string on. Specifying these two strings (INFO and a2ensite) isolates and processes the lines you want a lot quicker, particularly if you're dealing with a huge log file.

  • 5 specifies Warning. By specifying 5, you're telling the program to alert as WARNING if there are at least 5 occurrences of the search strings you specified

  • 10 specifies Critical. By specifying 10, you're telling the program to alert as CRITICAL if there are at least 10 occurrences of the search strings you specified.

  • -show specifies what type of response you'll get. By specifying -shown, you're saying if anything is found that matches the specified patterns, output to screen.

Sample run:

# ./autodrgrep.kl.sh notchef  /tmp/client.log   '2016-05-08_19:12:00,2016-05-08_21:13:00' 'INFO' 'a2ensite'  5  10  -show

[2016-05-08 19:12:58-07:00] INFO: Processing template[/usr/sbin/a2ensite] action create (apache2::default line 90)
[2016-05-08 19:12:58-07:00] INFO: Processing execute[a2ensite default] action run (apache2::default line 24)
[2016-05-08 19:12:58-07:00] INFO: execute[a2ensite default] ran successfully
[2016-05-08 19:13:09-07:00] INFO: Processing execute[a2ensite nagios3.conf] action run (logXrayServer::install line 24)
[2016-05-08 19:13:12-07:00] INFO: execute[a2ensite default] sending restart action to service[apache2] (delayed)
[2016-05-08 19:42:57-07:00] INFO: Processing template[/usr/sbin/a2ensite] action create (apache2::default line 90)
[2016-05-08 19:42:57-07:00] INFO: Processing execute[a2ensite default] action run (apache2::default line 24)
[2016-05-08 19:42:57-07:00] INFO: execute[a2ensite default] ran successfully
[2016-05-08 19:43:08-07:00] INFO: Processing execute[a2ensite nagios3.conf] action run (logXrayServer::install line 24)
[2016-05-08 19:43:11-07:00] INFO: execute[a2ensite default] sending restart action to service[apache2] (delayed)
[2016-05-08 20:12:58-07:00] INFO: Processing template[/usr/sbin/a2ensite] action create (apache2::default line 90)
[2016-05-08 20:12:58-07:00] INFO: Processing execute[a2ensite default] action run (apache2::default line 24)
[2016-05-08 20:12:58-07:00] INFO: execute[a2ensite default] ran successfully
[2016-05-08 20:13:10-07:00] INFO: Processing execute[a2ensite nagios3.conf] action run (logXrayServer::install line 24)
[2016-05-08 20:13:12-07:00] INFO: execute[a2ensite default] sending restart action to service[apache2] (delayed)
[2016-05-08 20:42:59-07:00] INFO: Processing template[/usr/sbin/a2ensite] action create (apache2::default line 90)
[2016-05-08 20:42:59-07:00] INFO: Processing execute[a2ensite default] action run (apache2::default line 24)
[2016-05-08 20:42:59-07:00] INFO: execute[a2ensite default] ran successfully
[2016-05-08 20:43:09-07:00] INFO: Processing execute[a2ensite nagios3.conf] action run (logXrayServer::install line 24)
[2016-05-08 20:43:12-07:00] INFO: execute[a2ensite default] sending restart action to service[apache2] (delayed)
[2016-05-08 21:12:59-07:00] INFO: Processing template[/usr/sbin/a2ensite] action create (apache2::default line 90)
[2016-05-08 21:12:59-07:00] INFO: Processing execute[a2ensite default] action run (apache2::default line 24)
[2016-05-08 21:12:59-07:00] INFO: execute[a2ensite default] ran successfully
23
2---78720---23---ATWFILF---(2016-05-08)-(19:12)---(2016-05-08)-(21:13) SEAGM

What if the user specified date range or time frame is not in the log?

Each run of the above command will always have a line (last line of the output) that either says "ATWFILF" or "ETWNFILF".

  • ATWFILF means that the actual date range or time frame you requested searched was found in the log. So this is very good.

  • ETWNFILF means the actual date range or time frame you requested searched was NOT found in the log. In this case, the closest time to the time you specified will be detected and used instead.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.