Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

When viewing Apache's log files there are a lot of error message repeated. I would like to find out which error messages are most frequent and which are unique.

I would like a simple script that can provide me with a count of the different unique lines.

I know of Perl/Python/etc., but I would prefer to use built-in tools such as cut/find/grep/sed, etc.

I can get a simple list of entries using sed. The following displays a list of unique error messages:

$ sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" error.sml.log | sort -u
Apache configured -- resuming normal operations
client denied by server configuration
Digest
File does not exist
request failed

This works to count the number of occurrences. It's just not that useful as it does not show what the counts relate to:

$ sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" error.sml.log | sort -u | xargs -I{} grep -oc {} error.sml.log
1
3886
2
6091
20

I would like the output to look more like this:

1    Apache configured -- resuming normal operations
3886 client denied by server configuration
2    Digest
6091 File does not exist
20   request failed
share|improve this question

1 Answer 1

up vote 2 down vote accepted
sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" error.sml.log | sort | uniq -c
      1  Apache configured -- resuming normal operations
   3886  client denied by server configuration
      2  Digest
   6091  File does not exist
     20  request failed

There is a tool which does exactly that uniq. I wish I had found this a couple of hours ago.

There are a couple of useful options such as uniq -d only show lines with more then 1 entry

$ sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" error.sml.log | sort | uniq -cd
   3886  client denied by server configuration
      2  Digest
   6091  File does not exist
     20  request failed

uniq -u only show uniq lines only 1 entry

$ sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" error.sml.log | sort | uniq -u  
  Apache configured -- resuming normal operations

Now I can process our billion line log files and make some sense of what is going on in side them.

share|improve this answer
1  
If I had been on I could've told you. It comes up a lot, a few searches should've found it (that's how I learned about it, I wanted to know how many times conky threw an error into my .xsession-errors file, it was over 182,000 times!) –  Rob Apr 19 '12 at 13:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.