Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want the bash to go through an apache2 error log file and list all its lines (similar to cat), but in a way that it will count how many times every line is contained in the file and list only DISTINCT lines with a total count of them.

So for example, if the error log looks like this:

Error 1: file failed
Error 2: client failed
Error 3: server failed
Error 1: file failed

I'd like to have this echoed:

[2] Error 1: file failed
[1] Error 2: client failed
[1] Error 3: server failed
share|improve this question
    
Is using bash alone a strict requirement? – Ярослав Рахматуллин Nov 6 '12 at 18:29

Use sort and uniq:

sort file | uniq -c

If you want the ouput sorted by the number of occurrences, add | sort -n at the end.

share|improve this answer
    
I use these so often. cut, sort, uniq, wc ... magic +1 – captain_G Nov 8 '12 at 23:15

Using awk:

$ awk '{a[$0]++}END{for (i in a)print "["a[i]"]",i;}' file
[1] Error 3: server failed
[2] Error 1: file failed
[1] Error 2: client failed

Count the no. of occurrences in array a. At the END label, print all the elements of a.

share|improve this answer

For any of the mentioned examples to be really meaningful, you will need to strip off the dates from every line. Assuming that a line in your log looks like this:

[Wed Oct 24 10:17:17 2012] [notice] caught SIGTERM, shutting down

then you could do something like:

tail -n 5000 /var/log/apache2/error_log |\
  cut -d] -f2- |\
  awk '{a[$0]++} END \
       {for (i in a) {printf "%s\t%s\n", a[i], i;} }' |\
  sort -n  

This will produce output similar to:

13   [error] [client 127.0.0.1] column: address_phonerename: phone, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_streetrename: street, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_zipcoderename: zipcode, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_activerename: active, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_idrename: id, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameFirstrename: nameFirst, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameLastrename: nameLast, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_usernamerename: username, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected '}' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 6, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected ';' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 9, referer: http://l27/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_cityrename: city, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_idrename: address_id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_namerename: addr_name, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_streetrename: street, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_zipcoderename: zipcode, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_activerename: active, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_idrename: id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameFirstrename: nameFirst, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameLastrename: nameLast, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_usernamerename: username, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_commentrename: comment, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_countryrename: country, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_emailrename: email, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_phonerename: phone, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
451  [error] [client 192.168.0.103] File does not exist: /var/www/localhost/htdocs/WebServices
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .