How would I use bash to count the total number of certain lines in a file?

Question

I want the bash to go through an apache2 error log file and list all its lines (similar to cat), but in a way that it will count how many times every line is contained in the file and list only DISTINCT lines with a total count of them.

So for example, if the error log looks like this:

Error 1: file failed
Error 2: client failed
Error 3: server failed
Error 1: file failed

I'd like to have this echoed:

[2] Error 1: file failed
[1] Error 2: client failed
[1] Error 3: server failed

Answer 1

Use sort and uniq:

sort file | uniq -c

If you want the ouput sorted by the number of occurrences, add | sort -n at the end.

Answer 2

Using awk:

$ awk '{a[$0]++}END{for (i in a)print "["a[i]"]",i;}' file
[1] Error 3: server failed
[2] Error 1: file failed
[1] Error 2: client failed

Count the no. of occurrences in array a. At the END label, print all the elements of a.

Answer 3

For any of the mentioned examples to be really meaningful, you will need to strip off the dates from every line. Assuming that a line in your log looks like this:

[Wed Oct 24 10:17:17 2012] [notice] caught SIGTERM, shutting down

then you could do something like:

tail -n 5000 /var/log/apache2/error_log |\
  cut -d] -f2- |\
  awk '{a[$0]++} END \
       {for (i in a) {printf "%s\t%s\n", a[i], i;} }' |\
  sort -n  

This will produce output similar to:

13   [error] [client 127.0.0.1] column: address_phonerename: phone, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_streetrename: street, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_zipcoderename: zipcode, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_activerename: active, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_idrename: id, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameFirstrename: nameFirst, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameLastrename: nameLast, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_usernamerename: username, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected '}' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 6, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected ';' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 9, referer: http://l27/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_cityrename: city, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_idrename: address_id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_namerename: addr_name, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_streetrename: street, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_zipcoderename: zipcode, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_activerename: active, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_idrename: id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameFirstrename: nameFirst, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameLastrename: nameLast, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_usernamerename: username, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_commentrename: comment, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_countryrename: country, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_emailrename: email, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_phonerename: phone, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
451  [error] [client 192.168.0.103] File does not exist: /var/www/localhost/htdocs/WebServices