Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I just wrote the most contorted command I've ever written and I want to know how I may make it better.

I wrote this:

grep -E '00[7-9]\.|0[1-9][0-9]\.' filename.log | awk '{print $6}' | sed 's/\(.*\):.*/\1/' | sort | uniq -c | sort -rn

An example input:

2011/06/30 07:59:43:81 20626 code_file.c (252): FunctionName: 009.63 seconds

Basically what it's doing is going through a log file that list the number of seconds that it took a command to execute and grabbing any of them that took between 7 and 99 seconds to execute. Then awk is printing the sixth word, which is the function name followed by a colon. Then sed is removing the colon and any trailing whitespace, then it's getting sorted, counted, and then sorted based on it's count.

I'm on HP-UX so some of my tools are limited, but I know that awk can do what I just did with sed. Can someone help me de-complicate my command?

share|improve this question
An example input line would be helpful. – grawity Jun 30 '11 at 20:57
You mentioned your tools are limited, can you elaborate? For instance, since Gravity's answer is in perl, do you have perl? – bbaja42 Jun 30 '11 at 21:15
What OS doesn't have Perl? – grawity Jun 30 '11 at 21:18
I have perl. I'm just stating that the HP-UX tools often have less functionality than their GNU counter part. For example, the HP-UX grep lacks -rAB and many more essential options, also uniq lacks -w, etc. – Malfist Jul 1 '11 at 0:24
up vote 3 down vote accepted
awk '/00[7-9]\.|0[1-9][0-9]\./ { # for lines matching the regex
       split($6, c, /:/)         # take the part of field 6 before the colon
       cs[ c[1] ]++              # and increment the counter for that string
     END {                       # after all lines have been read
       for (c in cs) {           # step through the counters
         print cs[c], c          # and output the count followed by the string
                                 #   ("," adds a space automatically)
     }' filename.log | sort -rn  # standard awk doesn't support sorting, sadly

I continue to be amazed at the number of people who apparently believe that neither awk nor sed can do pattern matching, so they have to add a grep invocation.

share|improve this answer
Do you care to break that down? I don't fully understand what you're doing. – Malfist Jul 1 '11 at 0:25
First stanza: on any line which matches the regex, split field 6 on colons and increment a counter based on the first component (this duplicates your sed). (awk arrays are more like Perl hashes or Python dicts than ordinary arrays.) Second stanza executes after all lines have been read, and prints out the items and counts in the array; this replaces the sort | uniq -c part. I can split the lines up and add some comments. – geekosaur Jul 1 '11 at 0:30

I'm so going to be downvoted for this...

#!/usr/bin/env perl
use strict;

my %counts;
while (my $line = <>) {
    my @line = split(/\s+/, $line);
    if ($line[6] >= 7) {
        $line[5] =~ /(.+):/ and $counts{$1}++;

my @sorted = sort {$counts{$b} <=> $counts{$a}} keys %counts;

printf("%7d\t%s\n", $counts{$_}, $_) for @sorted;
share|improve this answer
That's hardly a command eh? :p – BloodPhilia Jun 30 '11 at 21:23
@BloodPhilia: Half of the commands I use daily look like that. (The other half is bash and Python.) I could have written it as a perl -ne one-liner, but it would be next to impossible to understand, in comparison. – grawity Jun 30 '11 at 21:24
Perl could be SuperUser's jQuery. ;-) – Patches Jun 30 '11 at 21:54

Your command is a bit brittle as it will fail if the filename has a space in it. Otherwise, your command is actually not too bad. It somewhat a matter of taste, but I find a chain of simple piped commands much easier to grok than one complex command, such as the large awk someone posted. It's almost likely programming in a functional style.

You could, however, change the grep to eliminate the awk and sed, but now the regex is much harder to understand:

grep -P -o '(?<=\): ).+?(?=: 00[7-9]|0[1-9]|1)' | sort | uniq -c | sort -nr

To explain the regex, we use perl style re (-P param) and use look behind (?<=) and look-ahead (?=) to isolate the match to exactly the function name. Note that the look-behind and look-ahead are zero-width, meaning they aren't considered part of the match, but control what the match actually will be. Since the match is now exactly the function name, we can use -o to tell grep to only print the matching string rather than the entire line. I think you should leave what you have, unless you think a filename with spaces is a possibility.

share|improve this answer

While I'm at it:

grep -E '00[7-9]\.|0[1-9][0-9]\.' "$@" | awk '{print $6}' |
    sed 's/:$//' | sort | uniq -c | sort -rn

The original command is not that complicated, it's the repetition for every log that makes it look so. Stick it into a script file (or a function), call it sortbytime, and there – you have a simple one-word command.

share|improve this answer
Is there some reason this needs to be a separate answer? – nhinkle Jun 30 '11 at 23:02
@nhinkle: Probably not. Feel free to delete it. (I should merge them, but can't do that using the phone's browser.) – grawity Jun 30 '11 at 23:56
The're two different answers. If someone liked this one and didn't like the other they can vote that way. If they were merged, they couldn't. It'd be an all or nothing. – Malfist Jul 1 '11 at 0:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .