Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have an input file like this:

ATTACK-RESPONSES id check returned root
BACKDOOR ACKcmdC trojan scan
BACKDOOR hack-a-tack attempt 
BACKDOOR WinCrash 1.0 Server Active
ICMP Destination Unreachable Port Unreachable
ICMP Destination Unreachable Port Unreachable
ICMP Destination Unreachable Port Unreachable
SNMP trap tcp

Output:

1 ATTACK-RESPONSES id check returned root
3 BACKDOOR
3 ICMP Destination Unreachable Port Unreachable 
1 SNMP trap tcp 

I want to find and match the longest common substring in each line of the text and return the number of repetitions for each of them – so from input I'd need to get Output.

How could I do this?

share|improve this question
1  
Please give a more specific example of what your expected output is. I don't understand what "the longest common substring in each line" is. (Please capitalize the pronoun "I" in your posts.) What actual problem do you need to solve? It seems like you want to parse your first file for something specific? What would that be? –  slhck May 16 '13 at 19:09
1  
"longest common substring" limited to complete words I guess. –  Hauke Laging May 16 '13 at 19:11
1  
@slhck: I think what he’s trying to say is “take all the lines that begin with the same word, and then find the longest string (or perhaps the longest string of words) that is an initial substring of all of those lines.” (And also count that group of lines that begin with the same word.) –  Scott May 16 '13 at 19:25
1  
I guess the task is this: (a) Strip off the irrelevant parts of the lines. (b) Sort the lines. (c) Repeat for every line: Is the line identical to the following line? If not: Leave out the last word and compare again. If they match: Compare the following lines to this string and take all matching lines out of the set. The sorting is non-trivial though: Shorter lines must be later than lines whose beginning matches the shorter line. –  Hauke Laging May 16 '13 at 19:48
1  
I think the easiest way to solve this, still, would be to find out what the actual issue is. Why would one need the count of said lines? –  slhck May 16 '13 at 20:38

1 Answer 1

up vote 1 down vote accepted

This is rather difficult with a single pass, and even more difficult if you don't assume that the start needs to be the same.

You could write a perl script that matches regular expressions against previous lines, sort of like this:

my @words_on_line = split(/ /, $current_line);
my $i = 0; my $substring = ''; my $expression = '';
do {
  $expression = join(' ', $words_on_line[0..$i++]);
  if ($previous_line =~ m/^$expression/) {
    $substring = $expression;
  }
} until ($substring ne $expression);

Then, you'd also have to check the next line and potentiall reduce the substring match, e.g. when you have

FOO a b c
FOO a b
FOO d

The first match (from 2 to 1) would give you FOO a b, but comparing below, you'd only get FOO.

Which boils down to: you need to buffer your lines until you get a no-match line. So instead of printing, you'd do something like

unless ($substring) {
  push @buffer, $current_line;
  foreach (@buffer) {
    unless (m/$substring/) {
      $buffer_substring = $substring;
    }
  }
} else {
  print scalar @buffer, " $buffer_substring\n";
}

And then you'd just combine this.

If it's not "first common starting from beginning of line", you'd have to check every possible of sequence of words against every possible sequence of words in other lines, which is utterly complicated and which I will not reproduce here.

share|improve this answer
    
thank you but how i could run this? i try to run in this ay : perl yourscript.pl my file but it returns error of syntax error at sc.pl line 4, near "];" and i try to pipe bute this error still there! :( –  Christopher May 17 '13 at 5:37
1  
These are only parts of a script, not a whole solution. –  towo May 17 '13 at 17:15

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.