Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am testing the regex needed for creating field extraction with Splunk for nmap and think I might be close...

Example full line:

Host: 10.0.0.1 (host)   Ports: 21/open|filtered/tcp//ftp///, 22/open/tcp//ssh//OpenSSH 5.9p1 Debian 5ubuntu1 (protocol 2.0)/, 23/closed/tcp//telnet///, 80/open/tcp//http//Apache httpd 2.2.22 ((Ubuntu))/,  10000/closed/tcp//snet-sensor-mgmt///  OS: Linux 2.6.32 - 3.2  Seq Index: 257  IP ID Seq: All zeros

I've used underscore "_" as the delimiter because it makes it a little easier to read.

root@host:/# sed -n -e 's_\([0-9]\{1,5\}\/[^/]*\/[^/]*\/\/[^/]*\/\/[^/]*\/.\)_\n\1_pg' filename

The same regex with the escape characters removed:

root@host:/# sed -n -e 's_\([0-9]\{1,5\}/[^/]*/[^/]*//[^/]*//[^/]*/.\)_\n\1_pg' filename

Output:

... ... ...
Host: 10.0.0.1 (host)   Ports: 
21/open|filtered/tcp//ftp///, 
22/open/tcp//ssh//OpenSSH 2.0p1 Debian 2ubuntu1 (protocol 2.0)/, 
23/closed/tcp//telnet///, 
80/open/tcp//http//Apache httpd 5.4.32 ((Ubuntu))/, 
10000/closed/tcp//snet-sensor-mgmt///   OS: Linux 9.8.76 - 7.3  Seq Index: 257 IPID Seq: All zeros
... ... ...

As you can see, the pattern matching appears to be working - although I am unable to:

1 - match the pattern on both the end of line ( comma , and white/tabspace). The last line contains unwanted text (in this case, the OS and TCP timing info). A boolean "OR" for the two characters (comma and whitespace) seems not to match.

...(\,|\s)

and

2 - remove any of the un-necessary data - i.e. print only the matching pattern. It is actually printing the whole line. If i remove the sed -n flag, the remaining file contents are also printed. I can't seem to locate a way to only print the matched regex.

i.e why, when I explicitly tell it not to, is sed printing these lines? =>

Host: 10.0.0.1 (host) Ports:

and

OS: Linux 2.6.32 - 3.2  Seq Index: 257  IP ID Seq: All zeros

Being fairly new to sed and regex, any help or pointers is greatly appreciated!

share|improve this question
    
That is some crappy output you are working with. Is there no way to get more regular output to begin with? –  Daniel Andersson May 31 '12 at 7:12
    
it's standard output from greppable nmap (-oG). The crappy bit is that the pattern is repeated throughout the line - that's also doing my head in. Effectively it is searching the line repeatedly for: "number/state/proto//service//description/," –  Ovid May 31 '12 at 7:44
    
Have you read nmap.org/book/output-formats-grepable-output.html ? That contains some sample code to separate the fields. It is tab separated, which I did not see in your sample output, and which makes it a lot less troublesome. –  Daniel Andersson May 31 '12 at 9:56
    
Yes, I have read the Nmap doc. However, Splunk does not utilise awk, or perl in it's transforms when defining field extractions. e.g. [host] REGEX = Host:\s(\d+\.\d+\.\d+\.\d+) [portsnum] REGEX = ([0-9]*)/([^/]*)/([^/]*)//[^/]* I may be expecting/asking too much from Sed; from how far I've gotten it seems doable but again I'm new to it. I'll keep playing. Thanks for the feedback though, @DanielAndersson –  Ovid May 31 '12 at 23:47

1 Answer 1

First, I would encourage you to look into the XML output of Nmap (available with the -oX flag), which is the officially supported machine-readable output format. Greppable (-oG or .gnmap) output is deprecated, and so does not include helpful information from newer features of Nmap such as traceroute and NSE scripts.

To answer your questions directly,

  1. the issue with matching either a comma or a space is causing errors because the alternation pipe character (|) must be escaped, not the comma. Also, you probably always want to match a whitespace character, but only sometimes the comma. This is how I would do that:

    ,\?\s
    

I'm not using grouping, since there's no alternation ("or" pipe).

  1. sed is not printing "lines" that you don't want, it's printing the pattern space. The sed info page explains how sed works, and is a great reference for writing sed scripts. You essentially have 2 spaces to work with, and sed will print the entire contents of the pattern space when you use the p command.

As an example of how you might go about this, here's my take on a sed script to print just the port information from a .gnmap file:

#!/usr/bin/sed -n 

#First, strip the beginning (Host and Ports labels) off
s/.*Ports: //

#Now match a port entry, consuming the optional comma and whitespace
#The comma and whitespace are replaced with a newline
s_\([0-9]\{1,5\}/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*/\),\?\s_\1\n_

#If we made a successful substitution, jump to :matched, 
t matched
#otherwise skip to the next input line
d

:matched
#Print the pattern space up to the first newline
P
#Then delete up to the first newline and start over with what's left
D

All together in one line, that would look something like this:

sed -n -e 's/.*Ports: //;s_\([0-9]\{1,5\}/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*/[^/]*/\),\?\s_\1\n_;t matched;d;:matched;P;D' file.gnmap

Note also, that you can't count on some of the fields in the port specification to always be empty. If version detection was done on a RPC service, for instance, the SunRPC info field will be populated.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.