0

I'm using grep/sed to parse a large binary file full of nulls for some specific data. This is in a windows environment using gnuwin32.

Assuming I had some to return some data from a binary file that looked like this (hex representation):

42 9D E1 0A 01 FF FF FF FF FF FF FF 7F 00 FE FF FF 0A 01 E1 0A 01 C0 0B 00 4D 00
00 9C E1 0A 01 2C 41 38 B4 15 FB 49 00 4D 00 41 00 48 00 4F 00 53 00 54 00 31 00
32 00 33 00 2E 00 73 00 75 00 62 00 2E 00 64 00 6F 00 6D 00 61 00 69 00 6E 00 2E
00 73 00 74 00 61 00 2E 00 6F 00 72 00 67 00 2E 00 61 00 75

But only matching this part of the above:

49 00 4D 00 41 00 48 00 4F 00 53 00 54 00 31 00 32 00 33

How exactly would I go about it using grep and/or sed? The presence of nulls does some bad things to grep, especially since I actually need to return the data as part of an automated file parser.

The below grep comes close to doing what I want:

grep -Prino ".{0,100}I\000M\000A\000H\000O\000S\000T\0001\0002\0003.{0,100}" "d:\dhcp.mdb"

However it just returns "Binary file d:\dhcp.mdb matches" rather than the matched pattern, probably because of the nulls. If this did work, once I had 100 chars to either side I'd grep this subset for what I needed by also matching for the IP address (which follows a greppable format) and domain name (which always ends with 3 nulls)

Since it's a database file I could probably use a different method like actually interact with the database, but I feel I'm pretty close with this method.

Using Sed I'm able to actually see return data from a much smaller file where I've pasted some of the relevant data:

sed -rn "/\I\x00M\x00A\x00H\x00O\x00S\x00T\x001\x002\x003/p" "D:\cruft\Hxma.txt"

But I'm not sure how to return 100 characters before and 100 characters after the match like in the grep above, and when I run it against the 12MB database file it doesn't return anything (taking off the silent option so it prints the pattern space shows it getting ~10KB in then apparently giving up)

Does anyone know how I can proceed with solving this? I should add that the actual format of the data returned isn't too important so long as no data is lost (apart from the nulls, I don't need them at all).

If it helps, the actual format of the text data I want to match seems to be 2-byte unicode, although some of the data I need to return along with the match (ie. the first 4 bytes which are a little-endian IP address) is not in unicode.

0

You are telling grep what to look for. The concept that you don't know what you just told grep to look for is strange and I do not follow. The purpose of grep is to look for a pattern which you give it; not to find a string which you already know the location of. I think you need to more clearly define what piece of this puzzle you are missing.

Also, most databases have command line tools. Even though MS doesn't make good tools for it's databases, there is a package called MDB tools which will allow you to actually use SQL to search the DB. This is surely a better way to accomplish what you are trying to do here.

Lastly, if you really did want to search through a binary for strings using grep I would HIGHLY suggest using the strings tool in the binutils package. This will filter out non printable character before you search the data. It can also print out handy offsets (which is what you typically want in this situation). Here is a sample of how that would look (this will search the grep binary for any strings which contain the word 'deprecated'):

strings -a -t x /bin/grep|grep deprecated

As you can see, this is going to be much less error prone. I haven't checked the speed of this but I'm sure that'd be highly dependent of the actual binary contents anyway.

Update

Just realized that you are working in Windows. Systernals makes a version of string.exe which is fairly similar to the Linux strings program.

strings.exe -a -o C:\GnuWin\bin\grep.exe|grep deprecated

Also, you might try Jackcess for a command line MDB tool which'll work with any OS with JVM support.

  • Just to clarify, I don't know the location of the string before I start, but once I have the string location I know where other relevant data is in relation to that string. So I'm matching a hostname in the file, then deriving an IP that occurs several bytes before the hostname and a domain name which occurs directly after the hostname. That said, I will chase up the MDB tools or binutils method for now - if binutils can give me an index I can hopefully pull non-printable characters around that index (just in case there was an IP such as 10.225.157.0 which would start with a null: 009DE10A) – Bruno Sep 26 '14 at 4:58
  • In line with your update, The above should read "I'll chase up the Jackcess or strings method, hopefully string can pull around non-printable in case of octet = 0 in IP" etc. – Bruno Sep 26 '14 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.