Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I want to search for a specific string across multiple lines in a file and get the line in which the match was found.

However, my problem is, that the file contains one very long string, and not, i.e. words, and I want to search for a sub-sequence of this long string. Hence, I can not use pcregrep and just search for word1\nword2. Because I actually want to get the line number in which the match was found, I can not just remove all newline characters...

This is an example how my file looks like, I just capitalized the matching string so you could find it:

String to search:

gcbcdbfceebcfhfchaaccdgfcegffgedffaeaedcbaedhacebeeebcechbcbfeeccbdhcbfg

File to search in:

abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde
abcdeabcde***GCBCDBFCEEBCFHFCHAACCDGFCEGFFGEDFFAEAEDC
BAEDHACEBEEEBCECHBCBFEECCBDHCBFG***ggfbhbgcedabceedfa
fbaaechaabdbffbebecebaacfcfcdcggfchddcefbcbdegbbba

Does someone of you have an easy solution for this?

If there is no tool at hand to do it, I would just write a short python script to do it, but I guess any bash tool would be more efficient than that ...

EDIT:

Thank you very much for your answers, they work very well, if the position of the newline character is known.

However, I am sorry for being imprecise in my question. My problem is, that I do not know if there is a newline, or even more than one newline, within the string in the file and moreover, I do not know where it is. I corrected my search string by deleting the newline I had unwittingly inserted.

Is there some way to allow for a newline in any position of the string?

share|improve this question
up vote 1 down vote accepted

I'm adding a new answer now that I understand the problem better. I'm only posting this as a working example but I don't claim that it is a good one. :)

Also, I understand that the question seemed to want to NOT use Python because of some fears of inefficiencies. So I understand that this approach doesn't fulfill the entire request. :(

#!/usr/bin/env python
import sys

def findall_iter(S, pat):
  index = -1
  while True:
    try:
      index = S.index(pat, index+1)
      yield index
    except ValueError:
      raise StopIteration

def findall(S, pat):
  return list(findall_iter(S, pat))

# read in arguments
S = open(sys.argv[2]).read()
pattern = sys.argv[1]

# get indices of all newlines
newline_indices = findall(S, '\n')

# get psudo-indices of all pattern matches
pat_indices = findall(S.replace('\n', ''), pattern)

# iterate through each pattern match psudo-index and
# correlate it back to a real line number from the file
line_numbers = []
for pi in pat_indices:
  for i, ni in enumerate(newline_indices):
    if ni > pi+i:
      line = i + 1
      if line not in line_numbers:
        line_numbers.append(i+1)
      break

print '\n'.join(map(str, line_numbers))

Pros:

  • If the file is not too large (<1GB) all operations are performed in memory.
  • Uses the str.index method to find substrings instead of (slower) regular expression matching
  • Clearer than using regular expressions

Cons:

  • Won't work well with large files.
  • Creates two temporary strings to do the job.
  • The last for-loop is hard to understand.
  • Is Python (which I personally do not think is a con).
share|improve this answer
    
Hi Dave, thanks for your effort. This python script actually does exactly what I want and it is really quick! I tested it on a 400 MB text file and it took just a couple of seconds. The python script I wrote was more than 10 times slower. (I used a couple of more loops, probably that's the reason...) Thanks again to everyone who helped! – Tabea Apr 2 '13 at 9:19
    
Glad I could help. You could add larger file support into it by doing some buffering tricks. But that would take more time and make the code less readable. – Dave Apr 3 '13 at 19:02

I would do this with a sed script. Put this in a file, then use sed -nf to run it.

:restart
/gcbcdbfceebcfhfchaaccdgfcegffgedffaeaedc$/{
    #   Found the first part, now discard it
    s/^.*$//
    #   Read a new line into the buffer
    N
    #   Discard the new line inserted by the N operation
    s/^\n//
    #   If next line isn't a match, start over
    /^baedhacebeeebcechbcbfeeccbdhcbfg/!b restart
    #   If it is a match, print the line number
    =
    }

Here's what it looks like to run it under bash. Notice that it prints the line number of the second line matched.

bash-4.1$ cat sample.txt
abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde
abcdeabcde***gcbcdbfceebcfhfchaaccdgfcegffgedffaeaedc
baedhacebeeebcechbcbfeeccbdhcbfg***ggfbhbgcedabceedfa
fbaaechaabdbffbebecebaacfcfcdcggfchddcefbcbdegbbba
bash-4.1$
bash-4.1$ cat findmatch.sed
:restart
/gcbcdbfceebcfhfchaaccdgfcegffgedffaeaedc$/{
   #  Found the first part, now discard it
   s/^.*$//
   #  Read a new line into the buffer
   N
   #  Discard the new line inserted by the N operation
   s/^\n//
   #  If next line isn't a match, start over
   /^baedhacebeeebcechbcbfeeccbdhcbfg/!b restart
   #  If it is a match, print the line number
   =
   }
bash-4.1$
bash-4.1$ sed -nf findmatch.sed sample.txt
3
bash-4.1$
share|improve this answer

I'm a little confused as to what constraints you are operating under. However, if you need the line number, both grep and pcregrep can give it to you ala the -n flag.

$ pcregrep -nM "gcbcdbfceebcfhfchaaccdgfcegffgedffaeaedc\nbaedhacebeeebcechbcbfeeccbdhcbfg" | cut -d: -f1
2
baedhacebeeebcechbcbfeeccbdhcbfg***ggfbhbgcedabceedfa

pcregrep shows only the number of the first line matched, apparently so you'll have to skip every other line of the output with sed ( pipe the above to sed -n 'p;N') if you want only the line numbers as output.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .