Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to create a GUI for searching through a large number of huge configuration files (approx 60000 files, each one with a size between 20 KByte to 50 MByte). Those files are also updated frequently (~3 times/day).

So far I have found SOLR and Sphinx, but found no way to have them return the list of matching lines including a line number for each matching document.

What we currently do is we convert each text file to XML:

<xml>
   <line number="1">foobar</line>
   <line number="2">barfoo</line>
   ...
</xml>

and store the result in an eXist-db. However, storing documents is way too slow, so we need an alternative.

Any better ideas?

share|improve this question

1 Answer 1

Opinion: If you have large amounts of volatile text data you need fast access to, converting them to XML will make your problems much harder to solve.

Any better ideas?

Leave the files as text and use Lucene?

(I'm assuming that grep doesn't cut it)

share|improve this answer
1  
"Some people, when confronted with a problem, think “I know, I'll use XML” Now they have two problems." –  Paul Nov 11 '11 at 11:17
    
The only way to make Lucene return line numbers is by storing each line in a separate document. This, however, makes updating a document hard (and it is probably impossible to make updates fast). –  knipknap Nov 14 '11 at 8:52
    
@knipknap: I can't work out if that's how this example does it. (Applet with source) –  RedGrittyBrick Nov 14 '11 at 9:20
    
The demo only adds one file, but yes, it stores each line as a separate document using the loop at the comment "pull the data from our list and add it to the index" in SimpleLucene.java. Since they don't do updates, this is not a problem. –  knipknap Nov 14 '11 at 11:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.