I am trying to create a GUI for searching through a large number of huge configuration files (approx 60000 files, each one with a size between 20 KByte to 50 MByte). Those files are also updated frequently (~3 times/day).
What we currently do is we convert each text file to XML:
<xml> <line number="1">foobar</line> <line number="2">barfoo</line> ... </xml>
and store the result in an eXist-db. However, storing documents is way too slow, so we need an alternative.
Any better ideas?