I have a few gigabytes of source code.
using recursive grep for a term can take a while.
I am using ext3.
Is there a faster way? Would using find be faster and if so why? Would using a filesystem like XFS give noticeably better results?
I have a few gigabytes of source code.
using recursive grep for a term can take a while.
I am using ext3.
Is there a faster way? Would using find be faster and if so why? Would using a filesystem like XFS give noticeably better results?
You can get better performance with agrep, which uses a novel bitmasking algorithm for search.
If you're looking for symbols, ctags or etags might work well enough to build an index for search.
The only way you'll get a significant improvement over grep is to use an indexed search system like Strigi. The filesystem makes very little difference unless you have a huge number of very small files.
This should likely be on superuser.
Grepping is not the ideal solution to your problem since it performs a linear search.
Index your files for search using a desktop indexing solution such as Beagle or Google Desktop.
I don't think the FS is going to make a big difference; chances are it's compute bound. You could check this using top
to see if your CPUs are smoking.
You could also post your regexp here and let the smart people of SO have a crack at optimizing it. There are a variety of techniques for avoiding backtracking, etc.
fgrep
is faster because it doesn't use regular expressions, it only searches for fixed strings. It's just an alias for grep -F
.
Dec 16, 2009 at 20:44
Here is what I understand -
You can use the policy of divide-and-rule. Partition your set into multiple file-sets, run multiple greps parallely.
Not sure if your need is a one-off thing or something repetitive in nature.
If you only need to grep a subset of files then use find first. For example to only grep .h header files:
find path/to/source -name *.h -print0 | xargs -0 grep pattern
This will be faster since you're only accessing filenames most of the time, rather than file contents, which means many fewer disc accesses.
find path/to/source -name *.h -exec grep pattern {} \+
(less grep
invocations)