up vote 2 down vote favorite
1
share [g+] share [fb]

I have a few gigabytes of source code.

using recursive grep for a term can take a while.

I am using ext3.

Is there a faster way? Would using find be faster and if so why? Would using a filesystem like XFS give noticeably better results?

link|improve this question
1  
with exec it can use grep which would be faster than just using grep – Joshxtothe4 Dec 16 '09 at 20:46
and why would that be faster? – akira Dec 16 '09 at 22:57
feedback

migrated from stackoverflow.com Dec 16 '09 at 22:43

This question came from our site for professional and enthusiast programmers.

7 Answers

up vote 5 down vote

Have you tried ack? It works pretty well here, on a 1mm+ sized codebase.

link|improve this answer
ack is easier and faster than using find | grep, and I often use it, but it doesn't index the results anywhere for later use. – njd Feb 9 '10 at 12:49
feedback
up vote 3 down vote

You can get better performance with agrep, which uses a novel bitmasking algorithm for search.

If you're looking for symbols, ctags or etags might work well enough to build an index for search.

link|improve this answer
+1 ctags ...... – Ewan Dec 16 '09 at 20:51
Ctags indexes the results, so you can search quickly from your editor. Darren Hiebert's Exuberant Ctags (ctags.sourceforge.net) has improved options for recursive searching. – njd Feb 9 '10 at 12:54
feedback
up vote 2 down vote

The only way you'll get a significant improvement over grep is to use an indexed search system like Strigi. The filesystem makes very little difference unless you have a huge number of very small files.

link|improve this answer
feedback
up vote 1 down vote

This should likely be on superuser.

Grepping is not the ideal solution to your problem since it performs a linear search.

Index your files for search using a desktop indexing solution such as Beagle or Google Desktop.

link|improve this answer
feedback
up vote 1 down vote

I don't think the FS is going to make a big difference; chances are it's compute bound. You could check this using top to see if your CPUs are smoking.

You could also post your regexp here and let the smart people of SO have a crack at optimizing it. There are a variety of techniques for avoiding backtracking, etc.

link|improve this answer
fgrep is faster because it doesn't use regular expressions, it only searches for fixed strings. It's just an alias for grep -F. – Tim Sylvester Dec 16 '09 at 20:44
Right you are, thanks. I removed that part of my suggestion. – Carl Smotricz Dec 16 '09 at 20:50
feedback
up vote 1 down vote

Here is what I understand -

  • You are searching source code for a term
  • You'd like to see which source files use that term
  • You probably have thousands of files (adding up to GBs)
  • Do you want to know all the occurrences of 'term' within each file or a yes/no indication of whether its been used in a file or not? (the -l flag does this).

You can use the policy of divide-and-rule. Partition your set into multiple file-sets, run multiple greps parallely.

Not sure if your need is a one-off thing or something repetitive in nature.

link|improve this answer
feedback
up vote 1 down vote

If you only need to grep a subset of files then use find first. For example to only grep .h header files:

find path/to/source -name *.h -print0 | xargs -0 grep pattern

This will be faster since you're only accessing filenames most of the time, rather than file contents, which means many fewer disc accesses.

link|improve this answer
1  
Better: find path/to/source -name *.h -exec grep pattern {} \; – Ewan Dec 16 '09 at 20:50
2  
Even better: find path/to/source -name *.h -exec grep pattern {} \+ (less grep invocations) – filiprem Dec 16 '09 at 22:39
feedback

Your Answer

 
or
required, but never shown
discard