10

I find myself grepping the same codebase over and over. While it works great, each command takes about 10 seconds, so I am thinking about ways to make it faster.

So can grep use some sort of index? I understand an index probably won't help for complicated regexps, but I use mostly very simple patters. Does an indexer exist for this case?

EDIT: I know about ctags and the like, but I would like to do full-text search.

  • Are you using recursive oprtion for grep or some find/xargs like way? – Michał Šrajer Aug 9 '11 at 11:12
  • @Michał : yes, -R – Peltier Aug 9 '11 at 12:21
4

what about cscope, does this match your shoes?

Allows searching code for:

  • all references to a symbol
  • global definitions
  • functions called by a function
  • functions calling a function
  • text string
  • regular expression pattern
  • a file
  • files including a file
| improve this answer | |
  • That could be what I'm looking for, I'll take a look. Thanks! – Peltier Aug 9 '11 at 15:22
  • It looks like it just works well for C, maybe C++ and Java – neves Aug 10 '17 at 15:01
4

Full-text indexing

There are tools such as recoll, swish-e and sphinx but you'd have to check if they can support the sort of search criteria you need.

Recoll

Recoll is a personal full text search tool for Unix/Linux.

Swish-e

Swish-e is a fast, flexible, and free open source system for indexing collections of Web pages or other files.

Sphinx

Sphinx lets you either batch index and search data stored in an SQL database, NoSQL storage, or just files quickly and easily

grep

I'm surprised grep is as slow as you describe, can you reduce the number of files being searched? For example when I only need to search the source files for one executable (out of many in a project) I feed grep the names from a command that lists the source files for that program:

grep expression `sources myprogram`

sources is a program specific to my development environment but you may have (or be able to construct) something equivalent.

I'm assuming you've tried obvious techniques such as

find /foo/myproject -name "*.c" -exec fgrep -l searchtext

I've read a suggestion that the -P option of current grep can speed up searches significantly.

| improve this answer | |
  • 1
    AFAIK locate is only for filenames. recoll would work, but I would prefer a command-line tool. The code base is pretty big, and since I'm looking for a string, I don't know where it is, so it's hard to limit the number of files to be searched :) – Peltier Aug 9 '11 at 13:41
  • I think swish-e is command-line. I haven't tried any (grep is fast enough on my projects) – RedGrittyBrick Aug 9 '11 at 14:32
3

grep, no. But there are several programs which use indexes and aimed at code base. ctags (there is a version provided with vim), etags (aimed for use with emacs), global (more independent of the editor) are the one I'm thinking about now but there are probably other.

| improve this answer | |
  • I use ctags, but isn't that limited to searching function names? I want to do full-text search. – Peltier Aug 9 '11 at 9:46
  • I'm pretty sure that ctag can also search for class definition and ISTR that it also find some use. I'm certain that global does both. But it is true that those tools don't do a full-text search and are using language knowledge to limit their scope. – AProgrammer Aug 9 '11 at 9:50
3

You could copy your codebase on a RAM disk.

| improve this answer | |
2

if you want to use a fulltext search engine .. use one:

| improve this answer | |
  • That's always an option, but I was wondering if a more lightweight, quick and dirty grep speedup option would exist. – Peltier Aug 9 '11 at 14:29
  • 'more lightweight' but 'want to have my stuff fully indexed' are a bit of 2 extremes :) ctags is the best match for what you want, if you just want to go quick an dirty. with everything else you end up using a real fulltext-search-engine. eg, 'recoll' mentioned in @RedGrittyBrick answer is using xapian as the backend. – akira Aug 9 '11 at 14:52
  • 1
    They're not necessarily incompatible. Imagine if ctags had a --full-text option, for instance, and grep a --tag-file option. Of course the fact that it could exist doesn't mean that it does :) – Peltier Aug 9 '11 at 15:05
1

I did not try it yet.
but it seems promising
from the wikipedia page :
https://en.wikipedia.org/wiki/GLIMPSE

GLIMPSE stands for GLobal IMPlicit SEarch. While many text indexing schemes create quite large indexes (usually around 50% of the size of the original text), a GLIMPSE-created index is only 2-4% of the size of the original text

| improve this answer | |
-1

No, I don't think so. But there may be a simple solution: Try ack. I think if you give it a chance, you'll find it significantly faster than grep, requires shorter search strings to get better search results, and has many desirable features, while using much the same command switches. One thing that makes it faster (although not indexed) is that it ignores a lot more of the stuff that you don't want to search. It's written in Perl and uses Perl's regular expressions (and therefore also has Mac and Windows ports, too).

http://betterthangrep.com/

| improve this answer | |
  • Ack is pretty cool. But I really doubt it's any faster than grep, since it is based on the same mechanisms. – Peltier Dec 7 '12 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.