3

Is there a way to index a file or a file tree from the console so that searching for substrings can be done faster than plain grepping?

I don't necessarily need support for regular expressions, although case insensitivity would be good to have.

I'm looking for a simple console utility that would work similar to locate/slocate/mlocate but instead of file names would index file contents with specific mime types in a configurable location.

So far the only 'faster-than-grep' solution which I've found is fgrep constrained to ANSI rather than UTF-8 (from https://stackoverflow.com/a/13913220/191246) — while it provides an impressive speedup, it is still too slow for large files. I would like to know if there is some 'cheap' way to create an index and search against it.

I am considering whoosh as an option but that would require extra coding.

I am not interested in system-level indexing apps like spotlight on mac or their linux counterparts, since I am looking for something granular at file or subfolder level.

Follow-up

The accepted answer fits well the core original requirement. On the next level, after extensive searching for a granular desktop search solution with a friendly web UI specifically for searching in tens of thousands of HTML files, have finally settled on omega-xapian — a fast and minimalist indexed search engine which comes with a cgi script that can be run on a webserver (used via caddy with the cgi plugin). Omega is originally meant to be used for website search but can be repurposed for 'desktop search'. The steps for configuring xapian search for macos are listed here (should be easy to adapt to other OS's). Periodically updating the db via a crontab entry that runs omindex. It can also be configured to index binary formats (PDF, DOC, DOCX, etc) if you provide supported command line converters.

1

2 Answers 2

3

Other alternatives which don't require an index, include:

ripgrep https://blog.burntsushi.net/ripgrep/

ag aka the silver searcher: https://geoff.greer.fm/ag/

ack https://beyondgrep.com/

1
  • 1
    a non-regex search in a 9GB file took under 10 seconds (with ripgrep) — amazing result for non-indexed search! grep takes minutes..
    – ccpizza
    Feb 5, 2018 at 17:39
3

Google code search command line utilties (written in Go) fit the described use case.

On debian/ubuntu it can be installed with

sudo apt install codesearch

To compile from source:

If not already present, install first the golang environment, and define the GOPATH variable:

sudo mkdir -p /usr/local/go
sudo chown myusername /usr/local/go

## normally you'd put this in your ~/.bashrc
export GOPATH=/usr/local/go

## on ubuntu/debian:
sudo apt install golang

## on osx
brew install golang

Next, build cindex and csearch:

go get github.com/google/codesearch/cmd/...

Once installed from source you will have cindex and csearch under your $GOPATH/bin — either move them to somewhere under your $PATH, or add $GOPATH/bin to your $PATH and refresh/restart your shell.

Usage

Index a file or folder:
cindex myproject_dir

The index will be created in ~/.csearchindex.

Now you can search the index:

csearch sausage-and-spam

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .