25

I will like to use "find" and locate" to search for source files in my project, but they take a long time to run. Are there faster alternatives to these programs I don't know about, or ways to speed up the performance of these programs?

  • 2
    locate should already be plenty fast, considering that it uses a pre-built index (the primary caveat being that it needs to be kept up to date), while find has to read the directory listings. – afrazier Sep 29 '11 at 13:22
  • 2
    Which locate are you using? mlocate is faster than slocate by a long way (note that whichever package you have installed, the command is still locate, so check your package manager) – Paul Sep 29 '11 at 13:35
  • @benhsu, when I run find /usr/src -name fprintf.c on my OpenBSD desktop machine, it returns the locations of those source files in less than 10 seconds. locate fprintf.c | grep '^/usr/src.*/fprintf.c$' comes back in under a second. What is your definition of "long time to run" and how do you use find and locate? – Kusalananda Sep 29 '11 at 14:15
  • @Paul, I am using mlocate. – benhsu Sep 29 '11 at 17:15
  • @KAK, I would like to use the output of find/locate to open a file in emacs. the use case I have in mind is, I wish to edit the file, I type the file name (or some regexp matching the file name) into emacs, and emacs will use find/locate to bring up a list of files matching it, so I will like response time fast enough to be interactive (under 1 second). I have about 3 million files in $HOME, one thing I can do is make my find command prune out some of the files. – benhsu Sep 29 '11 at 17:22
18

Searching for source files in a project

Use a simpler command

Generally, source for a project is likely to be in one place, perhaps in a few subdirectories nested no more than two or three deep, so you can use a (possibly) faster command such as

(cd /path/to/project; ls *.c */*.c */*/*.c)

Make use of project metadata

In a C project you'd typically have a Makefile. In other projects you may have something similar. These can be a fast way to extract a list of files (and their locations) write a script that makes use of this information to locate files. I have a "sources" script so that I can write commands like grep variable $(sources programname).

Speeding up find

Search fewer places, instead of find / … use find /path/to/project … where possible. Simplify the selection criteria as much as possible. Use pipelines to defer some selection criteria if that is more efficient.

Also, you can limit the depth of search. For me, this improves the speed of 'find' a lot. You can use -maxdepth switch. For example '-maxdepth 5'

Speeding up locate

Ensure it is indexing the locations you are interested in. Read the man page and make use of whatever options are appropriate to your task.

   -U <dir>
          Create slocate database starting at path <dir>.

   -d <path>
          --database=<path> Specifies the path of databases to search  in.


   -l <level>
          Security  level.   0  turns  security checks off. This will make
          searchs faster.   1  turns  security  checks  on.  This  is  the
          default.

Remove the need for searching

Maybe you are searching because you have forgotten where something is or were not told. In the former case, write notes (documentation), in the latter, ask? Conventions, standards and consistency can help a lot.

| improve this answer | |
11

I used the "speeding up locate" part of RedGrittyBrick's answer. I created a smaller db:

updatedb -o /home/benhsu/ben.db -U /home/benhsu/ -e "uninteresting/directory1 uninteresting/directory2"

then pointed locate at it: locate -d /home/benhsu/ben.db

| improve this answer | |
6

A tactic that I use is to apply the -maxdepth option with find:

find -maxdepth 1 -iname "*target*"

Repeat with increasing depths until you find what you are looking for, or you get tired of looking. The first few iterations are likely to return instantaneously.

This ensures that you don't waste up-front time looking through the depths of massive sub-trees when what you are looking for is more likely to be near the base of the hierarchy.


Here's an example script to automate this process (Ctrl-C when you see what you want):

(
TARGET="*target*"
for i in $(seq 1 9) ; do
   echo "=== search depth: $i"
   find -mindepth $i -maxdepth $i -iname "$TARGET"
done
echo "=== search depth: 10+"
find -mindepth 10 -iname $TARGET
)

Note that the inherent redundancy involved (each pass will have to traverse the folders processed in previous passes) will largely be optimized away through disk caching.

Why doesn't find have this search order as a built-in feature? Maybe because it would be complicated/impossible to implement if you assumed that the redundant traversal was unacceptable. The existence of the -depth option hints at the possibility, but alas...

| improve this answer | |
  • 1
    ...thus performing a "breadth-first" search. – nobar Dec 1 '15 at 0:47
3

Another easy solution is to use newer extended shell globbing. To enable:

  • bash: shopt -s globstar
  • ksh: set -o globstar
  • zsh: already enabled

Then, you can run commands like this in the top-level source directory:

# grep through all c files
grep printf **/*.c

# grep through all files
grep printf ** 2>/dev/null

This has the advantage that it searches recursively through all subdirectories and is very fast.

| improve this answer | |
3

The Silver Searcher

You might found it useful for searching very fast the content of a huge number of source code files. Just type ag <keyword>. Here some of the output of my apt show silversearcher-ag:

I usually use it with:

-G --file-search-regex PATTERN Only search files whose names match PATTERN.

ag -G "css$" important

screenshot

| improve this answer | |
  • 2
    the ripgrep's algorythm is allegedly faster than silversearch, and it also honors .gitignore files and skips .git, .svn, .hg.. folders. – ccpizza Mar 24 '18 at 21:07
  • @ccpizza So? The Silver Searcher also honors .gitignore and ignores hidden and binary files by default. Also have more contributors, more stars on Github (14700 vs 8300) and is already on repos of mayor distros. Please provide an updated reliable third-party source comparison. Nonetheless, ripgrep looks a great piece of software. – Pablo A Mar 24 '18 at 23:35
  • good to know! i'm not affiliated with author(s) of ripgrep in any way, it just fit my requirement so I stopped searching for other options. – ccpizza Mar 24 '18 at 23:38
  • The silver searcher respects .gitignore too. That said, rg is absolutely amazing. First off, it has unicode support. In my experience rg consistently at least twice as fast as ag (YMMV), I guess it's due to Rust's regex parser, that obviously wasn't ready yet back in the years ag was new. rg can give deterministic output (but doesn't by default), it can blacklist file types where ag can only whitelist, it can ignore files based on size (bye bye logs). I still use ag in case I need multiline matching, which rg can't do. – The Pellmeister Apr 4 '19 at 0:43
3

For a find replacement, check out fd. It has a simpler / more intuitive interface than the original find command, and is quite a bit faster.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.