2

I have been using grep to search databases. I've been using the following command when searching multiple folders with multiple text files.

grep 'text_search' */*

is there a better command that will speed up the process of searching through large amounts of data? I am open to using other tools if needed.

1 Answer 1

4

I'm a fan of ripgrep

If you prefer to stick with grep, grep -F matches strings not patterns (which may or may not be faster; I'm not sure if modern greps simplify a simple pattern to a string search).

Running grep in parallel is also an option. I use GNU parallel for this.

find . -type f | parallel --jobs {#jobs} -n 500 -k -m grep -H {search-pattern} {}

(jobs and search-pattern aren't surrounded in braces; they indicate a variable you need to enter)

2
  • Theres also ack and ag, off the top of my head, and fzf depending on what youre doing. Jul 26, 2019 at 1:08
  • I'd avoid parallelism if you have a rotational hard disk, however fast it may be. Or at least benchmark it with a cold cache (echo 3 > /proc/sys/vm/drop_caches) before settling on it as a real benefit. The only time I have benefited from any parallelism is when the data can fit in RAM and I have to make multiple greps, one after another. In that case, the first one is slow (it reads from disk), but subsequent ones are very fast.
    – user1003916
    Jul 29, 2019 at 1:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .