I have two files. One huge one (200.000+ lines) called 'db' and one big one (15.000+ lines) called 'indices'.

What is the quickest way of filtering out the lines in 'db' containing any index (anywhere on the line) from 'indices'

The solution I could think of is

for index in $(cat indices); do
  grep $index db >> selection
done;

but this is taking a long time.

Is there a faster approach in bash, linux?

link|improve this question

feedback

1 Answer

up vote 5 down vote accepted
fgrep -f indices db 

should be faster.

link|improve this answer
Indeed, it is way, way faster. Thanks! – Peter Smit Feb 9 '10 at 9:41
"fgrep" usually isn't the fastest grep variant. You may find egrep is faster still. – njd Feb 10 '10 at 17:42
Are you sure about it? – Zsolt Botykai May 27 '10 at 6:47
feedback

Your Answer

 
or
required, but never shown

Not the answer you're looking for? Browse other questions tagged or ask your own question.