Every week or so I need to load a file that has millions of lines and start running greps on it. Some greps are positive, some are negative ("-v" flag). Some are grep and some are egrep.

I do it manually today - run a grep, save the results to a file, then run another grep on this file and save to file2, then file3, etc.

Is there some tool that can make my life easier where I can easily manipulate a list of greps, egreps, that the tool will calculate and show the results live? The tool needs to cache the results of previous greps to avoid re-running them all each time.

  • Um … put the commands that you currently run manually into a script? Commented Nov 15, 2012 at 23:33
  • No, it's highly dynamic. I need to be able to add greps, turn some on/off, etc.
    – Yon
    Commented Nov 18, 2012 at 8:41

1 Answer 1


I'm not quite sure what you mean by "show the results live" (but see the end of my answer for an idea) but generally I'd use make for this sort of task. For example, given a file srcfile.txt on which I'm going to run a number of greps, I'd put the greps in a makefile something like this:

all: e.out b.out zy.out

single: e.out b.out

e.out: srcfile.txt
    grep e srcfile.txt >e.out

b.out: srcfile.txt
    grep b srcfile.txt >b.out

zy.out: srcfile.txt
    grep zy srcfile.txt >zy.out

... I can then run all the greps that are required to bring the output files up-to-date with

> make

... I can run all the greps that search for single letters with

> make single

... etc. Over a few weeks in your situation, I think I'd quickly accumulate a set of make rules that covered all (or almost all) of the cases in which I was interested.

You might be able to save some more time and effort by putting the commands to load your million+ line file in the makefile as well.

If choosing one or more targets using the make command doesn't provide enough flexibility, you could use conditionals in the make file, based on variables that you set when running make, e.g. see the GNU makefile tutorial on conditionals and this article on passing variables from the make command line.

To show the results of the grepping as it happens, I would be inclined to use tail or less as recommended here. A difficulty is that either method only shows one file at a time. I'd need to know more about your need for "live" viewing to know if that might be satisfactory or if another solution might be required.

  • Thanks for the help, but not what I'm looking for.
    – Yon
    Commented Jan 12, 2013 at 19:05
  • In that case, could you expand a bit on just what you're looking for? Thanks.
    – Simon
    Commented Jan 12, 2013 at 22:32
  • Something that allows me to select a single file, add egreps as separate patterns, turn some on and off and see how it changes the result. It needs to be easy to use, not a script that I edit again and again.
    – Yon
    Commented Jan 13, 2013 at 3:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .