Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Every week or so I need to load a file that has millions of lines and start running greps on it. Some greps are positive, some are negative ("-v" flag). Some are grep and some are egrep.

I do it manually today - run a grep, save the results to a file, then run another grep on this file and save to file2, then file3, etc.

Is there some tool that can make my life easier where I can easily manipulate a list of greps, egreps, that the tool will calculate and show the results live? The tool needs to cache the results of previous greps to avoid re-running them all each time.

share|improve this question
    
Um … put the commands that you currently run manually into a script? –  Scott Nov 15 '12 at 23:33
    
No, it's highly dynamic. I need to be able to add greps, turn some on/off, etc. –  Yon Nov 18 '12 at 8:41

1 Answer 1

I'm not quite sure what you mean by "show the results live" (but see the end of my answer for an idea) but generally I'd use make for this sort of task. For example, given a file srcfile.txt on which I'm going to run a number of greps, I'd put the greps in a makefile something like this:

all: e.out b.out zy.out

single: e.out b.out

e.out: srcfile.txt
    grep e srcfile.txt >e.out

b.out: srcfile.txt
    grep b srcfile.txt >b.out

zy.out: srcfile.txt
    grep zy srcfile.txt >zy.out

... I can then run all the greps that are required to bring the output files up-to-date with

> make

... I can run all the greps that search for single letters with

> make single

... etc. Over a few weeks in your situation, I think I'd quickly accumulate a set of make rules that covered all (or almost all) of the cases in which I was interested.

You might be able to save some more time and effort by putting the commands to load your million+ line file in the makefile as well.

If choosing one or more targets using the make command doesn't provide enough flexibility, you could use conditionals in the make file, based on variables that you set when running make, e.g. see the GNU makefile tutorial on conditionals and this article on passing variables from the make command line.

To show the results of the grepping as it happens, I would be inclined to use tail or less as recommended here. A difficulty is that either method only shows one file at a time. I'd need to know more about your need for "live" viewing to know if that might be satisfactory or if another solution might be required.

share|improve this answer
    
Thanks for the help, but not what I'm looking for. –  Yon Jan 12 '13 at 19:05
    
In that case, could you expand a bit on just what you're looking for? Thanks. –  Simon Jan 12 '13 at 22:32
    
Something that allows me to select a single file, add egreps as separate patterns, turn some on and off and see how it changes the result. It needs to be easy to use, not a script that I edit again and again. –  Yon Jan 13 '13 at 3:26

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.