Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm looking for an app that I could feed my log file and then explain it how the columns should be interpreted (width based, separator based, string delimiter, etc; like you do on excel) and then save those interpretation rules as a template.

And then be able to filter lines based on querys, regexs, rules, etc..

I'm looking for a general app... not an app specific to X format of Y daemon.

share|improve this question
The answer to this will likely be OS dependent. Could you tell us which OS you want this for? – EBGreen Jan 8 '13 at 20:55

For an application this general, I'm always inclined to write a Python or (in the past) Perl script to do the job. The code to interpret each line of the logfile will generally take no more than a line or two using split() in Python or a regex in either language. I generally put each line into a list, each item in the list being a dictionary, with the dictionary keys being the column names. I generate the column names automatically from the first line of the logfile, if they're available there, or I set them specfically in the script, if they're not.

I don't think a set of interpretation rules that is sufficiently general to deal with a wide set of logfiles could be written or saved in a much more concise format than I'd be able to write it in Python -- even if it could, I'd be reluctant to learn yet another language to write the interpretation rules if I could do the job with regexes etc. in Python or Perl.

Having Python filter the list of dictionaries and write the filtered results back out again is also straightforward. I'd typically write the logfile interpretation code as a function then use that function from another script that does the filtering, or call the interpretation function to read in the data and then do ad hoc analysis from Python's interactive command line, which I can do very concisely by using list comprehensions to express the queries or rules.

Python is available for all major OSes.

In addition, I use R when I want to analyse a file graphically, but I almost always have to use Python to get the file into a format where I can use R on it anyway so, if I don't want a graphical output, I find I'm better off doing the work entirely in Python. There are graphical output options for Python too (e.g. matplotlib) but I haven't used them so I can't speak to their ease of use.

O'Reilly's "Data Analysis with Open Source Tools" by Philipp Janert ( is an excellent source on using Python and R for this kind of work.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .