Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have lots of files containing data records, stored as space-delimited ASCII files. Each record is a row containing numeric data, with some columns integer and others floating-point eg.

1 1  5711  4 22280.365035   75.917899   55.485326    4.0260    3.9460    1.7921   11.2400    0.0000    2.6735   54.7331   52.7375

I want to parse this data based on simple criteria (column 2 == 1, column 6 >= 53.275, etc.) and dump the matching records to another file.

Each file is ~1GB in size, corresponding to ~9M records. Currently I have some MATLAB code that runs through it line-by-line, but this takes a long time (~2hrs per file). The only reason I'm using MATLAB is simply that it's what I'll be processing the data in later.

How can I parse/process this more efficiently? Is it worth using a "proper" language for this, or am I unlikely to see a significant speed increase?

share|improve this question

1 Answer 1

A simple Python script is likely to be faster than anything you can do in bash. For example:

#!/usr/bin/python

with open("data") as data:
    with open("filtered", "w") as filtered:
        for row in data:
            values = row.split()
            if (int(values[1]) == 1) and (float(values[5]) >= 53.275):  # [1]
                filtered.write(row)

[1]: Indexing is zero-based in python, so values[1] and values[5] are the second and sixth columns respectively.

It's impossible to test properly without knowing exactly what your data looks like and how much of it matches your filter, but for a couple of quickly knocked-together sample files, I get these results:

data_1   1000000 rows       35 matching rows   1.5 seconds
data_2   1000000 rows   565722 matching rows   3.1 seconds
share|improve this answer
    
I'm not currently working in Bash, the code is written in MATLAB (I've edited my OP to make that clearer, sorry!). Your code snippet is basically what I'm doing, but from your results it's evidently much faster! Not surprising I guess given how much of a behemoth MATLAB is... –  user265560 Oct 21 '13 at 14:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.