I have lots of files containing data records, stored as space-delimited ASCII files. Each record is a row containing numeric data, with some columns integer and others floating-point eg.
1 1 5711 4 22280.365035 75.917899 55.485326 4.0260 3.9460 1.7921 11.2400 0.0000 2.6735 54.7331 52.7375
I want to parse this data based on simple criteria (column 2 == 1, column 6 >= 53.275, etc.) and dump the matching records to another file.
Each file is ~1GB in size, corresponding to ~9M records. Currently I have some MATLAB code that runs through it line-by-line, but this takes a long time (~2hrs per file). The only reason I'm using MATLAB is simply that it's what I'll be processing the data in later.
How can I parse/process this more efficiently? Is it worth using a "proper" language for this, or am I unlikely to see a significant speed increase?