preparing large data sets for analysis [closed]

Question

I have a fortran program that simulates a lot of ascii text file output. These are quite large to load the entirety of any single file into common data analysis programs (Matlab, R, SciPy).

I think a common thing to do in the scientific computing community is to extract a subset of this data using Fortran, C, Perl, or Python and storing these in a separate flat text file or netCDF (and more recently, HDF5) to be used by another program. However, programs like R are very well suited for pulling data out of an SQL database, in which case output -> SQLDB -> R seems to be a good solution (and probably SciPy also).

Does anyone have an opinion on this? First of all, creating a separate, intermediate file is unideal (seems like a redundant use of disk space) but unavoidable? Also - I haven't seen SQL databases used much in the scientific computing community, but SQLite seems just as amenable for transferring data between programs as HDF5, or there are downsides? Thanks for your comments.

Answer 1

Depends on how you are going to be slicing up your big data. If your data is an N-dimensional table then netCDF is a good idea (especially if its 3d data) if you are taking slices in your three dimensions. Its not so good for queries like get me all the values of Z1 where Z2 is greater than Z3. Then you probably do want a relational db and to make sure you define some indexes properly. SQLite is a nice portable way to do that, and there's interfaces to at least R and Python.