I have a fortran program that simulates a lot of ascii text file output. These are quite large to load the entirety of any single file into common data analysis programs (Matlab, R, SciPy).
I think a common thing to do in the scientific computing community is to extract a subset of this data using Fortran, C, Perl, or Python and storing these in a separate flat text file or netCDF (and more recently, HDF5) to be used by another program. However, programs like R are very well suited for pulling data out of an SQL database, in which case output -> SQLDB -> R seems to be a good solution (and probably SciPy also).
Does anyone have an opinion on this? First of all, creating a separate, intermediate file is unideal (seems like a redundant use of disk space) but unavoidable? Also - I haven't seen SQL databases used much in the scientific computing community, but SQLite seems just as amenable for transferring data between programs as HDF5, or there are downsides? Thanks for your comments.