Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a fortran program that simulates a lot of ascii text file output. These are quite large to load the entirety of any single file into common data analysis programs (Matlab, R, SciPy).

I think a common thing to do in the scientific computing community is to extract a subset of this data using Fortran, C, Perl, or Python and storing these in a separate flat text file or netCDF (and more recently, HDF5) to be used by another program. However, programs like R are very well suited for pulling data out of an SQL database, in which case output -> SQLDB -> R seems to be a good solution (and probably SciPy also).

Does anyone have an opinion on this? First of all, creating a separate, intermediate file is unideal (seems like a redundant use of disk space) but unavoidable? Also - I haven't seen SQL databases used much in the scientific computing community, but SQLite seems just as amenable for transferring data between programs as HDF5, or there are downsides? Thanks for your comments.

share|improve this question

closed as not a real question by Nifle, Shiki, studiohack Oct 16 '11 at 1:27

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

up vote 1 down vote accepted

Depends on how you are going to be slicing up your big data. If your data is an N-dimensional table then netCDF is a good idea (especially if its 3d data) if you are taking slices in your three dimensions. Its not so good for queries like get me all the values of Z1 where Z2 is greater than Z3. Then you probably do want a relational db and to make sure you define some indexes properly. SQLite is a nice portable way to do that, and there's interfaces to at least R and Python.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged .