Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm working on a project involving Twitter data. I have several hundreds thousand tweets downloaded and stored in files. The data was returned in json format, and the stream consumer I was using converted them to python dictionaries, so I it's all stored in text files, one tweet per line, as python dictionaries.

There is a lot of extraneous information, so I have a python script that reads each line in as a dict and extracts some useful information. What would be the best way to store this data now that's it's extracted? I was printing it back out to csv files, but I've been having some issues with that and have come across some people who seem to feel that that's not the best way to store it.

What would be the most effective way of storing this data? I will need to access it to find patterns, match similar items, etc. I was thinking of using a database - is that the best option? Are there others that are better?

share|improve this question
up vote 2 down vote accepted

If its just key pair stores, apparently nosql style databases work well - twitter does use these, and they might be a great fit if you need to handle a lot of data with very little structure. You could probably use a traditional rdbms, or maybe an embedded sqlite db if there are more than a simple key pair store and had structured data with relations.

It might also help to understand the weakness of a flat file store (no transaction logging or structure) , nosql (no ACID) and a traditional db(bulky, less scalable but well understood and often reasonably fast). With a small non updated set of data, any of them should work

share|improve this answer
Thanks - I think I'm going to try using Hadoop and HDFS. Thanks for your help! – bsg Mar 10 '13 at 17:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .