Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

So I have this uge csv file with more than 100 million records and I want to work on. I don't want to just edit it.

What I need is to use some kind of excel / basic database transformations (group by, extract some caracters, add a new column based on combined info, etc.).

What is the best way to do this ? Import into a database ? (such as Oracle, quite sure Access cannot manage such big file) Are other tools usable ? At work I used to have a software called Amadea ( that did some data transformations like this. Don't know of any other (free) tools that does this the same way.


share|improve this question

closed as off-topic by Ƭᴇcʜιᴇ007, Mokubai, Tog, Simon Sheehan, Moses Nov 9 '13 at 16:43

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve. Here are a few suggestions on how to properly ask this type of question." – Mokubai, Tog, Simon Sheehan, Moses
If this question can be reworded to fit the rules in the help center, please edit the question.

You could have a look at TALEND - powerful opensource – FreudianSlip Nov 8 '13 at 11:56
have you considered splitting the file into multiple files? Even if you were to import into a full database solution like MSSQL or MySQL you likely will need to split the data into multiple files. Even a full database solution is going to have trouble trying to parse 10GB of data. – Ramhound Nov 8 '13 at 12:42
Splitting is not a good idea as I will know how to split only after calculations – Choumarin Nov 8 '13 at 13:58
I'm going for Talend, seems to do what I'm looking for. However, learning curve seems a bit steep... – Choumarin Nov 8 '13 at 19:27

Definitely use a database. There are plenty of free ones, such as MySQL, or if you just want one computer to work on it at a time then Sqlite is extremely good.

Do check that your import has worked properly first, especially if there is non-ascii or large numbers in the input data set.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged .