Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

So I have this uge csv file with more than 100 million records and I want to work on. I don't want to just edit it.

What I need is to use some kind of excel / basic database transformations (group by, extract some caracters, add a new column based on combined info, etc.).

What is the best way to do this ? Import into a database ? (such as Oracle, quite sure Access cannot manage such big file) Are other tools usable ? At work I used to have a software called Amadea (http://www.isoft.fr/html/prod_amadea_en.htm) that did some data transformations like this. Don't know of any other (free) tools that does this the same way.

Thanks!

share|improve this question
    
You could have a look at TALEND - powerful opensource ETL.talend.com –  FreudianSlip Nov 8 '13 at 11:56
    
have you considered splitting the file into multiple files? Even if you were to import into a full database solution like MSSQL or MySQL you likely will need to split the data into multiple files. Even a full database solution is going to have trouble trying to parse 10GB of data. superuser.com/questions/98583/break-up-a-large-csv-file?rq=1 –  Ramhound Nov 8 '13 at 12:42
    
Splitting is not a good idea as I will know how to split only after calculations –  Choumarin Nov 8 '13 at 13:58
    
I'm going for Talend, seems to do what I'm looking for. However, learning curve seems a bit steep... –  Choumarin Nov 8 '13 at 19:27
add comment

closed as off-topic by techie007, Mokubai, Tog, Simon Sheehan, Moses Nov 9 '13 at 16:43

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they tend to become obsolete quickly. Instead, describe your situation and the specific problem you're trying to solve. Here are a few suggestions on how to properly ask this type of question." – Mokubai, Tog, Simon Sheehan, Moses
If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer

Definitely use a database. There are plenty of free ones, such as MySQL, or if you just want one computer to work on it at a time then Sqlite is extremely good.

Do check that your import has worked properly first, especially if there is non-ascii or large numbers in the input data set.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.