0

So I have this uge csv file with more than 100 million records and I want to work on. I don't want to just edit it.

What I need is to use some kind of excel / basic database transformations (group by, extract some caracters, add a new column based on combined info, etc.).

What is the best way to do this ? Import into a database ? (such as Oracle, quite sure Access cannot manage such big file) Are other tools usable ? At work I used to have a software called Amadea (http://www.isoft.fr/html/prod_amadea_en.htm) that did some data transformations like this. Don't know of any other (free) tools that does this the same way.

Thanks!

Improve this question
4
  • You could have a look at TALEND - powerful opensource ETL.talend.com
    – FreudianSlip
    Nov 8, 2013 at 11:56
  • have you considered splitting the file into multiple files? Even if you were to import into a full database solution like MSSQL or MySQL you likely will need to split the data into multiple files. Even a full database solution is going to have trouble trying to parse 10GB of data. superuser.com/questions/98583/break-up-a-large-csv-file?rq=1
    – Ramhound
    Nov 8, 2013 at 12:42
  • Splitting is not a good idea as I will know how to split only after calculations
    – Choumarin
    Nov 8, 2013 at 13:58
  • I'm going for Talend, seems to do what I'm looking for. However, learning curve seems a bit steep...
    – Choumarin
    Nov 8, 2013 at 19:27

1 Answer 1

Reset to default
2

Definitely use a database. There are plenty of free ones, such as MySQL, or if you just want one computer to work on it at a time then Sqlite is extremely good.

Do check that your import has worked properly first, especially if there is non-ascii or large numbers in the input data set.

Improve this answer

Not the answer you're looking for? Browse other questions tagged .