5

I am currently using python with matplotlib to display a 440+ k lines .csv but it takes 11 sec to display only one column. My .csv has always the same format. Is there a way to parse it quicker ? I choose to store all the column into lists then display it.

Here is the code I made :

csv_path = "C:/Users/mydata.csv"
csv_database = open(csv_path, delimiters=";")    
data_dict = csv.DictReader(csv_database, delimiter=";")

current_row = 0

number_list = []

for row in data_dict:

   current_row += 1 # Skip heading row

   if current_row == 1:
       continue

   # Here I add to a list of strings already created 
   name_list.append(row["Name"]) # Assuming the header of the column is "Name"

   # Here I add to a list of integer 
   if row['Number'] == 'NULL':
        int_list.append(0)

   elif row['Number'] != " ":

        int_list.append(int(row['Number'])) # Assuming the header is "Number"

   else:
        int_list.append(0)
  • What have you already tried? can you provide your sample code for us to comment on? – Fazer87 Jul 2 '14 at 8:09
  • I tried and it works, but before going any further I wanted to ask if I was going in the good direction. Anyway, code added :) – Jean Jul 2 '14 at 8:32
  • 2
    @JeanRene FYI: Fastest Python library to read a CSV file. – Franck Dernoncourt Jul 3 '14 at 20:31
  • 1
    @FranckDernoncourt Well thank you, that's a really useful benchmark! – Jean Jul 4 '14 at 8:53
1

Looks ok to me. This will work for your small scale CSVs (sub a couple of thousand rows).

When I had tp parse huge CSV Files (100k rows +) - I used the Cassava module which out performed the native modules by a long way.

Take a look at http://hackage.haskell.org/package/cassava

Hope this helps

  • 2
    Thanks but isn't Cassava for Haskell? Does it integrate well with a Python program? – Franck Dernoncourt Jul 3 '14 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.