Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Possible Duplicates:
Text Editor for very big file - Windows
What editor/viewer to use to inspect large text based files?

I have a CSV file which has about 4 million rows and is about 500 MB in size. Can you recommend any editor that can open the file without making the system crawl? I tried EmEditor but it is complaining that there are too many characters in a single line.

share|improve this question

marked as duplicate by Nifle, quack quixote May 26 '10 at 17:36

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

    
potential duplicates: superuser.com/questions/34749/… and superuser.com/questions/4044/… ... also see the large tag –  quack quixote May 6 '10 at 21:29
    
related Stack Overflow question: stackoverflow.com/questions/159521/… –  dag729 May 6 '10 at 21:58
1  
@dag: it's not exactly a duplicate if it's not on Super User. we can't close questions as duplicates of cross-site posts. –  quack quixote May 6 '10 at 22:08
    
I see, thanks for the info (and for the edit) :D –  dag729 May 6 '10 at 23:49
add comment

7 Answers 7

I've used JujuEdit (free) to work with 2GB CSVs in the past. Handles 'em pretty well. What are you doing with the contents? Another option might be to make an MS Access linked table pointed to the file if you need to do some querying/sorting/filtering.

share|improve this answer
add comment

You could give CSVed a try. It claims to load parts of the file on demand, so it might work for you.

share|improve this answer
add comment

Define "edit". How do you want to edit the file? Are you just viewing it, or do you want to make changes? Are they changes that you'd be making programmatically? If so, you may want to look at using awk, sed or Perl to do it.

As to "it is complaining that there are too many characters in a single line," it's probably that it's got line endings it's not able to handle. It would help if you'd tell us what platform you're on.

share|improve this answer
    
+1 for recommending good UNIX tools, and for the line ending issue troubleshooting suggestion. –  Chris W. Rea May 7 '10 at 0:45
add comment

Have you tried Excel? It handled CSV files.

I'm not sure of Excel 2007's file limitation but Excel 2010 64 bit now supports spreadsheets up to 4GB in size.

share|improve this answer
2  
Excel is terrible at round-tripping CSV files. Getting Excel to save a CSV file back out without mangling it somehow in the process (by Excel making incorrect assumptions about the data formats) is not trivial. For instance, I routinely find fields like employee IDs or SSNs are getting leading zeros removed on saving out from Excel because it assumed they're numeric columns. –  Chris W. Rea May 7 '10 at 0:42
    
Yes, a fair point. You can mitigate the problem by specifying the column type (eg Text for a column containing IDs/SSNs with leading zeros) but it must be done manually when opening the file, which is a pain. But if other CSV editors choke on the file size then the round-tripping through Excel may still be worth the additional vigilance required. –  Mike Fitzpatrick May 7 '10 at 1:08
    
I was hoping the 64-bit version of Excel would be the solution, but sadly it has the same row limit as the 32-bit version: 1,048,576. The only difference between the two is that 64 can handle files bigger than 2GB –  atraudes Aug 14 '12 at 14:45
add comment

Vim should work pretty well if you change a couple settings to turn off swap space and undo functionality: http://vim.wikia.com/wiki/VimTip611

share|improve this answer
add comment

I believe that UltraEdit probably could. I've used it on superhuge Gig+ files and it didn't crash out, but really, locating whatever data you are looking for is likely to be a pain. Can you do whatever you need to do programmetrically? Since you said CSV, assume it is text data, so maybe Perl might be a good choice due to its string handling and decent RegEx.

share|improve this answer
    
+1 for Perl recommendation. –  Chris W. Rea May 7 '10 at 0:45
    
Thanks, UltraEdit was the only one that managed to open the file. –  Giorgi May 8 '10 at 13:05
    
@Giorgi: "the only one" out of which? People might be curious to hear the other editors you tried that didn't work. –  Herbert Sitz May 8 '10 at 16:59
    
I tried all suggested in this thread. –  Giorgi May 8 '10 at 18:47
    
@Giorgi: There must be something extra-unusual about your file. On my wimpy netbook gVim (with no swap file and disabled undo) opens a 750MB test file in about 10 seconds, and jumps from the beginning to the end of 10 million line file are instantaneous. –  Herbert Sitz May 8 '10 at 21:52
show 1 more comment
up vote 1 down vote accepted

I tested all the editors suggested there and the only one that managed to open it (in a reasonable time) was UltraEdit.


Update

It turned out that the file did not have any line break in it (it used some other character for it) so that's why it was difficult to open it. I wrote a program which replaced all these chars with line break and EmEditor opened it in seconds.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.