Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am using Ubuntu and looking for a good editor to edit a file that is > 4GB. I just need to put content at the end and beginning of the file. I suppose I could use something like

cat "text to add" >> huge_file

To append to the file. Is that the route to go? What about prepending? In general, what is the best route if I wanted to edit somewhere in the middle?

I've tried VIM and it fails miserably. I assume emacs and nano would be even worse. What else is there? I assume to accomplish what I am looking for, the editor would have to be specifically designed for this by not keeping the entirety of the file's contents in memory.

share|improve this question
    
I'm curious as to why you have so much data in a text file. Not the most manageable format for big data, as you've discovered. –  Isaac Rabinovitch Nov 18 '12 at 17:29
add comment

5 Answers

up vote 4 down vote accepted

This is a partial answer, but yes, if you are simply appending I would use:

cat extra.txt >> huge_file.txt

or

echo "Single line to add" >> huge_file.txt

For prepending I would do:

cat extra.txt huge_file.txt > huge_file_new.txt
share|improve this answer
    
This is what I wound up using, but would be nice if there was a more robust method. –  Gordon Aug 2 '10 at 22:14
add comment

If all you need is to append, then >>huge_file is much better than what any editor can do, because it won't touch the existing data. Inserting data into a file requires rewriting everything after the insertion point, so it'll be slow even with the right tools.

With vim, make sure you try with the LargeFile plugin.

Bvi is a version of vi that targets binary files. It can edit a slice of a file (i.e. from position x to position y).

The wikipedia editor comparison page has a column with large file support. The situation is pretty bleak.

share|improve this answer
    
Vim should be able to handle files of up to 2 Gb on a 32 bit system. I think the comparison page lists it as 2^32 (4 Gb) not sure if that's correct in all versions of vim. –  tovare Aug 2 '10 at 21:37
    
AFAIK support for files over 2GB on a 32-bit system is new in the soon-to-be-released vim 7.3 (patch 7.2.413), though I don't know what you can do with a file that doesn't fit in the process address space. –  Gilles Aug 2 '10 at 22:30
add comment

Ultraedit is the only editor that does this well. I would, also, prefer an OSS. One doesn't exist. I'm particularly surprised that this isn't a capability of vi... it's such a swiss army knife.

share|improve this answer
add comment

UltraEdit is capable of editing a file without loading it into memory. I haven't tested the Linux version of it yet, and neither have I tested a >4GB file, so I cannot give any guarantees, but I edited some huge (~1.5 GB) files with the Windows version (and long ago, too).

share|improve this answer
    
I would prefer OSS if possible. –  Gordon Aug 2 '10 at 20:55
add comment

Just because I found this via google looking for something similar this is another method I found.

Situation: You have a 4.0 GB file and you need to edit line number 120.

Solution: Using head and tail to get everything except the line you want to edit.

# cat origfile.txt 
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
# head -n 5 origfile.txt >> newfile.txt
# echo "line 6" >> newfile.txt 
# tail -n +7 origfile.txt >> newfile.txt 
# cat newfile.txt 
 1
 2
 3
 4
 5
 line 6
 7
 8
 9
 10

From my testing tail will not load the entire file into memory all at once.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.