Disk Based Text Editor to Edit File Larger than Memory

Question

I am using Ubuntu and looking for a good editor to edit a file that is > 4GB. I just need to put content at the end and beginning of the file. I suppose I could use something like

cat "text to add" >> huge_file

To append to the file. Is that the route to go? What about prepending? In general, what is the best route if I wanted to edit somewhere in the middle?

I've tried VIM and it fails miserably. I assume emacs and nano would be even worse. What else is there? I assume to accomplish what I am looking for, the editor would have to be specifically designed for this by not keeping the entirety of the file's contents in memory.

Answer 1

This is a partial answer, but yes, if you are simply appending I would use:

cat extra.txt >> huge_file.txt

or

echo "Single line to add" >> huge_file.txt

For prepending I would do:

cat extra.txt huge_file.txt > huge_file_new.txt

Answer 2

If all you need is to append, then >>huge_file is much better than what any editor can do, because it won't touch the existing data. Inserting data into a file requires rewriting everything after the insertion point, so it'll be slow even with the right tools.

With vim, make sure you try with the LargeFile plugin.

Bvi is a version of vi that targets binary files. It can edit a slice of a file (i.e. from position x to position y).

The wikipedia editor comparison page has a column with large file support. The situation is pretty bleak.

Answer 3

UltraEdit is capable of editing a file without loading it into memory. I haven't tested the Linux version of it yet, and neither have I tested a >4GB file, so I cannot give any guarantees, but I edited some huge (~1.5 GB) files with the Windows version (and long ago, too).

Answer 4

Ultraedit is the only editor that does this well. I would, also, prefer an OSS. One doesn't exist. I'm particularly surprised that this isn't a capability of vi... it's such a swiss army knife.

Answer 5

Just because I found this via google looking for something similar this is another method I found.

Situation: You have a 4.0 GB file and you need to edit line number 120.

Solution: Using head and tail to get everything except the line you want to edit.

# cat origfile.txt 
 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
# head -n 5 origfile.txt >> newfile.txt
# echo "line 6" >> newfile.txt 
# tail -n +7 origfile.txt >> newfile.txt 
# cat newfile.txt 
 1
 2
 3
 4
 5
 line 6
 7
 8
 9
 10

From my testing tail will not load the entire file into memory all at once.