Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles).

Is there a simple way I can do this in bash?

share|improve this question
    
Is there some reason you can't just use a text editor? –  dangph Dec 14 '12 at 13:52
    
The file is too large, its over over 20 times larger than my maximum virtual memory on this machine. –  lynks Dec 14 '12 at 13:54
1  
fair enough. I was under the impression that good text editors could deal with very large files because they only load as much as they need to into memory, but after reading some of the other questions it seems that most of them have problems with them. –  dangph Dec 14 '12 at 14:11
    
@dangph True... may be odd but a few old and obsolete "text editors" (or so called) like edlin for ms-dos or ed for Unix didn't appear to have this problem but where only line editors... RAM at these times was a scarce and precious thing and a file size in Gigabytes was something impossible to believe! ;) –  laurent Dec 14 '12 at 19:29
    
To let things clear ;): I do NOT miss ed!!! but it would be a good fit in this case (not sure it could handle a file of this size...) –  laurent Dec 14 '12 at 19:31

2 Answers 2

up vote 13 down vote accepted

You can use less to see what you want to edit and use sed to make the changes. This way you edit without loading the entire file.

Another way is to split the file, edit and join again:

split -b 10000k <file>

and to join:

cat xa* > <file>
share|improve this answer
    
upvote for sed. –  atroon Dec 14 '12 at 13:58
    
Fantastic, split and join was exactly what I was looking for, thanks. –  lynks Dec 14 '12 at 14:01
    
sure, sed is better as it can search/replace the whole file easily but if he only needs to change the first line, split is not bad and faster too. –  laurent Dec 14 '12 at 14:01
3  
sed would have taken hours to run over the whole file (which is just under half a TB), the changes were only on the first line, splitting it off seems sensible. –  lynks Dec 14 '12 at 14:02

If your modification changes the length of the line, the whole file needs to be re-written, see for example this discussion on SO. You should probably consider saving the data to a database.

Keeping that in mind, you can stream edit the file with sed. To replace the first line, do something like this (GNU sed):

< oldfile sed '1c\new_heading' > newfile
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.