18

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles).

Is there a simple way I can do this in bash?

Improve this question
5
  • Is there some reason you can't just use a text editor?
    – dangph
    Dec 14, 2012 at 13:52
  • The file is too large, its over over 20 times larger than my maximum virtual memory on this machine.
    – lynks
    Dec 14, 2012 at 13:54
  • 1
    fair enough. I was under the impression that good text editors could deal with very large files because they only load as much as they need to into memory, but after reading some of the other questions it seems that most of them have problems with them.
    – dangph
    Dec 14, 2012 at 14:11
  • @dangph True... may be odd but a few old and obsolete "text editors" (or so called) like edlin for ms-dos or ed for Unix didn't appear to have this problem but where only line editors... RAM at these times was a scarce and precious thing and a file size in Gigabytes was something impossible to believe! ;)
    – laurent
    Dec 14, 2012 at 19:29
  • To let things clear ;): I do NOT miss ed!!! but it would be a good fit in this case (not sure it could handle a file of this size...)
    – laurent
    Dec 14, 2012 at 19:31

2 Answers 2

Reset to default
24

You can use less to see what you want to edit and use sed to make the changes. This way you edit without loading the entire file.

Another way is to split the file, edit and join again:

split -b 10000k <file>

and to join:

cat xa* > <file>
Improve this answer
4
  • upvote for sed.
    – atroon
    Dec 14, 2012 at 13:58
  • Fantastic, split and join was exactly what I was looking for, thanks.
    – lynks
    Dec 14, 2012 at 14:01
  • sure, sed is better as it can search/replace the whole file easily but if he only needs to change the first line, split is not bad and faster too.
    – laurent
    Dec 14, 2012 at 14:01
  • 5
    sed would have taken hours to run over the whole file (which is just under half a TB), the changes were only on the first line, splitting it off seems sensible.
    – lynks
    Dec 14, 2012 at 14:02
4

If your modification changes the length of the file, the whole file needs to be re-written, see for example this discussion on SO. You should probably consider saving the data to a database.

Keeping that in mind, you can stream edit the file with sed. To replace the first line, do something like this (GNU sed):

< oldfile sed '1c\new_heading' > newfile
Improve this answer
2
  • that answer is unclear, is it supposed to work when you change the length of line, or only when the length remains the same?
    – Carmageddon
    Oct 28, 2022 at 2:59
  • @Carmageddon: I meant: if the length of the file changes, the whole file must be rewritten. For example with sed.
    – Thor
    Oct 28, 2022 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.