Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a big text file (about 2GB). I want to do five search and replace actions on the same file, and would like to do this in one command. Normally I use vim, open the file, do one replace action, then the next, etc. There is one catch, as I noticed that after three or four searches vim crashes because of memory issues.

Here are two examples of the command I use in Vim:

:%s/www\.abcdef/www.test.abcdef/g 
:%s/www\.klmnop/www.test.klmnop/g

What is the best way to handle this?

share|improve this question
up vote 7 down vote accepted

I would use sed like this :

sed -i "s/www\.abcdef/www.test.abcdef/g;s/www\.kmlnop/www.test.klmnop/g;" yourfile.txt

-i option stands for "in place" replacement. You can tell sed to create a backup of your file providing an extension to this option ( -i.bak will backup yourfile.txt as yourfile.txt.bak ).

share|improve this answer
    
That's quick! Not only your answer ;-) but this script with 5 search and replaces is about 10x faster as just opening the file in vim. One thing confused me though. At first I thought the .bak file would be the edited file, but it's the original of course. – SPRBRN Jul 11 '13 at 12:36
    
Ten search and replace actions (with thousands of hits) in a 2GB file in one go, no memory problems. Less than two minutes on an average desktop - super! – SPRBRN Jul 11 '13 at 12:54
    
One question... You escape the dots in the replace string. Is this necessary? – SPRBRN Jul 11 '13 at 13:00
1  
You're welcome @rxt :) Actually, you're right, you can use non-escaped dots in the replacement string in sed. I tried, and it works. There's a good thread in Unix & Linux Stackexchange, and the accepted answer not mention dots as characters to escape. – ssssteffff Jul 11 '13 at 13:08
2  
@rxt you said replace string, sorry, no you don't need to escape them there. – terdon Jul 11 '13 at 13:09

If you have many more search patterns, you could save them in a file and read the substitutions from there. For example, say these are the contents of replacements.txt:

www\.abcdef www.test.abcdef 
www\.klmnop www.test.klmnop

You can then read a list of N replacements and replace them with this:

while read from to; do
  sed -i "s/$from/$to/" infile.txt ; 
done < replacements.txt 

NOTES:

  • This assumes your search strings do not contain spaces and any strange characters need to be escaped in replacements.txt.
  • It will run one sed per replacement which may take a while if you have many replacement operations.
  • It can deal with an arbitrary number of replacements (thousands or millions or whatever) as long as you don't mind that it will take a bit more time.

Another option would be to write the above as a sed script:

s/www\.abcdef/www\.test\.abcdef/g;
s/www\.kmlnop/www\.test\.klmnop/g;
s/aaaa/bbbb/g;
s/cccc/dddd/g;
s/eeee/ffff/g;

You can then run the script on your file and it will make all the replacements in one go:

sed -f replace.sed infile.txt 
share|improve this answer
    
+1 for the ,,other option''. Could be handy to have the replacements stored in a file! (I hope I'll remember that...) – mpy Nov 17 '13 at 11:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .