67

I need to substitute some text inside a text file with a replacement. Usually I would do something like

sed -i 's/text/replacement/g' path/to/the/file

The problem is that both text and replacement are complex strings containing dashes, slashes, blackslashes, quotes and so on. If I escape all necessary characters inside text the thing becomes quickly unreadable. On the other hand I do not need the power of regular expressions: I just need to substitute the text literally.

Is there a way to do text substitution without using regular expressions with some bash command?

It would be rather trivial to write a script that does this, but I figure there should exist something already.

  • Necessary to do it through bash? A simplistic solution would be to open in Word and do a find and replace all – Akash May 9 '12 at 15:04
  • 17
    @akash Because systems that have bash always ship with Microsoft Word? ;) No.. Just kidding. The OP might want to do this on a remote machine or for a batch of files though. – slhck May 9 '12 at 15:07
  • @slhck :) Well, I guess gedit should have a similar option – Akash May 9 '12 at 15:09
  • An option would be to somehow correctly escape everything before passing it to sed, which is probably a futile effort considering all the switches and platform differences. – l0b0 May 9 '12 at 15:11

10 Answers 10

7

When you don't need the power of regular expressions, don't use it. That is fine.
But, this is not really a regular expression.

sed 's|literal_pattern|replacement_string|g'

So, if / is your problem, use | and you don't need to escape the former.

ps: about the comments, also see this Stackoverflow answer on Escape a string for sed search pattern.


Update: If you are fine using Perl try it with \Q and \E like this,
perl -pe 's|\Qliteral_pattern\E|replacement_string|g'
RedGrittyBrick has also suggested a similar trick with stronger Perl syntax in a comment here

  • Thank you, I did not know about the difference between / and | – Andrea May 9 '12 at 15:18
  • 62
    I'm not sure this answer is useful... The only difference between s||| and s/// is that the seperator character is different and so that one character doesn't need escaping. You could equally do s###. The real issue here is that the OP doesn't want to have to worry about escaping the contents of literal_pattern (which is not literal at all and will be interpreted as a regex). – Benj May 9 '12 at 15:31
  • 15
    This will not avoid the interpretation of other special characters. What if search for 1234.*aaa with your solution it match much more than the intended 1234\.\*aaa. – Matteo May 9 '12 at 15:47
  • 19
    This answer should not be accepted – Steven Lu Aug 4 '16 at 18:01
  • 2
    This misses the point entirely. The text to be matched could contain any wierdness. In my case it's a random password. You know how those go – Christian Bongiorno Jun 11 '18 at 18:01
13
export FIND='find this'
export REPLACE='replace with this'
ruby -p -i -e "gsub(ENV['FIND'], ENV['REPLACE'])" path/to/file

This is the only 100% safe solution here, because:

  • It's a static substition, not a regexp, no need to escape anything (thus, superior to using sed)
  • It won't break if your string contains } char (thus, superior to a submitted Perl solution)
  • It won't break with any character, because ENV['FIND'] is used, not $FIND. With $FIND or your text inlined in Ruby code, you could hit a syntax error if your string contained an unescaped '.
  • I had to use export FIND='find this; export REPLACE='replace with this'; in my bash script so that ENV['FIND'] and ENV['replace'] had the expected values. I was replacing some really long encrypted strings in a file. This was just the ticket. – DMfll Jun 24 '16 at 12:45
  • This is a good answer answer because it's reliable and ruby is ubiquitous. Based on this answer I now use this shell script. – loevborg Jul 9 '18 at 8:49
  • Unfortunately doesn't work when FIND contains multiple lines. – adrelanos Jan 26 at 17:09
  • There's nothing that would prevent it from working with multiple lines in FIND. Use double quoted \n. – Nowaker Jan 27 at 19:20
7

The replace command will do this.

https://linux.die.net/man/1/replace

Change in place:

replace text replacement -- path/to/the/file

To stdout:

replace text replacement < path/to/the/file

Example:

$ replace '.*' '[^a-z ]{1,3}' <<EOF
> r1: /.*/g
> r2: /.*/gi
> EOF
r1: /[^a-z ]{1,3}/g
r2: /[^a-z ]{1,3}/gi

The replace command comes with MySQL or MariaDB.

  • 3
    take into account tht replace is deprecated and may not be disponible in the future – Rogelio Dec 11 '17 at 15:31
  • 1
    Why on earth does such basic command come with a database? – masterxilo Aug 8 '18 at 7:03
  • 3
    @masterxilo A better question might be – why does such a basic command not come with modern operating systems? ;-) – Mark Thomson Jan 22 at 4:06
3

You could also use perl's \Q mechanism to "quote (disable) pattern metacharacters"

perl -pe 'BEGIN {$text = q{your */text/?goes"here"}} s/\Q$text\E/replacement/g'
  • 3
    Or perl -pe 's(\Qyour */text/?goes"here")(replacement)' file – RedGrittyBrick May 9 '12 at 22:52
3

check out my Perl script. it do exactly what you need without implicit or explicit use of regular expression :

https://github.com/Samer-Al-iraqi/Linux-str_replace

str_replace Search Replace File # replace in File in place

STDIN | str_replace Search Replace # to STDOUT

very handy right? I had to learn Perl to do it. because I really really need it.

2

You can do it by escaping your patterns. Like this:

keyword_raw='1/2/3'
keyword_regexp="$(printf '%s' "$keyword_raw" | sed -e 's/[]\/$*.^|[]/\\&/g')"
# keyword_regexp is now '1\/2\/3'

replacement_raw='2/3/4'
replacement_regexp="$(printf '%s' "$replacement_raw" | sed -e 's/[\/&]/\\&/g')"
# replacement_regexp is now '2\/3\/4'

echo 'a/b/c/1/2/3/d/e/f' | sed -e "s/$keyword_regexp/$replacement_regexp/"
# the last command will print 'a/b/c/2/3/4/d/e/f'

Credits for this solutions goes here: https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern

Note1: this only works for non-empty keywords. Empty keywords are not accepted by sed (sed -e 's//replacement/').

Note2: unfortunately, I don't know a popular tool that would NOT use regexp-s to solve the problem. You can write such a tool in Rust or C, but it's not there by default.

  • This completely misses the OP's point. Obviously you can escape the pattern, but for some patterns this is tedious. – icecreamsword Oct 2 '18 at 19:35
  • @icecreamsword did you read my answer below the first line? The script does escaping automatically. – VasyaNovikov Oct 3 '18 at 4:24
1

I pieced together a few other answers and came up with this:

function unregex {
   # This is a function because dealing with quotes is a pain.
   # http://stackoverflow.com/a/2705678/120999
   sed -e 's/[]\/()$*.^|[]/\\&/g' <<< "$1"
}
function fsed {
   local find=$(unregex "$1")
   local replace=$(unregex "$2")
   shift 2
   # sed -i is only supported in GNU sed.
   #sed -i "s/$find/$replace/g" "$@"
   perl -p -i -e "s/$find/$replace/g" "$@"
}
  • Doesn't work with newlines. Also doesn't help to escape newlines with \n. Any solution? – adrelanos Mar 29 at 8:44
1

You can use php's str_replace:

php -R 'echo str_replace("\|!£$%&/()=?^\"'\''","replace",$argn),PHP_EOL;'<input.txt >output.txt

Note: You would still need to escape single quotes ' and double quotes ", though.

0

Node.JS equivalent of @Nowaker:

export FNAME='moo.txt'
export FIND='search'
export REPLACE='rpl'
node -e 'fs=require("fs");fs.readFile(process.env.FNAME,"utf8",(err,data)=>{if(err!=null)throw err;fs.writeFile(process.env.FNAME,data.replace(process.env.FIND,process.env.REPLACE),"utf8",e=>{if(e!=null)throw e;});});'
0

Heres one more "almost" working way.

Use vi or vim.

Create a textfile with your substitution in it:

:%sno/my search string \\"-:#2;g('.j');\\">/my replacestring=\\"bac)(o:#46;\\">/
:x

then execute vi or vim from the commandline:

vi -S commandfile.txt path/to/the/file

:%sno is the vi command to do search and replace without magic.

/ is my chosen separator.

:x saves and exits vi.

You need to escape backslashes '\' the forwardslash '/' may be replaced with e.g. a questionmark '?' or something else that is not in your search or replace-string, pipe '|' did not work for me tho.

ref: https://stackoverflow.com/questions/6254820/perform-a-non-regex-search-replace-in-vim https://vim.fandom.com/wiki/Search_without_need_to_escape_slash http://linuxcommand.org/lc3_man_pages/vim1.html

New contributor
Samuel Åslund is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.