79

I need to substitute some text inside a text file with a replacement. Usually I would do something like

sed -i 's/text/replacement/g' path/to/the/file

The problem is that both text and replacement are complex strings containing dashes, slashes, blackslashes, quotes and so on. If I escape all necessary characters inside text the thing becomes quickly unreadable. On the other hand I do not need the power of regular expressions: I just need to substitute the text literally.

Is there a way to do text substitution without using regular expressions with some bash command?

It would be rather trivial to write a script that does this, but I figure there should exist something already.

  • Necessary to do it through bash? A simplistic solution would be to open in Word and do a find and replace all – Akash May 9 '12 at 15:04
  • 19
    @akash Because systems that have bash always ship with Microsoft Word? ;) No.. Just kidding. The OP might want to do this on a remote machine or for a batch of files though. – slhck May 9 '12 at 15:07
  • @slhck :) Well, I guess gedit should have a similar option – Akash May 9 '12 at 15:09
  • An option would be to somehow correctly escape everything before passing it to sed, which is probably a futile effort considering all the switches and platform differences. – l0b0 May 9 '12 at 15:11

14 Answers 14

10

When you don't need the power of regular expressions, don't use it. That is fine.
But, this is not really a regular expression.

sed 's|literal_pattern|replacement_string|g'

So, if / is your problem, use | and you don't need to escape the former.

PS: About the comments, also see this Stackoverflow answer on Escape a string for sed search pattern.


Update: If you are fine using Perl try it with \Q and \E like this,

 perl -pe 's|\Qliteral_pattern\E|replacement_string|g'

@RedGrittyBrick has also suggested a similar trick with stronger Perl syntax in a comment here or here

| improve this answer | |
  • 72
    I'm not sure this answer is useful... The only difference between s||| and s/// is that the seperator character is different and so that one character doesn't need escaping. You could equally do s###. The real issue here is that the OP doesn't want to have to worry about escaping the contents of literal_pattern (which is not literal at all and will be interpreted as a regex). – Benj May 9 '12 at 15:31
  • 19
    This will not avoid the interpretation of other special characters. What if search for 1234.*aaa with your solution it match much more than the intended 1234\.\*aaa. – Matteo May 9 '12 at 15:47
  • 27
    This answer should not be accepted – Steven Lu Aug 4 '16 at 18:01
  • 5
    This misses the point entirely. The text to be matched could contain any wierdness. In my case it's a random password. You know how those go – Christian Bongiorno Jun 11 '18 at 18:01
  • 1
    Use str =~ s/\Q$replace_this\E/$with_this/; in case of variables – Jithin Pavithran Sep 30 '18 at 14:26
16
export FIND='find this'
export REPLACE='replace with this'
ruby -p -i -e "gsub(ENV['FIND'], ENV['REPLACE'])" path/to/file

This is the only 100% safe solution here, because:

  • It's a static substition, not a regexp, no need to escape anything (thus, superior to using sed)
  • It won't break if your string contains } char (thus, superior to a submitted Perl solution)
  • It won't break with any character, because ENV['FIND'] is used, not $FIND. With $FIND or your text inlined in Ruby code, you could hit a syntax error if your string contained an unescaped '.
| improve this answer | |
  • I had to use export FIND='find this; export REPLACE='replace with this'; in my bash script so that ENV['FIND'] and ENV['replace'] had the expected values. I was replacing some really long encrypted strings in a file. This was just the ticket. – dmmfll Jun 24 '16 at 12:45
  • This is a good answer answer because it's reliable and ruby is ubiquitous. Based on this answer I now use this shell script. – loevborg Jul 9 '18 at 8:49
  • Unfortunately doesn't work when FIND contains multiple lines. – adrelanos Jan 26 '19 at 17:09
  • There's nothing that would prevent it from working with multiple lines in FIND. Use double quoted \n. – Nowaker Jan 27 '19 at 19:20
  • 1
    you dont need export, you can pass variables to subshells in sh by declaring them before the command on the same line; FIND='find this' REPLACE='replace with this' ruby .... with no newlines. Especially handy if you were never working with variables in the first place: FIND=`echo command` REPLACE=`cat replace.txt` ruby .... Use `\` if you still want newlines in the actual script anyway ;) – Hashbrown Jan 23 at 0:05
10

The replace command will do this.

https://linux.die.net/man/1/replace

Change in place:

replace text replacement -- path/to/the/file

To stdout:

replace text replacement < path/to/the/file

Example:

$ replace '.*' '[^a-z ]{1,3}' <<EOF
> r1: /.*/g
> r2: /.*/gi
> EOF
r1: /[^a-z ]{1,3}/g
r2: /[^a-z ]{1,3}/gi

The replace command comes with MySQL or MariaDB.

| improve this answer | |
  • 3
    take into account tht replace is deprecated and may not be disponible in the future – Rogelio Triviño Dec 11 '17 at 15:31
  • 2
    Why on earth does such basic command come with a database? – masterxilo Aug 8 '18 at 7:03
  • 8
    @masterxilo A better question might be – why does such a basic command not come with modern operating systems? ;-) – Mark Thomson Jan 22 '19 at 4:06
  • 1
    I installed MariaDB, cp /usr/bin/replace ~/.local/bin (in $PATH) and then uninstalled it. – Pablo A Jun 16 at 4:58
  • @PabloA Nice idea! – Nikhil Jun 21 at 17:40
3

You could also use perl's \Q mechanism to "quote (disable) pattern metacharacters"

perl -pe 'BEGIN {$text = q{your */text/?goes"here"}} s/\Q$text\E/replacement/g'
| improve this answer | |
  • 3
    Or perl -pe 's(\Qyour */text/?goes"here")(replacement)' file – RedGrittyBrick May 9 '12 at 22:52
3

check out my Perl script. it do exactly what you need without implicit or explicit use of regular expression :

https://github.com/Samer-Al-iraqi/Linux-str_replace

str_replace Search Replace File # replace in File in place

STDIN | str_replace Search Replace # to STDOUT

very handy right? I had to learn Perl to do it. because I really really need it.

| improve this answer | |
2

I pieced together a few other answers and came up with this:

function unregex {
   # This is a function because dealing with quotes is a pain.
   # http://stackoverflow.com/a/2705678/120999
   sed -e 's/[]\/()$*.^|[]/\\&/g' <<< "$1"
}
function fsed {
   local find=$(unregex "$1")
   local replace=$(unregex "$2")
   shift 2
   # sed -i is only supported in GNU sed.
   #sed -i "s/$find/$replace/g" "$@"
   perl -p -i -e "s/$find/$replace/g" "$@"
}
| improve this answer | |
  • Doesn't work with newlines. Also doesn't help to escape newlines with \n. Any solution? – adrelanos Mar 29 '19 at 8:44
2

You can do it by escaping your patterns. Like this:

keyword_raw='1/2/3'
keyword_regexp="$(printf '%s' "$keyword_raw" | sed -e 's/[]\/$*.^|[]/\\&/g')"
# keyword_regexp is now '1\/2\/3'

replacement_raw='2/3/4'
replacement_regexp="$(printf '%s' "$replacement_raw" | sed -e 's/[\/&]/\\&/g')"
# replacement_regexp is now '2\/3\/4'

echo 'a/b/c/1/2/3/d/e/f' | sed -e "s/$keyword_regexp/$replacement_regexp/"
# the last command will print 'a/b/c/2/3/4/d/e/f'

Credits for this solutions goes here: https://stackoverflow.com/questions/407523/escape-a-string-for-a-sed-replace-pattern

Note1: this only works for non-empty keywords. Empty keywords are not accepted by sed (sed -e 's//replacement/').

Note2: unfortunately, I don't know a popular tool that would NOT use regexp-s to solve the problem. You can write such a tool in Rust or C, but it's not there by default.

| improve this answer | |
  • 2
    This completely misses the OP's point. Obviously you can escape the pattern, but for some patterns this is tedious. – icecreamsword Oct 2 '18 at 19:35
  • @icecreamsword did you read my answer below the first line? The script does escaping automatically. – VasyaNovikov Oct 3 '18 at 4:24
1

You can use php's str_replace:

php -R 'echo str_replace("\|!£$%&/()=?^\"'\''","replace",$argn),PHP_EOL;'<input.txt >output.txt

Note: You would still need to escape single quotes ' and double quotes ", though.

| improve this answer | |
1

You can do this in sh without any script (though putting this "one-liner" into a script would be better) or non-standard external program (I reeeally liked @Nowaker's answer thanks to it's safety against injection, but this old CentOS box I needed this on didn't have ruby!). as long as perl isn't non-standard for you

Without attempting to escape the string (and account for issues with doing it correctly syntactically, knowing all the special characters, et cetera), we can just blanket encode all the strings so that nothing has the possibility of being special.

cat path/to/the/file | xxd -p | tr -d '\n' \
| perl -pe "s/$(printf '%s' 'text' | xxd -p | tr -d '\n')(?=(?:.{2})*\$)/$(printf '%s' 'replacement' | xxd -p | tr -d '\n')/g" \
| xxd -p -r

This was just to match the asker's example, other users can obviously replace 'text' with "$text" if using a variable, or cat path/to/the/file with printf '%s' "$input" if not using a file.

You can even replace the /g with / to make it replace-once, or otherwise edit the regex outside the $() to "escape" only portions of the matcher (say, add a ^ after s/ to make it match only the start of the file).
If in the above you need ^/$ to match ends-of-lines again you'll need unencode those:

cat path/to/the/file | xxd -p | tr -d '\n' | sed 's/0a/\n/g'\
| perl -pe "s/^$(printf '%s' 'text' | xxd -p | tr -d '\n')(?=(?:.{2})*\$)/$(printf '%s' 'replacement' | xxd -p | tr -d '\n')/g" \
| sed 's/\n/0a/g' | xxd -p -r

Which'll replace all lines in the file begining with 'text' to instead start with 'replacement'


Test:

Within ^/.[a]|$0\\{7}!!^/.[a]|$0\\{7}!!^/.[a]|$0\\{7}, replace literal ^/.[a]|$0\\{7} with literally $0\\

printf '%s' '^/.[a]|$0\\{7}!!^/.[a]|$0\\{7}!!^/.[a]|$0\\{7}' \
| xxd -p | tr -d '\n' \
| perl -pe "s/$(printf '%s' '^/.[a]|$0\\{7}' | xxd -p | tr -d '\n')(?=(?:.{2})*\$)/$(printf '%s' '$0\\' | xxd -p | tr -d '\n')/g" \
| xxd -p -r

Output:

$0\\!!$0\\!!$0\\
| improve this answer | |
  • (1) This looks a LOT like wef’s answer to a very similar question.  I’m not accusing you of copying it — it’s certainly possible for people to discover / devise the same solution independently — just advising you that somebody else posted this before you.  … (Cont’d) – G-Man Says 'Reinstate Monica' Apr 1 at 4:28
  • (Cont’d) …  (2) When I first saw this answer, I thought it was brilliant.  A few minutes later, I realized that, like a diamond, it was flawed.  For example, if you try to change E to g in a file that contains $Q, it will change to &q.  This is because E is 45, g is 67, and $Q is 2451, so, when you do s/45/67/, you change 2451 to 2671, which is &q (26 + 71).  … I have posted an answer that addresses this issue. – G-Man Says 'Reinstate Monica' Apr 1 at 4:28
  • thanks for that @G-ManSays'ReinstateMonica' not too hard to fix, it'll work now – Hashbrown Apr 1 at 6:33
0

Node.JS equivalent of @Nowaker:

export FNAME='moo.txt'
export FIND='search'
export REPLACE='rpl'
node -e 'fs=require("fs");fs.readFile(process.env.FNAME,"utf8",(err,data)=>{if(err!=null)throw err;fs.writeFile(process.env.FNAME,data.replace(process.env.FIND,process.env.REPLACE),"utf8",e=>{if(e!=null)throw e;});});'
| improve this answer | |
0

Heres one more "almost" working way.

Use vi or vim.

Create a textfile with your substitution in it:

:%sno/my search string \\"-:#2;g('.j');\\">/my replacestring=\\"bac)(o:#46;\\">/
:x

then execute vi or vim from the commandline:

vi -S commandfile.txt path/to/the/file

:%sno is the vi command to do search and replace without magic.

/ is my chosen separator.

:x saves and exits vi.

You need to escape backslashes '\' the forwardslash '/' may be replaced with e.g. a questionmark '?' or something else that is not in your search or replace-string, pipe '|' did not work for me tho.

ref: https://stackoverflow.com/questions/6254820/perform-a-non-regex-search-replace-in-vim https://vim.fandom.com/wiki/Search_without_need_to_escape_slash http://linuxcommand.org/lc3_man_pages/vim1.html

| improve this answer | |
0

Using a Simple Python Script

Most systems have python ready to go these days. So here's a simple script that'll work for ya:

# replace.py
# USAGE: python replace.py bad-word good-word target-file.txt
#
import sys

search_term = sys.argv[1]
replace_term = sys.argv[2]
target_file = sys.argv[3]

with open(target_file, 'r') as file:
        content = file.read()

content = content.replace(sys.argv[1], sys.argv[2])

with open(target_file, 'w') as file:
        file.write(content)

One Caveat: This works great if your good/bad words are already in system/environment variables. Just make sure you use double-quotes to wrap the variables when passing to the script.

For example:

python replace.py "$BAD_WORD" "$GOOD_WORD" target-file.txt

However, these will not work:

# This breaks on $ or " characters
BAD_WORD="your-artibrary-string"

# This breaks on ' characters
BAD_WORD='your-artibrary-string'

# This breaks on spaces plus a variety of characters
BAD_WORD=your-artibrary-string

Handling Arbitrary Literal Characters

1. Write the Chars to Disk

If I need to provide a arbitrary literal value to a script (skipping any escaping), I generally write it to disk using this method:

head -c -1 << 'CRAZY_LONG_EOF_MARKER' | tee /path/to/file > /dev/null
arbitrary-one-line-string
CRAZY_LONG_EOF_MARKER

... where:

  • We're employing the Here Document mechanism to write literal text
  • We're using head and tee to delete the trailing newline that Here Docs create
  • We're preventing evalution of variables inside the Here Doc by quoting the EOL marker string

Here's a quick demo with tricky chars:

head -c -1 << 'CRAZY_LONG_EOF_MARKER' | tee /path/to/file > /dev/null
1"2<3>4&5'6$7 # 8
CRAZY_LONG_EOF_MARKER

2. Use Modified Python Script

Here's an updated script that reads from word files:

# replace.py
# USAGE: python replace.py bad-word.txt good-word.txt target-file.txt
#
import sys

search_term_file = sys.argv[1]
replace_term_file = sys.argv[2]
target_file = sys.argv[3]

print [search_term_file, replace_term_file, target_file]

with open(search_term_file, 'r') as file:
        search_term = file.read()
with open(replace_term_file, 'r') as file:
        replace_term = file.read()
with open(target_file, 'r') as file:
        content = file.read()

print [search_term, replace_term]
content = content.replace(search_term, replace_term)

with open(target_file, 'w') as file:
        file.write(content)
| improve this answer | |
0

Working on an alpine docker container, i wasn't keen to install python / pearl / ruby / python just to do the very simple operation of a find and replace. All these solutions are horribly complex!!

There are two viable solutions to this:

  1. Use a different find + replace from elsewhere (e.g. python/pearl/etc)
  2. Escape all the regex metacharacters. We can use sed for this purpose.

I cannot do the first in my case as I am working in a minimal docker container.
This solution can be used for the second

In my case i had a known string in my file: {{replace_me}} and a user input. Lets call it $replace_text.

sed -i "s/{{replace_me}}/$(sed 's/[&/\]/\\&/g' <<<"$replace_text")/g" path/to/file

How does it work?

We use sed -i for an inplace conversion. Here i use the \ as a delimiter as I am specifically escaping this in my replacement line. This protects against the user putting down my\string.

The $(sed 's/[&/\]/\\&/g' <<<"$replace_text") bit is explained in detail here in a great answer where this solution is derived from. In this case I am using it as a one liner

In answer to OPs initial question, here is a sed one liner that should do the trick:

sed -i "s/$(sed 's/[^^]/[&]/g; s/\^/\\^/g' <<<"$search_text")/$(sed 's/[&/\]/\\&/g' <<<"$replace_text")/g" path/to/file

but i guess he probably no longer cares given that it has been 7 years!

| improve this answer | |
  • This is complicated. (1) When I started to read this, I thought “This is only trying to solve the second half of the problem — the replacement string.  That’s not ideal, but a partial answer is better than none.”  But then I saw that your answer was trying to solve the whole problem, and it’s wrong.  If $initial_text is 3.14, then the printf outputs 3.14.  But . is a special character in regular expressions, so s/3.14/foo/ will find and replace 3614, 3q14, 3%14, etc. … (Cont’d) – G-Man Says 'Reinstate Monica' Mar 27 at 4:33
  • (Cont’d) …  (2) You say that one can use “anything” as the delimiter for the s command.  That’s misleading; you can use any single character.  There are, indeed, times when doing text manipulation that we assume that we can choose a string that’s so “obscure” that it will not appear naturally in the data.  But you can’t do that with a single character; the initial_text and the replace_text can, between them, contain every valid ASCII character.  So what do you use as the delimiter then? … (Cont’d) – G-Man Says 'Reinstate Monica' Mar 27 at 4:33
  • (Cont’d) …  (3) Gotcha!  It seems like you don’t fully understand how your answer works.  printf "%q" "my|string" outputs my\|string, and so it will work even if you use | as the delimiter!  (4) In your answer, you put the $(…) command substitutions outside quotes.  This will cause it to break if initial_text or replace_text contains space(s). … (Cont’d) – G-Man Says 'Reinstate Monica' Mar 27 at 4:33
  • (Cont’d) …  (5) Your fourth paragraph (the third one after the first code block) uses sloppy terminology.  You say "$(printf "%q" "$replace_text")" converts a regex string to an escaped string.  But replace_text isn’t a regex — the replacement string in an s command is a specially formatted string that needs special handling, but it’s not a regex.  Please try to use correct terminology. (6) Your initial example is misleading.  In the context you show, { and } aren’t special characters and don’t need to be escaped. – G-Man Says 'Reinstate Monica' Mar 27 at 4:34
  • I concede that the "replace" and the python options are simpler, but as stated my solution is for a minimal docker container. There isn't really another solution here for that use case. Okay onto your points: (1) - valid, my understanding of printf mistook "ready for shell interpretation" as "regex safe" - clearly not the same thing. Will look for a replacement and update the answer (and my own code!). (2/3) fair, people not knowledgeable in sed wouldn't get that i meant any single char. Will update. Your insight is interesting. Any Regexp char works, if we know that others are escaped. – ThatGuyCalledRob Mar 27 at 16:12
0

This is an enhancement of Hashbrown’s answer (and wef’s answer to a very similar question).

We can remove the issue of the special meaning of various special characters and strings (^, ., [, *, $, \(, \), \{, \}, \+, \?, &, \1, …, whatever, and the / delimiter) by removing the special characters. Specifically, we can convert everything to hex; then we have only 0-9 and a-f to deal with.  This example demonstrates the principle:

$ echo -n '3.14' | xxd
0000000: 332e 3134                                3.14

$ echo -n 'pi'   | xxd
0000000: 7069                                     pi

$ echo '3.14 is a transcendental number.  3614 is an integer.' | xxd
0000000: 332e 3134 2069 7320 6120 7472 616e 7363  3.14 is a transc
0000010: 656e 6465 6e74 616c 206e 756d 6265 722e  endental number.
0000020: 2020 3336 3134 2069 7320 616e 2069 6e74    3614 is an int
0000030: 6567 6572 2e0a                           eger..

$ echo "3.14 is a transcendental number.  3614 is an integer." | xxd -p \
                                                        | sed 's/332e3134/7069/g' | xxd -p -r
pi is a transcendental number.  3614 is an integer.

whereas, of course, sed 's/3.14/pi/g' would also change 3614.

The above is a slight oversimplification; it doesn’t account for boundaries.  Consider this (somewhat contrived) example:

$ echo -n 'E' | xxd
0000000: 45                                       E

$ echo -n 'g' | xxd
0000000: 67                                       g

$ echo '$Q Eak!' | xxd
0000000: 2451 2045 616b 210a                      $Q Eak!.

$ echo '$Q Eak!' | xxd -p | sed 's/45/67/g' | xxd -p -r
&q gak!

Because $ (24) and Q (51) combine to form 2451, the s/45/67/g command rips it apart from the inside.  It changes 2451 to 2671, which is &q (26 + 71).  We can prevent that by separating the bytes of data in the search text, the replacement text and the file with spaces.  Here’s a stylized solution:

encode() {
        xxd -p    -- "$@" | sed 's/../& /g' | tr -d '\n'
}
decode() {
        xxd -p -r -- "$@"
}
left=$( printf '%s' "$search"      | encode)
right=$(printf '%s' "$replacement" | encode)
encode path/to/the/file | sed "s/$left/$right/g" | decode

I defined an encode function because I used that functionality three times, and then I defined decode for symmetry.  If you don’t want to define a decode function, just change the last line to

encode path/to/the/file | sed "s/$left/$right/g" | xxd -p –r

Note that the encode function triples the size of the data (text) in the file, and then sends it through sed as a single line — without even having a newline at the end.  GNU sed seems to be able to handle this; other versions might not be able to.  Also, this does not modify the file in place; you’ll need to write the output to a temporary file and then copy it over the original file (or one of the other tricks for doing that).

As an added bonus, this solution handles multi-line search and replace (in other words, search and replacement strings that contain newline(s)).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.