13

I am having some problems with config files which have the non-breaking space chars in them.

How should I specify that character with sed so I can replace it with a space.

sed -n 's/ / /g'

examples of the errors

service named restart
Stopping named:                                            [  OK  ]
Starting named: 
Error in named configuration:
named.localhost:2: unknown RR type 'SOA '
named.localhost:8: unknown RR type '@'
named.localhost:9: unknown RR type '127.0.0.1'
named.localhost:10: unknown RR type '::1'
.....

I tried to include a line form the original offending file in this post. It does not seams to be working. Pastebin download seams to be the only tool that keeps all the original binary. http://pastebin.com/ZqT1EWbS. You should be able to copy and past the original line and have it work in your terminal.

3
  • you could googled for a sed manual, and look up hex you'd see you can specify characters in hex e.g. the non-breaking space \xA0 You can specify in decimal too. \d160 but you know when you look up hex in the manual it'd be talking about specifying characters in hex. There are a small handful of mentions of decimal so it's not hard to look up decimal in the manual either. I have a manual for GNU 4.2.1 of Sed. It has about \d and \x specifying hex.
    – barlop
    Dec 11, 2012 at 15:48
  • @barlop can you paste a link to documentation that demonstrates hex and decimal use of sed.
    – nelaaro
    Dec 12, 2012 at 11:20
  • mine is from gnuwin32 sed comes with a pdf. There is a manual online here gnu.org/software/sed/manual/sed.html do edit..find..hex you find the same contents (though I see in one place, one heading like "some sample scripts" in my pdf, vs "examples" in that one, differ very slightly) but most of rest is same word for word, I could edit..find phrases in one and find the same content in the other and the same smaller significant headings too. so details are the same. It seems to be probably almost identical to what I have.
    – barlop
    Dec 12, 2012 at 12:24

3 Answers 3

24

The answer to this question depends on which of the non-breaking space characters you are encountering.

Below are examples of how to replace each of the non-breaking space characters mentioned in the questions title and additionally the UTF-8 version (C2 A0) that the OP is actually asking about according to the pastebin output.

All examples use printf to generate the output as it is more portable than echo. The space characters are replaced by X's to make the output clearer.

Examples

html

printf '&#nbsp;\n' | sed 's/&#nbsp;/X/g'
printf ' \n'  | sed 's/ /X/g'
printf ' \n'  | sed 's/&#x[aA]0;/X/g'

octal 240 = decimal 160 = hex A0

printf '\xA0\n' | sed 's/\xA0/X/g'

Or with tr:

printf '\xA0\n' | tr '\240' 'X'

U+00A0

printf '\x00\xA0\n' | sed 's/\x00\xA0/X/g'

UTF-8

printf '\xC2\xA0\n' | sed 's/\xC2\xA0/X/g'

Result

Output in all of the above cases is:

X

Answer

Now to your question, you have data that looks like this:

printf '@       IN SOA  @ rname.invalid. (' | od -x

Output:

0000000 c240 c2a0 c2a0 c2a0 c2a0 c2a0 20a0 4e49
0000020 5320 414f a0c2 4020 7220 616e 656d 692e
0000040 766e 6c61 6469 202e 0a28
0000052

In order to replace the C2 A0s with ordinary space, use this:

printf '@       IN SOA  @ rname.invalid. (' | sed 's/\xC2\xA0/ /g' | od -x

Output:

0000000 2040 2020 2020 2020 4e49 5320 414f 2020
0000020 2040 6e72 6d61 2e65 6e69 6176 696c 2e64
0000040 2820 000a
0000044
7
  • what does the -n do? maybe I did something wrong but without -n it works. with -n I get no output and the file remains the same
    – barlop
    Dec 11, 2012 at 14:55
  • @barlop: it makes sed not print the pattern space, and is not supposed to be there. Fixed, thanks.
    – Thor
    Dec 11, 2012 at 14:59
  • If one wanted to do the sed destructively rather than constructively(to borrow a phrase that I think is from programming), you can do C:\>sed -i "s/\xA0/ /g" a.txt It's probably better to do it constructively as you have. But that's a line which will show nothing on the screen, but do something(I mention it since we mentioned suppressing output). sed -i t writes the result into a.txt or works in a.txt so the desired output is in there, and no output on screen.
    – barlop
    Dec 11, 2012 at 15:42
  • 2
    I would probably add to @barlop's response to suggest, assuming they want to save the changes to the file, sed -i.orig 's/\xa0/ /g' That way it't updates the original file but saves the originial as originalname.txt.orig (or whatever extension specified, such as .bak)
    – nerdwaller
    Dec 11, 2012 at 16:16
  • 2
    Unicode no-break space is U+00A0, which is encoded as C2 A0 in UTF-8. If your file is encoded in UTF-8, sed 's/\xa0/ /g' will remove only the A0 character and leave the C2. That is, at least the behaviour I'm seeing, even with a UTF-8 locale in use. Apr 25, 2014 at 10:36
2

Thanks for all those who help me get to a working solution.

I tried to include a line form the original offending file in this post. It does not seams to be working. Pastebin download seams to be the only tool that keeps all the original binary. http://pastebin.com/ZqT1EWbS. You should be able to copy and past the original line and have it work in your terminal.

So here is what happens if I remove the octal \0240 or hex \xA0. It adds some other funky characters.

$ echo "@       IN SOA  @ rname.invalid. (" | sed -e "s/\xA0//g"
@������ IN SOA� @ rname.invalid. (

There is some extra data not printed in the actual files. I found the od (octal dump) tool quite useful to show me what the actual hex / oct / binary values for the whole line are.

$ echo "@       IN SOA  @ rname.invalid. (" | od -x
0000000 c240 c2a0 c2a0 c2a0 c2a0 c2a0 20a0 4e49
0000020 5320 414f a0c2 4020 7220 616e 656d 692e
0000040 766e 6c61 6469 202e 0a28
0000052

The other character that kept showing up was \xC2 It is not printed when the non breaking space \xA0 is there, but shows up if the nbsp is removed. So I had to modify the sed line in the @Thor answer to remove it as well.

This is what worked for me.

$ echo "@       IN SOA  @ rname.invalid. (" | sed -e "s/\xC2\xA0/ /g"
@       IN SOA  @ rname.invalid. (
10
  • looks like the characters get broken when putting it in pastebin too. Here is the output I get from that echo line pastebin.com/raw.php?i=KfWPSXWN notice the hex is different from your hex. Can you upload the file to ge.tt ? (I know you solved it's still worthwhile).
    – barlop
    Dec 12, 2012 at 14:18
  • @barlop ge.tt/72UA2QT/v/0
    – nelaaro
    Dec 12, 2012 at 14:24
  • pastebin.com/raw.php?i=ZqT1EWbS does not work. Go the link directly pastebin.com/ZqT1EWbS. and use the raw out section at the bottom. I tested it and it worked as I expect. I think it is probably some thing to do with the web browser / server / php doing character conversion / filtering.
    – nelaaro
    Dec 12, 2012 at 14:28
  • I think non-breaking spaces can come out funny when you pipe them to some commands, od included. so for me, echo a <non-brekaing spaces> | od -tx1 prints 61 ff ff ff ff ff ff ff ff ff ff 0d 0a. So to remove my non-breaking spaces, I have to do echo a b | sed "s/\xff/we/g" Your non-breaking spaces come out funny but a different funny code to my funny code. (I can copy/paste non-breaking spaces that echo outputs, so echo doesn't mess them up, but they get messed up when piped). So what we're able to do, is use sed but on the messed up codes, and we see them with od..
    – barlop
    Dec 12, 2012 at 14:33
  • and by the way, I suggest od -tx1 or -td1 for decimal. as I find it looks clearer like that and you might too. Weirdly things can come out backwards otherwise. Try echo abc | od -x. Then try with | od -tx1 I am not sure why when you did od -x it didn't come out backwards or maybe it did i'm not sure. I can't test as my scrambled code was ff. But compare -tx1 with -x, and see what you think.
    – barlop
    Dec 12, 2012 at 14:36
2

Replace non-breaking space with space in a UTF-8 file

The TLDR version of Thors answer

sed -i 's/\xC2\xA0/ /g' filename

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .