So, for my statistics class, I've been using a lot of linux trickery to organize my numbers, copy/paste them around, etc. Now, I haven't had any issues until just recently... I had copy pasted a bunch of values out of Open Office (could be the culprit), and was trying to format them like so...

echo "5  8  6  25  4  21  10  1  24  12  4  16  
9  2  12  28  14  17  12  1  16  18  18  3  
12  6  6  12  10  20  9  6  8  6  8  15" | sed 's/\s\+/\n/g' | grep -v ^$ | sed 's/[[:space:]]*$//g' > test.txt

This would put each number on it's own line just fine, but when I pasted into statdisk, I was getting some invalid character errors. It APPEARED to be a space, but I tried using sed and tr with varations on [:space:], [:blank:], and things like s/[ \t]*$//g to try and remove whatever was trailing at the end (which I could delete just fine in Vim). But nothing on the CLI worked.

Running a hexdump on that file gives me

5   �   �  \n   8   �   �  \n   6   �   �  \n   2   5   �   �

What the heck are those? hexdump -C

35 c2 a0 0a 38 c2 a0 0a  36 c2 a0 0a 32 35 c2 a0  |5...8...6...25..|

Anyone have any idea what those c2 a0 bytes are? Is there an easy/elegant way to nuke them with sed or tr or something? Or being non-ascii would I need to do something clever with byte manipulation... Any thoughts/suggestions?

  • Are the strange characters in your OpenOffice output file? I remember Calc was pretty good at some number fiddling, but I don't know if it's got a sed interface/plugin – Xen2050 Nov 12 '15 at 4:10

c2a0 is UTF-8 U+00A0, NO-BREAK SPACE. You could easily use tr to remove them, e.g.,

> echo -n $'5\u00a0 8\u00a0 6' | tr -d $'\u00a0' | xxd
00000000: 3520 3820 36                             5 8 6

Note that the \u escape only works in Bash 4.2+, but of course you could replace with $'\xc2\xa0'.

  • What does the $'string' do? I know $(command) for command substitution, that replaces the depracated backtick, but I don't know the difference between $'\u00a0' and "\u00a0" – Arvandor Nov 12 '15 at 16:13
  • ANSI-C quoting. gnu.org/software/bash/manual/bash.html#ANSI_002dC-Quoting. – 4ae1e1 Nov 12 '15 at 18:55
  • Aha, thank you for that. It seems to me like many utilities do this automatically? Good to know that I can do it right in the shell though, for utilities that might not parse them out right. – Arvandor Nov 12 '15 at 20:23
  • I'm not sure what you mean by "many utilities do this automatically". What utilities are you talking about? Example? At least standard *ix utilities shouldn't treat U+00A0 differently than any other random codepoint. The POSIX blank character class in full unicode range might include this character, but that's as far as how special it is. – 4ae1e1 Nov 12 '15 at 20:26
  • By the way, does this answer solve your problem? If so, you may accept it; if it doesn't, say what you want exactly. – 4ae1e1 Nov 12 '15 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.