1

I have a text file (actually a .csv exported from .xlsx in LibreOffice) with lines that have gibberish when opened in LibreOffice Calc. In VIM they appear as the following:

Joe User, joe@user.com, p@ss<96>w0rd

The <96> was some funny character that was similar to -, but MS Office decided to change it to something else. The file is therefore not ASCII:

$ file -i Users.csv
Users.csv: text/plain; charset=unknown-8bit

Note that all of these 'funny' characters have codepoints less than 127, so I really don't know how Calc and VIM realize that this is not ASCII. What other encodings have all the code points below 127, i.e. are not ASCII compatible? Also, how can I search for these characters in VIM? For non-ASCII characters above 127 I use /[^\x00-\x127] to search, but of course that will not find <96>.

The original .xlsx file is still found on another user's Windows computer, and sure enough I can verify in his MS Office that the <96> character is not ASCII 96 ` but rather some funny variation on '.

2

<96> is not ASCII 96: it is the representation of UTF-8 U+0096 (part of latin 1 extended), "start of protected area", whatever that means.

You can hit ga to see information on the character under the cursor. For <96>, you'd get the following:

<<96>> 150, Hex 0096, Octal 226

You can remove that character from your buffer with:

:%s/<C-v>u0096//g

which should look like:

:%s/<96>//g

Or you can do:

:%s/\%u0096//g    " hex form

or:

:%s/\%d150//g     " decimal form

or:

:%/\%o226//g      " octal form

See :help ga, :help utf-8-typing, :help \%u.

  • Thank you! I thought that <96> is decimal, I did not realize that it is hexadecimal, and therefore above \x7F (127). Thank you! – dotancohen May 4 '14 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.