Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I need to process the following text to get rid of the strange symbols such as:

â<80><99> â<80><9c> â<80>?

Example text:

With the mystery unexplained, the Hyatt tried to give its guests a sense of security by posting a guard in its lobby. But Wolf couldnâ<80><99>t shake the notion that a thief could re-enter her room at any time. â<80><9c>I had dreams about it for many nights,â<80>?says Wolf, a 66-year-old Dell IT services consultant traveling in Houston for business.

Can anyone help me with it? I hope to either manually delete it with some command in Vi or do it with script.

share|improve this question

migrated from Dec 25 '12 at 5:20

This question came from our site for professional and enthusiast programmers.

Looks like you're editing a UTF-8 file in a vi that doesn't understand UTF-8, try using vim instead. – mu is too short Dec 24 '12 at 6:59
You may need to change the language support to UTF-8 available in Window Preferences of application you are using. – Neeraj T Dec 24 '12 at 7:01
@muistooshort vi became vim at least 25 years ago. – Shiplu Dec 25 '12 at 5:20
@Shiplu vim was only publicly released 21 years ago. Some OSes still use vi as default (IIRC, this includes FreeBSD). It's a valid suggestion. – Bob Dec 25 '12 at 9:22

I found the text in question here:

The characters causing the problem are fancy quotes and an apostrophe, that are not the standard ascii symbols for quotes and apostrophe.

I pasted that text into my copy of vim, and it handled those characters just fine.

But here's how to do replaces when this kind of thing happens:

share|improve this answer
Many thanks! It works and is just what I need. I didn't intend to inspire any interest in the text content though. Lesson learned :) – querystack Dec 24 '12 at 8:49

Mostly this issue will occur if you transfer your file from Windows or DOS machines. To get rid of those unwanted special character use "dos2unix" utility

mkannan@talksense-dr:~/tmp$ dos2unix 
dos2unix: converting file to UNIX format ...
share|improve this answer
dos2unix converts line endings (from CR + LF to just LF). This looks more like a problem with the encoding of quotes. (@querystack confirmed in a comment on the other answer that it is an issue with 'smart quotes', which are not part of the ASCII character set.) – Bob Dec 25 '12 at 9:23

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.