Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to process the following text to get rid of the strange symbols such as:

â<80><99> â<80><9c> â<80>?

Example text:

With the mystery unexplained, the Hyatt tried to give its guests a sense of security by posting a guard in its lobby. But Wolf couldnâ<80><99>t shake the notion that a thief could re-enter her room at any time. â<80><9c>I had dreams about it for many nights,â<80>?says Wolf, a 66-year-old Dell IT services consultant traveling in Houston for business.

Can anyone help me with it? I hope to either manually delete it with some command in Vi or do it with script.

share|improve this question
    
Looks like you're editing a UTF-8 file in a vi that doesn't understand UTF-8, try using vim instead. –  mu is too short Dec 24 '12 at 6:59
    
You may need to change the language support to UTF-8 available in Window Preferences of application you are using. –  Neeraj T Dec 24 '12 at 7:01
    
@muistooshort vi became vim at least 25 years ago. –  Shiplu Dec 25 '12 at 5:20
    
@Shiplu vim was only publicly released 21 years ago. Some OSes still use vi as default (IIRC, this includes FreeBSD). It's a valid suggestion. –  Bob Dec 25 '12 at 9:22
add comment

migrated from stackoverflow.com Dec 25 '12 at 5:20

This question came from our site for professional and enthusiast programmers.

2 Answers

I found the text in question here: http://www.forbes.com/sites/andygreenberg/2012/11/26/security-flaw-in-common-keycard-locks-exploited-in-string-of-hotel-room-break-ins/

The characters causing the problem are fancy quotes and an apostrophe, that are not the standard ascii symbols for quotes and apostrophe.

I pasted that text into my copy of vim, and it handled those characters just fine.

But here's how to do replaces when this kind of thing happens: http://aditya.sublucid.com/2008/01/18/replacing-those-pesky-smart-quotes-in-vim/

share|improve this answer
    
Many thanks! It works and is just what I need. I didn't intend to inspire any interest in the text content though. Lesson learned :) –  querystack Dec 24 '12 at 8:49
add comment

Mostly this issue will occur if you transfer your file from Windows or DOS machines. To get rid of those unwanted special character use "dos2unix" utility

mkannan@talksense-dr:~/tmp$ dos2unix test.sh 
dos2unix: converting file test.sh to UNIX format ...
share|improve this answer
    
dos2unix converts line endings (from CR + LF to just LF). This looks more like a problem with the encoding of quotes. (@querystack confirmed in a comment on the other answer that it is an issue with 'smart quotes', which are not part of the ASCII character set.) –  Bob Dec 25 '12 at 9:23
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.