Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a text encoded in ANSI:

ANSI text file

When I tried to convert it into UTF-8 (using the Notepad++ menu Encoding > UTF-8), I get some weird characters:

UTF-8 text file

I thought that UTF-8 was a superset of ANSI and that I subsequently wouldn't have such issues. Is there anyway I can avoid the apparition of those weird characters?

share|improve this question
    
I don't see what's weird in the latter image –  Lưu Vĩnh Phúc Jun 2 at 2:01
    
I meant the non-displayable characters. Any way to avoid this? –  user3658425 Jun 2 at 2:09
    
Did you edited the question? I think I see the images reversed –  Lưu Vĩnh Phúc Jun 2 at 2:10
1  
Yes sorry I pushed my last edit right after somebody added the images, hereby erasing the addition: if you can add them back that would be great! –  user3658425 Jun 2 at 2:12
2  

1 Answer 1

up vote 12 down vote accepted

UTF-8 is not a charset, just an encoding for Unicode. The first 128 characters are the same as ASCII but differ in the high 128 bytes. A byte with the high bit set (or >= 0x80) is an extended character in ASCII while in UTF-8 it indicates the start byte of a multi-byte sequence. That's the case of 0x93 or 0x94 above. However, I don't see anything strange in the file. Those are smart quotes or quotes with different forms for opening and closin quotes, which you often see when using a rich text editor such as MS Word

Edit

The question has edited. I think that's because you have chosen the wrong tool. The encode menu items are for changing the encoding if you have wrong character displays. It just treats the same byte sequence read from disk as another encoding. Since ASCII and UTF-8 are different, you'll have an illformed UTF-8 byte sequence and see the result above. You need to choose convert to UTF-8 for it to change the whole input bytes

notepad encoding

You also have confused ANSI and ASCII. ANSI often refers to Windows-1252, which is a character set used in English Windows and some Western Europe languages. It's a superset of ISO 8859-1, although ISO 8859-1 may also be refered to as ANSI. ISO 8859-1 is also the first 256 codepoints of Unicode, so it's a subset of Unicode, but it's not compatible with UTF-8 encoding. ASCII is a 7-bit character set and is a subset of the ANSI which is encoded by 8 bits, but it's also sometimes refered to as ANSI, although not very correct

https://en.wikipedia.org/wiki/ANSI_character_set

In general the relationship between character sets is as follow

ASCII < ISO 8859-1 < Windows-1252
        ^
        Unicode
share|improve this answer
    
Thanks, sorry for messing up charset with encoding. Is there any way to avoid having those non displayable characters and have some displayable quotes instead? –  user3658425 Jun 2 at 2:17
    
@user3658425 please see my edit –  Lưu Vĩnh Phúc Jun 2 at 2:20
1  
I have also edited the ANSI/ASCII information above –  Lưu Vĩnh Phúc Jun 2 at 2:41
2  
Note that, in general, "UTF-8 without Byte Order Mark" is what you want. As several others have said in the past, "UTF-8 is the only useful string encoding." –  Riking Jun 2 at 6:23
1  
@WillihamTotland utf8everywhere.org –  Lưu Vĩnh Phúc Jun 2 at 8:16

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.