I have a text encoded in ANSI:

ANSI text file

When I tried to convert it into UTF-8 (using the Notepad++ menu Encoding > UTF-8), I get some weird characters:

UTF-8 text file

I thought that UTF-8 was a superset of ANSI and that I subsequently wouldn't have such issues. Is there anyway I can avoid the apparition of those weird characters?

up vote 35 down vote accepted

UTF-8 is not a charset, just an encoding for Unicode. The first 128 characters are the same as ASCII but differ in the high 128 bytes. A byte with the high bit set (or >= 0x80) is an extended character in ASCII while in UTF-8 it indicates the start byte of a multi-byte sequence. That's the case of 0x93 or 0x94 above. However, I don't see anything strange in the file. Those are smart quotes or quotes with different forms for opening and closin quotes, which you often see when using a rich text editor such as MS Word

Edit

The question has edited. I think that's because you have chosen the wrong tool. The encode menu items are for changing the encoding if you have wrong character displays. It just treats the same byte sequence read from disk as another encoding. Since ASCII and UTF-8 are different, you'll have an illformed UTF-8 byte sequence and see the result above. You need to choose convert to UTF-8 for it to change the whole input bytes

notepad encoding

You also have confused ANSI and ASCII. ANSI often refers to Windows-1252, which is a character set used in English Windows and some Western Europe languages. It's a superset of ISO 8859-1, although ISO 8859-1 may also be refered to as ANSI. ISO 8859-1 is also the first 256 codepoints of Unicode, so it's a subset of Unicode, but it's not compatible with UTF-8 encoding. ASCII is a 7-bit character set and is a subset of the ANSI which is encoded by 8 bits, but it's also sometimes refered to as ANSI, although not very correct

https://en.wikipedia.org/wiki/ANSI_character_set

In general the relationship between character sets is as follow

ASCII < ISO 8859-1 < Windows-1252
        ^
        Unicode
  • Thanks, sorry for messing up charset with encoding. Is there any way to avoid having those non displayable characters and have some displayable quotes instead? – user3658425 Jun 2 '14 at 2:17
  • @user3658425 please see my edit – phuclv Jun 2 '14 at 2:20
  • 1
    I have also edited the ANSI/ASCII information above – phuclv Jun 2 '14 at 2:41
  • 3
    Note that, in general, "UTF-8 without Byte Order Mark" is what you want. As several others have said in the past, "UTF-8 is the only useful string encoding." – Riking Jun 2 '14 at 6:23
  • 3
    @WillihamTotland utf8everywhere.org – phuclv Jun 2 '14 at 8:16

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.