Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm learning about UTF-8 standards and this is what I'm learning :

Definition and bytes used
UTF-8 binary representation         Meaning
0xxxxxxx                            1 byte for 1 à 7 bits chars
110xxxxx 10xxxxxx                   2 bytes for 8 à 11 bits chars
1110xxxx 10xxxxxx 10xxxxxx          3 bytes for 12 à 16 bits chars
11110xxx 10xxxxxx 10xxxxxx 10xxxxxx 4 bytes for 17 à 21 bits chars

And I'm wondering, why 2 bytes UTF-8 code is not 10xxxxxx instead, thus gaining 1 bit all the way up to 22 bits with a 4 bytes UTF-8 code? The way it is right now, 64 possible values are lost (from 1000000 to 10111111). I'm not trying to argue the standards, but I'm wondering why this is so?

** EDIT **

Even, why isn't it

UTF-8 binary representation         Meaning
0xxxxxxx                            1 byte for 1 à 7 bits chars
110xxxxx xxxxxxxx                   2 bytes for 8 à 13 bits chars
1110xxxx xxxxxxxx xxxxxxxx          3 bytes for 14 à 20 bits chars
11110xxx xxxxxxxx xxxxxxxx xxxxxxxx 4 bytes for 21 à 27 bits chars

...?

Thanks!

share|improve this question
add comment

1 Answer

up vote 8 down vote accepted

UTF-8 is self-synchronising. Something examining the bytes can tell if it's at the start of a UTF-8 character, or part-way through one.

Let's say you have two characters in your scheme: 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx

If the parser picks up at the second octet, it can't tell that it's not to read the second and third octets as one character. With UTF-8, the parser can tell that it's in the middle of a character and continue ahead to the start of the next one, while emitting some state to mention the corrupted symbol.

For the edit: if the top bit is clear, UTF-8 parsers know that they're looking at a character represented in one octet. If it is set, it's a multi-octet character.

It's all about error recovery and easy classification of octets.

share|improve this answer
    
alright. it makes sense. Thank you. –  Yanick Rochon Jan 13 '11 at 13:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.