5

UNICODE uses 2 bytes for one character, so it has big or little endian difference. For example, the character 哈 is 54 C8 in hex. And its UTF-8 therefore is:

11100101 10010011 10001000

UTF-8 uses 3 bytes to present the same character, but it does not have big or little endian. Why?

6
  • 42
    To be fair, the term "Unicode" as an encoding is only used in Windows to refer specifically to UTF-16 Little Endian. Hence, "Unicode" doesn't use bytes or have endianness as it's not an encoding for the vast majority of the usage of that term. It might help to update the question to state "UTF-16" instead of "UNICODE" to be clearer for most readers, especially those who don't work in Windows enough to know how it misused the term "Unicode". – Solomon Rutzky May 13 at 17:23
  • Logically, though, UTF-8 is simply mandated by the standard to be "network byte order", aka big endian, on the wire. Indeed, other varieties of "prefix vli" encoding exist which are little endian on the wire. – oakad May 14 at 3:47
  • 5
    Also, please see Unicode's FAQ on "UTF-8, UTF-16, UTF-32 & BOM" as it should answer many questions you might have 😺. – Solomon Rutzky May 14 at 17:38
  • The field of Unicode has become so convoluted and complicated, it wouldn't surprise me if there were multiple reasons. – Mast May 15 at 14:10
  • @Mast There is only one reason: the concept of ordering cannot apply to single items (in this case: bytes). Please see my answer for the official explanation. – Solomon Rutzky May 16 at 19:06
35

Note: Windows uses the term "Unicode" for UCS-2 due to historical reasons – originally that was the only way to encode Unicode codepoints into bytes, so the distinction didn't matter. But in modern terminology, both examples are Unicode, but the first is specifically UCS-2 or UTF-16 and the second is UTF-8.

UCS-2 had big-endian and little-endian because it directly represented the codepoint as a 16-bit 'uint16_t' or 'short int' number, like in C and other programming languages. It's not so much an 'encoding' as a direct memory representation of the numeric values, and as an uint16_t can be either BE or LE on different machines, so is UCS-2. The later UTF-16 just inherited the same mess for compatibility.

(It probably could have been defined for a specific endianness, but I guess they felt it was out of scope or had to compromise between people representing different hardware manufacturers or something. I don't know the actual history.)

Meanwhile, UTF-8 is a variable-length encoding, which can use anywhere from 1 to 6 bytes to represent a 31-bit value. The byte representation has no relationship to the CPU architecture at all; instead there is a specific algorithm to encode a number into bytes, and vice versa. The algorithm always outputs or consumes bits in the same order no matter what CPU it is running on.

11
  • 18
    Hi there. Some notes: 1) "Unicode" in Windows world is specifically UTF-16 Little Endian given that they also have a "BigEndianUnicode" that is UTF-16 BE. 2) UTF-16 is also variable-length due to supplementary characters being comprised of surrogate pairs of two UTF-16 code units. And 3) For all practical purposes, UTF-8 uses a max of 4 bytes given that the Unicode Standard states explicitly that there is a hard limit of encoding just the 1,114,111 max code points defined by Unicode (i.e. the UTF-16 limit). – Solomon Rutzky May 13 at 18:42
  • 14
    UTF-16 is an encoding. It’s just not an encoding into bytes, but 16-bit words. Encoding those words into bytes is where endianness comes into play. – user3840170 May 14 at 5:09
  • 2
    Note that in recent Windows 10 versions (1903 to be exact), the "ASCII" Win32 API also accepts valid UTF-8 if you set up the process codepage correctly: docs.microsoft.com/en-us/windows/uwp/design/globalizing/…. – rubenvb May 14 at 7:29
  • 3
    @Solomon "Unicode" in Windows refers to UCS2 not UTF-16 most of the time (there are I think some newer APIs that are UTF-16 compliant, but the vast majority aren't). When talking about various or fixed length encoding in regards to Unicode people generally mean unicode codepoints and not glyphs.. after all you yourself state that UTF-8 is limited to 4 byte in the next sentence - if we were talking about glyphs the maximum size would be waaaayy longer. (In practice one should care about glyphs not codepoints though I agree which makes the fact that UTF-16 is fixed length rather irrelevant) – Voo May 14 at 9:16
  • 2
    @Voo "Unicode" in Windows is UTF-16 LE. A) "Save As..." dialogs, choosing "save with encoding", offers both Unicode (CP 1200) and Unicode (Big-Endian ; CP 1201) (see docs.microsoft.com/en-us/windows/win32/intl/…). B) UCS-2 hasn't been an encoding since Unicode 2.0 in 1996; it just indicates a process not interpreting surrogate pairs into their supplementary code point. Please see unicode.org/faq/utf_bom.html#utf16-11 and unicode.org/versions/Unicode13.0.0/appC.pdf#G1823 (section C.2 Encoding Forms in ISO/IEC 10646 on page 11). – Solomon Rutzky May 14 at 17:32
22

Exactly the same reason why an array of bytes (char[] in C or byte[] in many other languages) doesn't have any associated endianness but arrays of other types larger than byte do. It's because endianness is the way you store a value that's represented by multiple bytes into memory. If you have just a single byte then you only have 1 way to store it into memory. But if an int is comprised of 4 bytes with index 1 to 4 then you can store it in many different orders like [1, 2, 3, 4], [4, 3, 2, 1], [2, 1, 4, 3], [3, 1, 2, 4]... which is little endian, big endian, mixed endian...

Unicode has many different encodings called Unicode Transformation Format with the major ones being UTF-8, UTF-16 and UTF-32. UTF-16 and UTF-32 work on a unit of 16 and 32 bits respectively, and obviously when you store 2 or 4 bytes into byte-addressed memory you must define an order of the bytes to read/write. UTF-8 OTOH work on a byte unit, hence there's no endianness in it

5
  • 6
    Technically speaking, UTF-8 works on octets, not bytes. A byte is simply the smallest individually addressable unit of memory. On many popular architectures, a byte is 8 bits, but that is in no way guaranteed. While 6-bit, 7-bit, 9-bit, 10-bit, 11-bit, and 12-bit bytes are mostly a historic curiosity at this point, there are architectures in use today that have e.g. 16-bit, 24-bit, and 32-bit bytes, as well as architectures that have variable-sized bytes and architectures that don't have bytes at all (or have 1-bit bytes, however you want to look at it). – Jörg W Mittag May 13 at 19:30
  • 2
    And if you implementing unicode on a 9 bit architecture, then maybe datatracker.ietf.org/doc/html/rfc4042 could be relevant, but realistically you are probably going to just pretend you have 8 bit bytes with UTF-8 and ignore the extra bit. – user1937198 May 13 at 20:56
  • 1
    @user1937198: Gaaah! I have no beef with the existence of 36-bit architectures, but code for such platforms which needs to exchange data with other platforms should process it as a stream of octets. Having such systems use other formats for data that will be exchanged with other systems makes them less useful rather than more useful. – supercat May 15 at 1:54
  • 1
    @supercat note the date on that RFC – kbolino May 15 at 17:44
  • @kbolino: Heh. Most of the other Foolish April RFCs I've noticed put less effort into appearing serious. – supercat May 16 at 16:45
0

Here is the official, primary source material (published in March, 2020):

"The Unicode® Standard, Version 13.0"
Chapter 2: General Structure (page 39 of the document; page 32 of the PDF)

2.6 Encoding Schemes

The discussion of Unicode encoding forms (ed. UTF-8, UTF-16, and UTF-32) in the previous section was concerned with the machine representation of Unicode code units. Each code unit is represented in a computer simply as a numeric data type; just as for other numeric types, the exact way the bits are laid out internally is irrelevant to most processing. However, interchange of textual data, particularly between computers of different architectural types, requires consideration of the exact ordering of the bits and bytes involved in numeric representation. Integral data, including character data, is serialized for open interchange into well-defined sequences of bytes. This process of byte serialization allows all applications to correctly interpret exchanged data and to accurately reconstruct numeric values (and thereby character values) from it. In the Unicode Standard, the specifications of the distinct types of byte serializations to be used with Unicode data are known as Unicode encoding schemes.

Byte Order. Modern computer architectures differ in ordering in terms of whether the most significant byte or the least significant byte of a large numeric data type comes first in internal representation. These sequences are known as “big-endian” and “little-endian” orders, respectively. For the Unicode 16- and 32-bit encoding forms (UTF-16 and UTF32), the specification of a byte serialization must take into account the big-endian or little-endian architecture of the system on which the data is represented, so that when the data is byte serialized for interchange it will be well defined.

A character encoding scheme consists of a specified character encoding form plus a specification of how the code units are serialized into bytes. The Unicode Standard also specifies the use of an initial byte order mark (BOM) to explicitly differentiate big-endian or little-endian data in some of the Unicode encoding schemes. (See the “Byte Order Mark” subsection in Section 23.8, Specials.)

When a higher-level protocol supplies mechanisms for handling the endianness of integral data types, it is not necessary to use Unicode encoding schemes or the byte order mark. In those cases Unicode text is simply a sequence of integral data types.

For UTF-8, the encoding scheme consists merely of the UTF-8 code units (= bytes) in sequence. Hence, there is no issue of big- versus little-endian byte order for data represented in UTF-8. However, for 16-bit and 32-bit encoding forms, byte serialization must break up the code units into two or four bytes, respectively, and the order of those bytes must be clearly defined. Because of this, and because of the rules for the use of the byte order mark, the three encoding forms of the Unicode Standard result in a total of seven Unicode encoding schemes, as shown in Table 2-4.

The endian order entry for UTF-8 in Table 2-4 is marked N/A because UTF-8 code units are 8 bits in size, and the usual machine issues of endian order for larger code units do not apply. The serialized order of the bytes must not depart from the order defined by the UTF-8 encoding form. Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature. See the “Byte Order Mark” subsection in Section 23.8, Specials, for more information.

Please also see the following related information:

-1

The reason is very simple. There are big and little endian versions of UTF-16 and UTF-32 because there are computers with bit and little endian registers. If the endianness of a Unicode file matches the endianness of the processor the character value can be read directly from memory in a single operation. If they do not match, a second conversion step is required to flip the value around.

In contrast the endianness of the processor is irrelevant when reading UTF-8. The program must read the individual bytes and perform a series of tests and bit shifts to get the character value into a register. Having a version where the byte order was reversed would be pointless.

3
  • 4
    It's not really about the registers (though of course it affects which endianness you choose), it's just the fact that in UTF-16, the smallest unit is 16 bits, in UTF-32 it's 32 bits and in UTF-8 it's 8 bits. We've managed to almost perfectly solve how to store 8-bit numbers in a multi-platform way, but the war over 16 and 32 bit numbers is still ongoing :) Or said another, way, UTF-8 is a stream of bytes - it already has a fixed order. UTF-16 is a stream of 16-bit numbers, again ordered; but they way we actually store the individual 16-bit numbers still isn't uniform. – Luaan May 15 at 16:18
  • 2
    Registers don't have an endianness at all. By the time data arrives there, endianness is done for. The whole processor probably has a slight or strong preference for little- or big-endian though, as avoiding it is not economic. Some can be configured (at least for integral numbers), floating point can be different, instructions can be different, and then there are some weird ones which are neither. – Deduplicator May 15 at 16:18
  • @Deduplicator I didn't say registers have endianness. The endianness actually resides in the machine language instructions which copy words between registers and RAM. For example, x86 processors can load a 16 and 32 bit little-endian word into a register in a single instruction, but several instructions are required to load a bit-endian word from RAM. – David42 May 16 at 12:42
-3

According to some Windows documentation, the encoding maps to a stream of up to 4 bytes. Also it is says it does not matter the processor endianness. So what i think this means to the developer is that you aren't supposed to worry about endianess with utf-8 on Windows. That is the design philosophy. So you should now focus on how appropriately you should use the windows functionality so that it does not matter. Now streams coming in would matter, but the decoding and encoding of utf-8 you should be able to not have to deal with.

However it is possible to go underneath this, to fully understand, which may help. But basically Windows says you don't need to know the endianess of the system to deal with utf-8 for encoding and decoding streams to utf-8.

12
  • I looked up this on msdn if that helps. – marshal craft May 14 at 12:52
  • 5
    Hello Marshal. You mention documentation, so please provide links to exactly what you read so that others may review. Also, endianness is very much not Windows-specific. Endianness is the ordering of bytes within multi-byte datatypes (e.g. 16, 32, and 64 bit ints). A single byte does not have order; it just is. Since UTF-8 uses 8 bit code units, the concept of endianness can't apply. UTF-8 is multiple, single-byte units whereas UTF-32 is a single, multi-byte unit. UTF-16 is multiple, multi-byte units. Please see phuclv's answer (correct) answer. – Solomon Rutzky May 14 at 13:47
  • 6
    "you aren't supposed to worry about endianess with utf-8 on Windows" - the "on Windows" part is redundant, you don't need to worry about UTF-8 endianness no matter which OS you encounter it on. That's one of its beauties. – Mark Ransom May 14 at 21:36
  • I expected this. It's funny how hateful you are to things which are more sophisticated then you, even when you immediately steal from it. – marshal craft May 16 at 9:57
  • I posted a Linux perspective answer here one time, and then disproportionate amount of up votes. This answer is perfectly fine. If I read a Linux system specific answer that's fine. This answer is great for a Windows developer using utf-8. I do not need to post document, it's easily found. Don't bother commenting further, with the same lines of argument cause i do not care, block ban what ever but just save your self the time maybe or don't. – marshal craft May 16 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.