Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

When I try to save a text file with non-English text in Notepad, I get an option to choose between Unicode, Unicode Big Endian and UTF-8. What is the difference between these formats?

Assuming I do not want any backward compatibility (with older OS versions or apps) and I do not care about the file size, which of these formats is better?

(Assume that the text can be in languages like Chinese or Japanese, in addition to other languages.)

Note: From the answers and comments below it seems that in Notepad lingo, Unicode is UTF-16 (Little Endian), Unicode Big Endian is UTF-16 (Big Endian) and UTF-8 is well UTF-8.

share|improve this question

5 Answers 5

up vote 14 down vote accepted

Dunno. Which is better: a saw or a hammer? :-)

Unicode isn't UTF

There's a bit in the article that's a bit more relevant to the subject at hand though:

  • UTF-8 focuses on minimizing the byte size for representation of characters from the ASCII set (variable length representation: each character is represented on 1 to 4 bytes, and ASCII characters all fit on 1 byte). As Joel puts it:

“Look at all those zeros!” they said, since they were Americans and they were looking at English text which rarely used code points above U+00FF. Also they were liberal hippies in California who wanted to conserve (sneer). If they were Texans they wouldn’t have minded guzzling twice the number of bytes. But those Californian wimps couldn’t bear the idea of doubling the amount of storage it took for strings

  • UTF-32 focuses on exhaustiveness and fixed-length representation, using 4 bytes for all characters. It’s the most straightforward translation, mapping directly the Unicode code-point to 4 bytes. Obviously, it’s not very size-efficient.

  • UTF-16 is a compromise, using 2 bytes most of the time, but expanding to 2 * 2 bytes per character to represent certain characters, those not included in the Basic Multilingual Plane (BMP).

Also see The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

share|improve this answer
4  
The problem comes from the fact that Unicode is an 'encoding', but not in the numbers-into-bytes sense. UTF-8/16/32 are all Unicode encodings, but Unicode itself is a mapping from symbols to numbers. They could have used more unique terminology to avoid this confusion I think. –  jerryjvl Jul 22 '09 at 3:12
4  
Regardless though, to the OP of the question, odds are that the application means 'UTF-16' where it says 'Unicode'. –  jerryjvl Jul 22 '09 at 3:13
2  
I'm not sure that UTF-8's goal is "conservation" as opposed to backwards-compatibility with ASCII. –  Mr. Shiny and New 安宇 Sep 2 '09 at 17:36
    
@Johannes: The Unicode Consortium has decided never to assign code points above U+10FFFF because they cannot be represented in UTF-16. This had the effect of restricting UTF-8 to 4 bytes. –  user46971 Aug 21 '10 at 17:47
1  
"Unicode isn't UTF" -- for many, it is WTF ;) –  mlvljr Jun 7 '12 at 12:14

"Unicode" is another term for "UTF-16", which is an encoding of the Unicode character set into sixteen-bits per character. UTF-8 encodes it into eight bits per character.

In both cases, any overflow is allocated to another 16 or eight bits.

share|improve this answer
    
Which one is better then? –  R. Martinho Fernandes Jul 22 '09 at 3:08
    
"it depends" on the situation. –  John Saunders Jul 22 '09 at 6:37
    
Though for this specific question it seems that "Unicode" is indeed ABUSED as another term for "UTF-16", it's not so in general -- see Jason's answer. –  Arjan Oct 23 '09 at 14:31
1  
You mean "per code unit", not "per character"; both UTF-8 and UTF-16 can use multiple code units to represent a character. And "Unicode" an "UTF-16" are NOT the same thing, except in Microsoft terminology. –  user46971 Aug 21 '10 at 17:48

The only real advantage with small files like text files is the resulting file size. UTF-8 generally produces smaller files. But this difference may be less pronounced with Chinese/Japanese text.

share|improve this answer
    
Bear in mind that there's also a difference in network bandwidth and memory usage. –  Jason Baker Jul 22 '09 at 3:18
1  
"UTF-8 generally produces smaller files": Not generally. UTF-8 produces smaller files for ASCII files. If a file only consists of Unicode codepoints above U+0800, it will be larger in UTF-8 than in UTF-16. –  sleske Nov 17 '09 at 12:09

For European languages, UTF-8 is smaller. For Oriental languages, the difference is not so clear-cut.

Both will handle all possible Unicode characters, so it should make no difference in compatibility.

share|improve this answer

There are more Unicode character encodings than you may think.

  • UTF 8

    The UTF-8 encoding is variable-width, ranging from 1-4 bytes, with the upper bits of each byte reserved as control bits. The leading bits of the first byte indicate the total number of bytes used for that character. The scalar value of a character's code point is the concatenation of the non-control bits. In this table, x represents the lowest 8 bits of the Unicode value, y represents the next higher 8 bits, and z represents the bits higher than that.

    Unicode              Byte1     Byte2     Byte3     Byte4
    U+0000-U+007F       0xxxxxxx    		
    U+0080-U+07FF       110yyyxx  10xxxxxx  		
    U+0800-U+FFFF       1110yyyy  10yyyyxx  10xxxxxx    
    U+10000-U+10FFFF    11110zzz  10zzyyyy  10yyyyxx  10xxxxxx
    
  • UCS-16
  • UCS-16BE
  • UCS-16LE

  • UTF-16
  • UTF-16BE
  • UTF-16LE

  • UTF-32
  • UTF-32-BE
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.