Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

So I use plain text for pretty much all information I keep. The files are (or will be) used with shell scripts, emacs, vim, terminal etc. on OSX and Linux. It would be ideal to be able to use English, Chinese and Scandinavian characters with a minimum of headache. Assuming performance is not an issue but portability and interoptability is, would utf-8 or utf-16 be the best alternative for encoding? If no option seems clearly better, what are the relevant tradeoffs?

share|improve this question
What's your question again? – Daniel Beck Feb 22 '12 at 21:00
added the question from the headline to the text – Tor Thommesen Feb 22 '12 at 21:51

Summary: UTF-8 preferred.

The differences between UTF-8 and UTF-16 are few. Both can encode any codepoint of the 1,112,064 supported by Unicode, and it is trivial to convert from one to the other. The major difference is support by programs.

While almost all text editors support both encodings, UTF-8 is preferred due to its compatibility with ASCII in the first 128 bytes – in other words, #!/usr/bin/env bash will be the same in both; this means that writing shell-scripts in UTF-8 can be done with no modification at all to the operating system or to the various interpreters. (Be sure to disable the "byte order mark" function in UTF-8, though.)

On Linux, where most programs use the glibc locale to decide between character sets, UTF-8 is the only choice (beside the legacy charsets) and UTF-16 is not supported at all.

One minor difference is the space required for storage. UTF-8 is variable-length, using one to four bytes, while UTF-16 uses two-byte units. If the text uses a mostly-Latin alphabet with just the occasional Scandinavian character, then UTF-16 will use twice as much space as UTF-8, since the latter can represent Latin characters as single ASCII bytes, with the occasional two- or three-byte sequences. On the other hand, if the text is mostly Chinese, UTF-8 would require three-byte sequences to represent every character, resulting in files 33% larger than UTF-16. However, for text files, this is very much insignificant given terabyte-sized disks.

However, the use of two-byte "code units" in UTF-16 is also a disadvantage: the encoding requires support for both "big-endian" and "little-endian" byte orderings; both 54 00 6f 00 72 00 and 00 54 00 6f 00 72 are equivalent. This means that programs need to support both, and try to guess which byte-order is used in a given file. 54 00 might mean both U+0054 or U+5400, so including the BOM – byte order mark – is often a necessity (ff fe can only mean U+FEFF, never U+FFFE). If a single byte is lost, the rest of a document becomes unsynchronized. UTF-8 avoids all those problems.

In the end, though, conversion between Unicode encodins is cheap: iconv -f utf16 -t utf8 is all you need.

See also UTF-8 – compared to UTF-16 on Wikipedia, or the original UTF-8 document from Bell Labs.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .