So I use plain text for pretty much all information I keep. The files are (or will be) used with shell scripts, emacs, vim, terminal etc. on OSX and Linux. It would be ideal to be able to use English, Chinese and Scandinavian characters with a minimum of headache. Assuming performance is not an issue but portability and interoptability is, would utf-8 or utf-16 be the best alternative for encoding? If no option seems clearly better, what are the relevant tradeoffs?
Summary: UTF-8 preferred.
The differences between UTF-8 and UTF-16 are few. Both can encode any codepoint of the 1,112,064 supported by Unicode, and it is trivial to convert from one to the other. The major difference is support by programs.
While almost all text editors support both encodings, UTF-8 is preferred due to its compatibility with ASCII in the first 128 bytes – in other words,
On Linux, where most programs use the glibc locale to decide between character sets, UTF-8 is the only choice (beside the legacy charsets) and UTF-16 is not supported at all.
One minor difference is the space required for storage. UTF-8 is variable-length, using one to four bytes, while UTF-16 uses two-byte units. If the text uses a mostly-Latin alphabet with just the occasional Scandinavian character, then UTF-16 will use twice as much space as UTF-8, since the latter can represent Latin characters as single ASCII bytes, with the occasional two- or three-byte sequences. On the other hand, if the text is mostly Chinese, UTF-8 would require three-byte sequences to represent every character, resulting in files 33% larger than UTF-16. However, for text files, this is very much insignificant given terabyte-sized disks.
However, the use of two-byte "code units" in UTF-16 is also a disadvantage: the encoding requires support for both "big-endian" and "little-endian" byte orderings; both
In the end, though, conversion between Unicode encodins is cheap: