Unicode filenames in Windows vs. Mac OS X

Question

I was trying to use a couple of Windows file syncing programs (Microsoft SyncToy, FreeFileSync) to sync between two folders: a local Windows folder and a OS X folder mounted on the Windows machine using WebDAV. However, the sync didn't work too well: whenever a filename used unusual characters (accents, chinese letters...), the program didn't detect it as the same file, and tried to copy it two ways: first from the Windows box to the OS X one, and then the other way. Basically, it treated the 2 copies of the file as completely different.

This got me thinking: what exactly are the variants between the Unicode used for filenames in OS X and Windows? (I suppose that they all support Unicode by now). What can one do to prevent incompatibilities of this type?

The two machines are using Windows 7 SP 1 and OS X 10.9.5.

Answer 1

Windows uses UTF-16. Most codepoints are encoded in two bytes. Codepoints outside the BMP (Basic Multilingual Plane) are encoding in a "surrogate pair" that takes four bytes. Windows does not normalize filenames using any Unicode Normalization Form.

This means you could have two filenames that look identical with one using a precomposed "é" consisting of a single codepoint and the other using a regular ASCII "e" followed by a Unicode combining acute accent, thus two codepoints.

OS X uses UTF-8. Codepoints are encoded using between one and five bytes. OS X uses Unicode NFD (Normalization Form Canonical Decomposition).

This means that when a Unicode character such as "é" is used in a filename it will always be normalized by the system into a regular ASCII "e" followed by a Unicode combining acute accent, and will always take two codepoints.

In fact OS X uses the spec of Unicode NFD from either Unicode version 2.1 or 3.2, depending on the version of OS X.

Here's a nice page that covers the subtleties in the filename encoding of OS X's / HFS+.