1

I was trying to use a couple of Windows file syncing programs (Microsoft SyncToy, FreeFileSync) to sync between two folders: a local Windows folder and a OS X folder mounted on the Windows machine using WebDAV. However, the sync didn't work too well: whenever a filename used unusual characters (accents, chinese letters...), the program didn't detect it as the same file, and tried to copy it two ways: first from the Windows box to the OS X one, and then the other way. Basically, it treated the 2 copies of the file as completely different.

This got me thinking: what exactly are the variants between the Unicode used for filenames in OS X and Windows? (I suppose that they all support Unicode by now). What can one do to prevent incompatibilities of this type?

The two machines are using Windows 7 SP 1 and OS X 10.9.5.

2

Windows uses UTF-16. Most codepoints are encoded in two bytes. Codepoints outside the BMP (Basic Multilingual Plane) are encoding in a "surrogate pair" that takes four bytes. Windows does not normalize filenames using any Unicode Normalization Form.

This means you could have two filenames that look identical with one using a precomposed "é" consisting of a single codepoint and the other using a regular ASCII "e" followed by a Unicode combining acute accent, thus two codepoints.

OS X uses UTF-8. Codepoints are encoded using between one and five bytes. OS X uses Unicode NFD (Normalization Form Canonical Decomposition).

This means that when a Unicode character such as "é" is used in a filename it will always be normalized by the system into a regular ASCII "e" followed by a Unicode combining acute accent, and will always take two codepoints.

In fact OS X uses the spec of Unicode NFD from either Unicode version 2.1 or 3.2, depending on the version of OS X.

Here's a nice page that covers the subtleties in the filename encoding of OS X's / HFS+.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.