Trouble viewing files with non-English names on hard-disk

Question

I have a problem that I don't even know where to begin looking for help (google isn't helping)

I have an external hard drive formatted as Mac OS Extended (Journaled). I had been using it with my Macbook OS X for a long time. There are many files on this hard drive that are titled with Korean characters.

Recently I got a new computer and installed Xubuntu 11.04 on it, along with Korean language support. Using the GUI I can mount the external hard drive and see only those files that aren't titled in Korean. The ones titled in Korean don't even show up. Now if I go into the hard drive via the command line, and ls the drive, I do get a listing of the Korean files with their Korean names, but only after an error message for each one of those files that goes like "ls: cannot access : No such file or directory".

Now back on my Mac, if I rename the korean file to an english name, the file can be seen no problem by my linux machine. Also, if I then rename the file to a korean name on my LINUX machine (remember, my linux machine also has korean support), then the linux machine has no problem recognizing or reading it, but now the MAC will not be able to recognize the file. That is, if I go in with my Mac now and try to open the file via the gui, the file (really actually a directory) is empty. If I try to cd into the file via the command line it says that it gives me a "no such file or directory" error. But doing an ls gives no errors.

In short, it seems that if I try to name a file with a foreign-language on my mac, my linux won't recognize it, and if I try to do it with my linux, my mac won't recognize it. The only way for the two computers to both be able to use the external hard drive seems to be if all the files are named in English. I want to avoid renaming my hundreds of Korean files to English names. Is there anyone who knows what might be going on here, or point me to a direction which might help me tackle this?

Thank you! Bryan

Answer 1

Encoding of filenames?

Does perl see the Korean filenames?

perl -e 'opendir(D, ".");while($f=readdir(D)){print "--$f--\n";}'

If it does show the filenames (or some unrecognisably mangled version of them), you may be able to work out if it is an encoding issue. Korean encodings include KS X 1001, EUC-KR and ISO-2022-KR. However I believe both OSX and Linux normally use UTF-8.

See this answer which says

OS X's HFS+ filesystem requires that all filenames be stored in the UTF-8 representation of their fully decomposed form.

There are tools for converting filename encodings but you need to persuade both operating systems to use a common encoding.

See this question describing Linux filename encodings.

I have run convmv -f UTF-8 -t UTF-8 on the directory, and discovered these 500 filenames are not encoded in UTF-8 (convmv is able to detect and ignore filenames already in UTF-8)

Is there an easy way I can find out which language encoding they are currently using?

Answer 2

Although old question it's useful to know that even with Unicode you can have built in problems when transferring data between regions.

This is an encoding issue due to Unicode having two different ways to form a character (same thing happens with European symbols with umlauts etc). OSX appears to combine 3 glyphs to form the symbol and Linux/Ubuntu can use both. in Linux try
$ echo -e "\u1112\u1161\u11AB \uD55C"

$ 한 한

they visually appear to be the same but they represent different characters to the computer- as confirmed by hex dump

$ echo -e "\u1112\u1161\u11AB \uD55C" | xxd

$ 0000000: e184 92e1 85a1 e186 ab20 2020 20ed 959c

$ 0000010: 0a

OSX probably uses 9bytes/3glyps per symbol and your Linux setup/keyboard uses 3byes/1 glyph {\xed\x95\x9c)

Your Keyboard Korean setup is probably set up to have one glyph/character and you only think you are typing the same symbol but it is NOT the same as the OSX keyboard.

You may have to remap your keyboard to ensure it outputs same character set as OSX.