As we know, Linux and Unix style filesystems store file name as byte stream, as apposed to Unicode streams on Windows NTFS/FAT.

So there is the question. If I get an old hard disk(from around 2002) with non-ASCII filenames encoded as "gbk" on its Ext3 partitions, and I want to attach that old disk to my modern Linux(utf8-encoded filenames) so that I can copy the old files out, preserving filename character meaning .

Then what's the best practice to do that?

Explaination from a programmers perspective is specially welcome.

  • I have an old laptop that I don't use anymore, but it has Linux and Windows XP and when Linux is running it can read the Windows partition. Why not just plug it in and see if it works? Apr 27 '13 at 1:40
  • Thank you. I believe it can, because Windows explicitly/definitly stores Unicode character on FS so there is no ambiguation. I just want to know how guys in pure Linux world can cope with that problem.
    – Jimm Chen
    Apr 27 '13 at 2:56

No. Most Linux programs don't care about disk or mountpoint boundaries, and use the same encoding for all file names everywhere.

However, most utilities still treat filenames as byte strings, so there should be no problem with copying or using files named in weird encodings. (For example, I renamed a few my files to ISO-8859-13 and I can still open them in command-line or GNOME programs, although KDE has troubles.)

You can use tools such as convmv to rename all files to UTF-8 names – either in your copy, or directly on the original disk:

convmv -f gbk -t utf-8 -r /mnt/old-disk

(On Linux, it is possible to write overlay filesystems using FUSE, which could present the files from a given path using a different name encoding. I don't know if such a filesystem exists, though. bindfs has a similar purpose, but it only translates file ownership/permissions. Either way, I'm sticking with my recommendation to just convert all names to UTF-8.)

  • Thank you for referring to convmv, and I find it's a perl script. After some trying, I'm starting to worry about its reliability. convmv's processing seems to rename on a file basis, instead of on a directory basis. So, when some gbk file name sequence happens to be UTF8 sequence( touch $(echo '0xE7 0x94 0xB5 0xE8 0x84 0x91'|xxd -r).txt ), it says Skipping, already UTF-8.
    – Jimm Chen
    Apr 28 '13 at 1:39
  • 1
    Use the --nosmart option. (The tool is most commonly used for fixing ISO-8859-𝒏 filenames, which don't have this problem.)
    – user1686
    Apr 28 '13 at 8:31
  • 2
    @Peter: In your case, the name is UTF-8, but encoded twice (a program mistakenly thinking that the input isn't UTF-8 yet). Therefore you must convert it in the opposite direction. Try convmv -f utf-8 -t iso8859-1.
    – user1686
    Jul 11 '14 at 13:20
  • 1
    @Peter: [Notice how each accented letter corresponds to multiple gibberish characters, always starting with something similar to Ã; a sign that each UTF-8 byte is being interpreted in one of the old single-byte charsets. It's a very distinctive pattern.]
    – user1686
    Jul 11 '14 at 13:22
  • 1
    @Peter: However, before doing anythimg, make sure that the filename is incorrect, and not your terminal misinterpreting the correct filename. Run printf 'R\xc3\xb6kkurr\xc3\xb3\n' and see if it shows up correctly. If it doesn't, do not rename anything.
    – user1686
    Jul 11 '14 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.