Converting Filename Encoding

Question

my operating system is Arch Linux. I am trying to extract a .zip archive that contains CJK characters in its filenames. It was most likely created on a Windows machine.

I tried the unzip utility and it produced invalid symbols. The same with 7za, but with slightly different ones. My LANG variable was set to en_US.UTF-8, but setting it to ja_JP.ujis seems to have no effect. I'm assuming that this means that the CJK filenames were encoded into the archive in a format other than UTF-8, and I need to convert it to UTF-8 for them to display properly.

I know of convmv, and I used a shell script to test every possible encoding from convmv --list to no avail. I have the Unicode equivalents of the filenames for the most part, but in a format that's cumbersome to manually rename all of them, but with them I can verify if the conversion was successful or not.

Observing the hex dump of ls and with positional deduction I concluded that U+4EBA (人) is represented as 0xC9 0x6C with the unzip output and 0xC2 0x90 0x6C with the 7za output. This also means it isn't impossible that I'm not dealing with the original encoding in the first place.

So, why would two zip extractors produce different results, and are there any other leads to help me convert these filenames to UTF-8 correctly?

Answer 1

My first guess, when dealing with UTF8 pathnames, is to attempt to use the Python zipfile library - I'm guessing that its cross-platform enough for your needs (OTOH, the module docs mention nothing about UTF8...).

Here is a small script to attempt this:

#!/usr/bin/python
import zipfile
import sys
import os

if len(sys.argv) < 2:
    print "I require a file name and a directory to unzip to"
    sys.exit()
zip = zipfile.ZipFile(sys.argv[1])
if not os.path.exists(sys.argv[2]):
    os.mkdir(sys.argv[2])

zip.extractall(sys.argv[2])

This can be chmod +x'd and run - see if it works in your case.

In all infinite improbability, this will solve your problem.