1

NOTE: This question is different from decompress ZIP with given encoding because none of those solutions work, and I'm unsure if a properly-functioning unzip program can even retrieve the correct filename.

There is a zip file with a Japanese character file name. unzip can specify character encodings, which usually works to retrieve correct filenames. (It isn't in the man or info page for unzip for my distribution, but it is in the executable's help output.)

But it doesn't work for some zip files, including this one. This is a collection of files for the animating program MikuMikuDance. Several files have a 'garbage' filename when used with unzip -O sjis -l archive.zip, or when decompressed with file-roller.

Another file in the archive refers to the garbage filenames, although it refers to the shift-JIS encoding of the name and not UTF8. It tries to load these files, so if they're named wrong, they can't load. The question here is whether the zip file is such that the correct filenames could be extracted by a different program, and whether they were named correctly on the uploader's system.

The desired filename is 'Figure_その他.fx' ("other" in Japanese). It appears to have two different representations in the zip file, one as "����" which is SJIS being interpreted as UTF8, and elsewhere as U+0082 and some Thai characters:

grep binary file This second version is the output from unzip, with or without -O sjis option. There doesn't seem to be any way to convert this back to the original filename.

The original zip file can be downloaded here (25 MB), and its contents can be opened with the free program MikuMikuDance although the '.fx' file is used for MikuMikuEffect which requires several native libraries to run in Wine. But maybe someone can answer this without the zip file.

Do zip files store an alternate filename that is normally unused? Is backslashes being displayed as yen signs on Japanese computers related in any way to the filename not being used? If you take the string that shows up as 'Normalmap Effect\Skin\Figure_���̑�.fx' and pipe it to iconv -f sjis -t utf8, the output is 'Normalmap Effect¥Skin¥Figure_その他.fx'. Wikipedia talks about an 'extra field'; is this being used to store and retrieve the 'bad' filename?

  • There is no formal specification for the charset used to encode entries in a zip file. – Ignacio Vazquez-Abrams May 3 '18 at 23:13
  • Possible duplicate of decompress ZIP with given encoding – JakeGould May 3 '18 at 23:39
  • The solution—as suggested in the dupe post I connected this to—is to use 7-Zip or Unrar instead of plain old Zip. – JakeGould May 3 '18 at 23:40
  • It's definitely not a duplicate of that post. I mention the top answer to that question in the first paragraph. Another answer mentioned 7z's encoding selection; for i in {1..3000}; do 7z l -scs$i 'archive.zip' |grep その他; echo $i; done returns nothing for i<3000, though there are a bunch of identifiers above 3000. – Misaki May 4 '18 at 0:04
  • 1
    So the compressing program interpreted SJIS as iso8859-11 or something very similar, one of the bytes was somehow changed in the process, this wrong result was stored as "Unicode filename", and decompressing programs use the wrong result. The correct 'wrong' filename as iso8859-11 would be "<U+0082>ป<U+0082>ฬ<U+0091>ผ", but in the zipfile it's "Figure_<U+0082>ป<U+0082>ฬ‘ผ.fx". – Misaki May 5 '18 at 20:27
3

I have just tested this for myself, using the link you provided, and downloaded a file entitled Thank You For You Project.zip, which I had to unzip using passcode 864 with 7-zip, and encountered no problems at all. Did I have the right file?

0

Backslashes displaying as yen signs is unrelated. The second filename is the UTF-8 filename, as documented in sections 4.5, 4.6, and Appendix D of the .ZIP specification.

In this particular case, the filename was passed to the packaging program in Shift-JIS, but was assumed to be ISO-8859-11 or TIS-620 (Thai encodings) by the packaging program. One of the bytes of the input was assumed to be, or converted into a different character in Unicode, preventing recovery of the correct filename.

When the Info-ZIP Unicode Path Extra Field is used, programs like 7z or Info-ZIP's unzip assume that it's valid, and ignore the basic filename field and the -O option used with unzip to set the source character encoding.

A patch that adds an option to override this behaviour would likely be welcomed by the program's maintainers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.