Fix encoding of German umlauts in directories and filenames (ü = u╠ê and so on)

Question

I have many zip-files where there are encoding errors for the German umlauts (äüöÄÜÖß). They show up in both the filename.zip as well as in the included directories and files like this:

• Fünf = Fu╠ênf
• Räuber = Ra╠êuber
• Überfall = U╠êberfall

and so on. Usually I use Linux, but because of this issues I also tried a Windows7 VM but it results in the same encoding mess up. On Linux I played around with convmv and detox, but with no success.

When I use

• convmv -f iso-8859-1 -t utf8 --replace --notest -r *

I get "Skipping, already UTF-8".

Any thoughts about this?

Answer 1

The reason that you're getting the "already UTF-8" warning is that those strings are really already in UTF-8. The "ü" character was encoded OSX-style as a 'u' followed by the two bytes "\xCC" and "\x88". These two bytes together make up the UTF-8 representation of \u0308, the combining diaeresis.

If you look at the code page 437 listing here, you'll see the \xCC character as "╠" and the \x88 character as "ê".

Whatever it is that you're using to display those character sequences is not interpreting them as UTF-8 but rather as CP437.

A quick proof, if you read ruby, that displays as expected in my UTF-8 terminal:

$ ruby -e 'puts "u\xCC\x88"' | iconv -f cp437 -t utf-8
ü
$ ruby -e 'puts "u\xCC\x88"'
ü

Answer 2

Warning: Draft!

I tried to fix this with "detox" but couldn't find a way to chain characters together. Based on the answer of @S2VpdGgA I made this compendium.

Because I preview all I do (with the echo) it is safe for me on those rare occasions.

But really somebody may want to do this properly. There may be loads of other cases like "é", "à", "è" etc...

######
# Preparation and tests
# You may need to extract your own character group from your filename, if this gets lost via this web form.
# My reconstruction as follows:

# note: the echo sends the chain of chars, copied from the console.

echo 'ä' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'ä' | sed -e 's/a\xcc\x88/ä/'

echo 'ö' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'ö' | sed -e 's/o\xcc\x88/ö/'

echo 'ü' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'ü' | sed -e 's/u\xcc\x88/ü/'

echo 'Ä' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'Ä' | sed -e 's/A\xcc\x88/Ä/'

echo 'Ö' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'Ö' | sed -e 's/O\xcc\x88/Ö/'

echo 'Ü' | perl -pe 's/([^x\0-\x7f])/"\\x" . sprintf "%x", ord $1/ge'
echo 'Ü' | sed -e 's/U\xcc\x88/Ü/'

# Final version
# test all at once
echo 'Ä' | sed -e 's/a\xcc\x88/ä/' | sed -e 's/o\xcc\x88/ö/' | sed -e 's/u\xcc\x88/ü/' | sed -e 's/A\xcc\x88/Ä/' | sed -e 's/O\xcc\x88/Ö/' | sed -e 's/U\xcc\x88/Ü/'


# wrap into a recursion
# note: not recursive as-is because folder can change

cd /path/to/dir
find . -maxdepth 1 | while read FILE ; do
    newfile="$(echo ${FILE} | sed -e 's/a\xcc\x88/ä/' | sed -e 's/o\xcc\x88/ö/' | sed -e 's/u\xcc\x88/ü/' | sed -e 's/A\xcc\x88/Ä/' | sed -e 's/O\xcc\x88/Ö/' | sed -e 's/U\xcc\x88/Ü/')" ;
    echo mv -T "${FILE}" "${newfile}";
done 

# (remove the 'echo ' to actually make changes)
#######

Answer 3

My guess is the filesystem that you are attempting to decompress or manipulate the files. FAT32 isn't going to like your umlauts. Try copying these files off of the flash drive (or what have you) and then decompress the zip file to see what kind of characters the filenames produce.

Both NTFS (Windows) and Ext4 (Mint) shouldn't have a problem with the name encoding.

The name encoding of the zip files themselves on the FAT32 system are most likely not going to change or be fixed when you copy them to a proper supporing filesystem, but the subdirectories when decompressed should be fine.

Answer 4

First note that character encoding is its own section of hell. In the Windows world there still exists a nasty dualism between UTF-8 and M$ playing stupid for a long time and insisting on ISO-8859 (guess who came up with it). As mentioned above it has almost certainly something to do with the file system. My solution is not a technical one, but one that has worked for me for many years now:

My personal bit of advice for file names is always the same: Just stick with alphanumerics plus dash ( - ) and underscore ( _ ). Write umlaute as ae, ue and oe. Don't use spaces and other special characters. It is a little bit inconvenient at first, but it will save you a lot of pain in unexpected places.

As a side note: yes this is sort of a nasty "hack" but if you work cross platform you often have to fall back to the last common denominator. You would take it for granted that something basic like character encoding would be a hard standard, but it turns out standards are a hard thing to get. This XKCD summes it up quite nicely