I have files of ISO-8859 encoded Text sent to me regularly by customers that contain debug information.
Recently, they have started to turn up in my inbox as ASCII encoded with a few extra characters in the file as a result, breaking the parser I wrote for these files.
I narrowed it down to the E-Mail Client being used by the customer which changes encoding.

Is it possible to revert this false change of encoding?
Because I'm working with Linux, I'd favor a solution for Linux but would be interested in solutions for other OSes as well.

Here's some output for reference and information:

$ file /tmp/{wrong,right}_enc.txt
/tmp/wrong_enc.txt: ASCII text
/tmp/right_enc.txt: ISO-8859 text, with very long lines, with CRLF line terminators

$ file -bi /tmp/{wrong,right}_enc.txt
text/plain; charset=us-ascii
text/plain; charset=iso-8859-1

Content examples (Info is different, structure should be the same but wrong_enc.txt has aditional characters behind '=' and at EOL for example):

$ head /tmp/{wrong,right}_enc.txt
==> /tmp/wrong_enc.txt <==
Tue Jun 11 15:51:54 2019=0D
###Eth-eth0-driver =3D igb=0D
###Eth-eth0-ipaddr =3D
###Eth-eth0-link =3D yes=0D
###Eth-eth1-driver =3D e1000e=0D
###Eth-eth1-ipaddr =3D =0D
###Eth-eth1-link =3D no=0D

==> /tmp/right_enc.txt <==
Wed Apr 17 16:47:21 2019

###Eth-eth0-driver = e1000e
###Eth-eth0-ipaddr =
###Eth-eth0-link = yes
###Eth-ethnames = eth0
###Eth-ethtool-eth0 =

The shown encoding is Quoted-Printable, and is completely normal to see if you are looking at "raw" email source text – most non-ASCII messages and text attachments, and even some purely-ASCII ones, are encoded using QP (or even Base64).

If the sending mail client correctly indicated its usage in the Content-Transfer-Encoding header, then the receiving mail client will automatically decode QP into normal text. (If the MIME header is correct but your mail client doesn't perform this decoding, then it is your mail client that is broken, not the customer's.)

If you need to perform the decoding yourself, there are various libraries for it:

ruby -e 'print STDIN.read.unpack("M")[0]' < wrong.txt > fixed.txt
perl -0777 -ne 'use MIME::QuotedPrint; print decode_qp($_)' < wrong.txt > fixed.txt

While you should use the existing module, it is relatively simple to decode QP directly:

perl -0777 -pe 's/=\n//gs; s/=([0-9A-F]{2})/pack("C", hex($1))/gse' < wrong.txt > fixed.txt
  1. = followed by newline is removed (lines joined together);
  2. = followed by two hexadecimal digits is replaced with the corresponding byte.
  • Wow, thanks! I would've never found the clue to Quoted-Printable! – wullxz Jun 24 '19 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.