4

We're a small group that is promoting the spread of Unicode in India (here legacy encodings are deeply entrenched). But I have a problem when I convert a document in unicode text in any Indic language to PDF format. The text displays as intended, but on copy pasting the content partially turns gibberish.

I am using inDesign CC for typesetting on a Win 7. I can export to epub format just fine. But the exported PDF has this problem. I also tried printing to Adobe PDF printer and PrimoPDF, it only got worse. On checking out PDF's on the internet, turns out this problem exists in all such unicode encoded Indic PDF (and probably all East Asian complex scripts). Is that a problem in the PDF specs?

Check out the PDF here http://www.rajbhasha.nic.in/pdf/dolebook-4.pdf

Copy any text and match with the original, you'll see characters are replaced by other characters, unnecessary white space has crept in.

Now we're promoting unicode on grounds that it'll make copy-pasting and searching/indexing easier. This problem totally destroys that. Any ideas?

5
  • I can confirm that copy/pasting your document on a Mac also alters the characters. I can't read it, but there are a few noticeable differences. That might suggest the source conversion is at fault. Maybe have a look at Calibre (freeware) to do the conversion instead. It might at least tell you where the issue starts. – Tetsujin Sep 15 '16 at 10:18
  • Can confirm copy/pasting problem on Linux with xpdf. I looked at the PDF with mutool, it uses special fonts that don't use unicode encoding. You need some other program to do your typesetting (instead of inDesign CC), and one that produces PDFs with unicode encoding (no, I don't know any option for Windows 7). This question has technical details about unicode in PDF, it seems possible, but not easy to do. – dirkt Sep 15 '16 at 11:35
  • Correction: the example PDF actually uses /ToUnicode mappings, but they don't seem to work for some reason. Don't know yet what goes wrong. – dirkt Sep 15 '16 at 11:51
  • See also: stackoverflow.com/questions/12703387/… – user1686 Sep 15 '16 at 13:35
  • @Tetsujin: I did try to convert the resulting epub to pdf using various tools, including calibre. The problem actually worsened. – coldbreeze16 Sep 17 '16 at 17:27
5

I decompressed the pdf with mutool clean and had a look at. The problem seems to be that as described as in this stackoverflow question, it's difficult to use unicode encoding for the fonts. For this reason, the fonts that the PDF contains use a different encoding. However, it also contains /ToUnicode objects for each font with a complicated mapping from the font glyphs to the unicode characters.

Now many PDF viewers (like e.g. xpdf on Linux) don't seem to pay attention to this complicated mapping (or at least not to a mapping with such a complexity, though they may work on more simple mappings), which is why you get garbage when trying to copy and paste. However, with other PDF viewers (like mupdf) it works, as I've confirmed.

So the problem is located in the PDF viewer, not in the document. Also, PDFs and unicode don't go together that well, as you can see from the complicated means necessary to do the translation.

Possible solutions: (1) pressure the developers of PDF viewers to fully support \ToUnicode mappings. Maybe fix them yourself for open source ones. (2) Promote the usage of a particular PDF viewer that works with the mappings. (3) Try to use fonts inside the PDF where the glyph encoding matches the unicode encoding. This seems possible with 16-bit unicode codepoints (and the Indian characters seem to be 16-bit as far as I can tell), but I don't know how well this will work, or which application you should use to produce such PDFs.

12
  • This seems to be correct, because with different PDF readers I get different outputs on copying which is only possible if each one implemented the ToUnicode table interpretation differently. As for your proposed solutions, 1 and 2 are not feasible because people wont shift their OS and pdf reader just for this. And all pdf readers i've tested have problems (Adobe Acrobat, PDF X, Foxit, Google pdf viewer). As for 3, I didn't get it. This document uses standard Unicode Hindi font Mangal supplied with Win Vista and above. – coldbreeze16 Sep 17 '16 at 17:41
  • Mupdf also works on Windows, so you can try that too. qpdfview on Linux also works. I'll try to make an example file for (3) to see if it works even without ToUnicode tables, but that might take some time. – dirkt Sep 17 '16 at 19:31
  • It looks like xpdf just ignores any "complex" characters besides ASCII for cut and paste, while mupdf produces an UTF8 encoded paste. That means I can't properly test here on Linux. I've created a decompressed PDF file with xetex that is not as complex as your example. You can inspect it with a text editor. Glyphs are in the 0200-0400 range, corresponding unicode is 09xx. Test your viewers with it, if you can paste unicode chars in the 0200-0400 range, creating a special font should work with that viewer. – dirkt Sep 19 '16 at 9:25
  • I was away from home, just returned and tested mupdf on Win 7 and Ubuntu 14.04. The same problem persists on copying. I am not sure, what I am doing wrong. I tried your PDF on all my viewers as well. No luck. – coldbreeze16 Sep 19 '16 at 18:54
  • Huh. I'm on Debian, which is very close to Ubuntu, and mupdf works fine (shift-right button to select). Where were you pasting it into? Can you do a xclip -o | hexdump -C from the commandline on the selection and post the results? (Packages xclip, bsdmainutils if not installed). Also, can you post what exactly are the results for my PDF with the various viewers? A tool like inside clipboard helps, IIRC it also shows hex. – dirkt Sep 20 '16 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.