cleaning up pdftotext font issues

Question

I'm using pdftotext to make an ASCII version of a PDF document (made with LaTeX), because collaborators prefer a simple document in MS word.

The plain text version I see looks good, but upon closer inspection the f character seems to be frequently mis-converted depending on what characters follow. For example, fi and fl often seem to become one special character, which I will try to paste here: fi and fl.

What is the best way to clean up the output of pdftotext? I am thinking sed might be the right tool, but am not sure how to detect these special characters.

Answer 1

By default, pdftotext outputs unicode (UTF-8) data. If your terminal or text editor doesn't support UTF-8, ligatures such as "fi" and "fl" (which can be represented as a single character in unicode) will appear strangely, as you have noticed.

The simple fix is to tell pdftotext to output ASCII instead of unicode:

pdftotext -enc ASCII7 input.pdf output.txt

This should produce clean ASCII output, removing your need to clean it up manually afterwards.

Answer 2

Assuming you're on some kind of Unix-based system, you could run this on the output of pdftotext:

sed -i -e 's/ffi/ffi/g' -e 's/fi/fi/g' -e 's/ff/ff/g' -e 's/fl/fl/g' -e 's/ffl/ffl/g' output.txt

That should replace the ligatures with the individual letters they break into. (See my comments above for what ligatures have to do with this.)

I tested that on a text file generated through pdftotext from a LaTeX-generated PDF. And it worked fine. But if the LaTeX used a nonstandard encoding or font with additional ligatures there may be more to do.

You'll probably want to make sure the font you're using in your terminal has characters for the f-series ligatures. DejaVu Sans Mono is a good choice.