Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm using pdftotext to make an ASCII version of a PDF document (made with LaTeX), because collaborators prefer a simple document in MS word.

The plain text version I see looks good, but upon closer inspection the f character seems to be frequently mis-converted depending on what characters follow. For example, fi and fl often seem to become one special character, which I will try to paste here: fi and fl.

What is the best way to clean up the output of pdftotext? I am thinking sed might be the right tool, but am not sure how to detect these special characters.

share|improve this question
fl, fi, ff, ffl, and ffi are common typographic ligatures, commonly replaced by a single character (and definitely with TeX): - perhaps you just need to check that the font you're viewing the output in has them, and that the encoding is right. – frabjous Dec 10 '10 at 3:28
oh, and you mean pdftotext from poppler, right, not pdftotex ? – frabjous Dec 10 '10 at 3:28
Do you have the original TeX source? Why not use, e.g., latex2rtf or oolatex (from TeX4ht) to generate a Word Processor file for the Word junkies? Compiling to PDF and then converting to plain text seems like a very weird route for conversion. – frabjous Dec 10 '10 at 3:40
Oh, and if you DO want to convert PDF to plain text, consider using ebook-convert from calibre ( rather than pdftotext. It allows plain text output (and a variety of other formats), and handles ligatures for you. – frabjous Dec 10 '10 at 3:43
I did mean pdftotext. Typo fixed. I have original TeX source, but latex2rtf and oolatex do not work as well as pdftotext. I use additional packages like siunitx and glossaries, and therefore it seems like going via the PDF is the best solution. I wish there were a better way. – user31752 Dec 10 '10 at 18:06
up vote 4 down vote accepted

By default, pdftotext outputs unicode (UTF-8) data. If your terminal or text editor doesn't support UTF-8, ligatures such as "fi" and "fl" (which can be represented as a single character in unicode) will appear strangely, as you have noticed.

The simple fix is to tell pdftotext to output ASCII instead of unicode:

pdftotext -enc ASCII7 input.pdf output.txt

This should produce clean ASCII output, removing your need to clean it up manually afterwards.

share|improve this answer
Thanks. I found the ebook-convert suggestion above to be the best. Your advice might improve the default behavior of pdfottext, but I think my terminal does support UTF-8, and ebook-convert seems to handle superscripts and other things better. – user31752 Jan 11 '11 at 16:00

Assuming you're on some kind of Unix-based system, you could run this on the output of pdftotext:

sed -i -e 's/ffi/ffi/g' -e 's/fi/fi/g' -e 's/ff/ff/g' -e 's/fl/fl/g' -e 's/ffl/ffl/g' output.txt

That should replace the ligatures with the individual letters they break into. (See my comments above for what ligatures have to do with this.)

I tested that on a text file generated through pdftotext from a LaTeX-generated PDF. And it worked fine. But if the LaTeX used a nonstandard encoding or font with additional ligatures there may be more to do.

You'll probably want to make sure the font you're using in your terminal has characters for the f-series ligatures. DejaVu Sans Mono is a good choice.

share|improve this answer

You must log in to answer this question.