Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm using pdftotext to make an ASCII version of a PDF document (made with LaTeX), because collaborators prefer a simple document in MS word.

The plain text version I see looks good, but upon closer inspection the f character seems to be frequently mis-converted depending on what characters follow. For example, fi and fl often seem to become one special character, which I will try to paste here: fi and fl.

What is the best way to clean up the output of pdftotext? I am thinking sed might be the right tool, but am not sure how to detect these special characters.

share|improve this question
    
fl, fi, ff, ffl, and ffi are common typographic ligatures, commonly replaced by a single character (and definitely with TeX): en.wikipedia.org/wiki/Typographic_ligature#Computer_typesetting - perhaps you just need to check that the font you're viewing the output in has them, and that the encoding is right. –  frabjous Dec 10 '10 at 3:28
    
oh, and you mean pdftotext from poppler, right, not pdftotex ? –  frabjous Dec 10 '10 at 3:28
    
Do you have the original TeX source? Why not use, e.g., latex2rtf or oolatex (from TeX4ht) to generate a Word Processor file for the Word junkies? Compiling to PDF and then converting to plain text seems like a very weird route for conversion. –  frabjous Dec 10 '10 at 3:40
    
Oh, and if you DO want to convert PDF to plain text, consider using ebook-convert from calibre (calibre-ebook.com) rather than pdftotext. It allows plain text output (and a variety of other formats), and handles ligatures for you. –  frabjous Dec 10 '10 at 3:43
    
I did mean pdftotex*t*. Typo fixed. I have original TeX source, but latex2rtf and oolatex do not work as well as pdftotext. I use additional packages like siunitx and glossaries, and therefore it seems like going via the PDF is the best solution. I wish there were a better way. –  user31752 Dec 10 '10 at 18:06
show 1 more comment

2 Answers

up vote 1 down vote accepted

By default, pdftotext outputs unicode (UTF-8) data. If your terminal or text editor doesn't support UTF-8, ligatures such as "fi" and "fl" (which can be represented as a single character in unicode) will appear strangely, as you have noticed.

The simple fix is to tell pdftotext to output ASCII instead of unicode:

pdftotext -enc ASCII7 input.pdf output.txt

This should produce clean ASCII output, removing your need to clean it up manually afterwards.

share|improve this answer
    
Thanks. I found the ebook-convert suggestion above to be the best. Your advice might improve the default behavior of pdfottext, but I think my terminal does support UTF-8, and ebook-convert seems to handle superscripts and other things better. –  user31752 Jan 11 '11 at 16:00
add comment

Assuming you're on some kind of Unix-based system, you could run this on the output of pdftotext:

sed -i -e 's/ffi/ffi/g' -e 's/fi/fi/g' -e 's/ff/ff/g' -e 's/fl/fl/g' -e 's/ffl/ffl/g' output.txt

That should replace the ligatures with the individual letters they break into. (See my comments above for what ligatures have to do with this.)

I tested that on a text file generated through pdftotext from a LaTeX-generated PDF. And it worked fine. But if the LaTeX used a nonstandard encoding or font with additional ligatures there may be more to do.

You'll probably want to make sure the font you're using in your terminal has characters for the f-series ligatures. DejaVu Sans Mono is a good choice.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.