I am making a text search engine. I need to first convert binary documents to text. I want to go with cross-platform (we develop both on windows and linux) command line (so that I can get the output via python subprocess). What are the choices for this?
There's pdftotext for PDFs and antiword for DOC (not sure about DOCX). Both free software for command line. Used both on Linux before, worked fairly well.
Sphider, a free search engine, uses pdftotext and catdoc. catdoc I've used as well, worked fine as well.
For DOC files there's also wv and abiword (abiword is a GUI but also has a command line interface). Haven't been able to test these out, though.
One tip for DOCX:
I have used this to convert DOC to PDF before, was quite good. Perhaps it supports TXT as well, or you could use pdftotext. LiveDocX also can be accessed via a SOAP web service
PS: there's also
One supplement for pdf to text, i know another open source: PDFLib tool. this library can convert whole pdf page to text with the page layout. and it supports multiple languages(all the ascii language), such as english, french, latin, besides, the CJK language is also supported, so it's very useful for your text search engine. and i find a online pdf to text tool using PDFLib, so you can make a performance of the PDFlib.