Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to develop a search engine based on the Term frequency in the document, i have a dataset of 17000 PDF documents, Is there a way i can count all the words and their occurrences in a particular document?

Example: Would you like to have responses to?

Output: Would=1 you=1 like=1 to=2 responses=1

share|improve this question
    
What operating system are you using? –  Indrek Oct 23 '12 at 20:40

1 Answer 1

The general problem of get text from a PDF is more complicated than it sounds. Once you've solved that problem to your satisfaction, the problem boils down to determining the term frequency in a bunch of text files. You should be able to implement that directly, or else get some advice on stackoverflow.

To get text from a PDF, you have to consider the way that data is structured in a PDF.

"Text" can be any of the following in a PDF:

  • Characters rendered as images
  • Individual text characters with spacing elements between them (which can make it hard to distinguish between "words", because how do you define how much space has to be between words to make it a separate word?)
  • Spans of ordinary text
  • Dynamic content (HTML, links, form fields, videos, etc.)

If all of your source PDFs adhere to a similar structure or were created using the same program, it should be easy to create a program to accurately parse the structure and extract the text semi-reliably. However... if your PDFs are from different authors and third-parties whose document production you have no control over, it could be a bit more complicated.

The following techniques may apply to extracting text from PDF:

  • Using Optical Character Recognition (OCR) technology to view the final rendered PDF and extract the text from what the OCR program "sees".
  • Using tools that understand the low-level architecture of the PDF document to parse the logical elements and determine which of them constitute text (this method alone will not be able to glean any text from images, but may work well for simple documents where all the text is stored as plain text or lightly-formatted text).
  • Using proprietary or open source tools that can do a combination of the above two techniques.

So your approach should go something like this:

  • Determine whether there are any commonalities among the input PDFs, such as the consistent presence (or absence) of text rendered in images, in order to define the scope of your requirements, in terms of which extraction techniques you need to use. If you need to work with the general case of any input PDF, then you should plan for the worst and assume that there will be text in images, and that you will need to do OCR.
  • Based on your distilled requirements, determine whether there is existing software (proprietary or open source or otherwise, depending on your preference) which implements the technique(s) you require.
  • Of the available software, determine which is easiest to integrate into the programming environment / architecture you're using for your search engine (is it in C? Java? .NET? And so on.)
  • Figure out if you need to do any sort of custom "by hand" parsing or scanning on top of the library functionality. You can do this by randomly selecting a small batch of PDFs (say, 25) out of your stack; run the "PDF to text" algorithm on them; and manually verify whether the extracted output is accurate. If not, you may have to customize the libraries implementing the techniques, or create your own.
  • Once you have the PDF to text functionality working to your satisfaction, your problem boils down to indexing the term frequency in plain text. There are many techniques for this, from Map/Reduce (see Hadoop) to databases to simply storing a hash map in memory. The technique you use will depend on the scale of your program; how much hardware you can throw at it (one desktop? a cluster? a large server? a mainframe?); and how frequently you have to run the job (constantly? nightly? once a year?).
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.