Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I wish to create a concepts map from unstructured text. For example

Desired input: find "/" -name "*.txt"
Desired output: concepts-graph.dot

In other words, I want to mine my text files and create some kind of structured representation of key words/concepts. Loosely a poor-man's Google text analyser.

Is there an open source tool/API that can find relationships between terms in a plaintext file?

share|improve this question
add comment

1 Answer 1

There are many tools you could build with:

As far as key words go, there are basic tools, like Porter stemmers, available in most programming languages, and lots more options for specific languages.

For example, there's NLTK (natural language toolkit) - a Python text classification system - which you can use for things like part-of-speech tagging (http://nltk.org/)

Also, there are various text mining packages you can use within R: http://tm.r-forge.r-project.org/, for example (also see these slides: http://www.zinkov.com/posts/2010-10-21-slides_from_larug/tm_slides.pdf).

If you can provide a clearer idea of the sort of text analysis you have in mind it would be easier to suggest specific packages that might be relevant?

share|improve this answer
    
Thanks for the reply Soz. Basically, my todo.txt contains a lot of URL + Title pairs from websites I've visited (I save them all before closing my browser window every session). I want to get a pictorial representation of what I have been spending my time reading, in the form of a spider diagram (or graph). So the graph might contain paths like: (1) root -> nosql -> cassandra (2) root -> nosql -> neo4j (3) root -> soccer -> brazilian players -> Ronaldo So instead of spending hours reading through my txt file, I can just look at a diagram and extract useful content from it. –  Sridhar-Sarnobat Jun 12 '12 at 2:33
    
Understood. Well, in that sort of instance (bespoke datasets), my experience is that the easiest way is picking your favourite of Perl, Python or other similar language and building a dot file directly. Regarding dot files: I suggest the keyword 'strict' when declaring the graph to get rid of duplicate paths, and try edge [penwidth=0.2] or so to keep the lines suitably light. Regarding title parsing, part-of-speech tagging may help to pull out likely-relevant candiddate terms. –  Soz Jun 12 '12 at 2:47
    
I guess that's all the info I need in theory. The hard part is finding a simple-to-use package. I tried maui and jate but gave up on both. –  Sridhar-Sarnobat Jun 12 '12 at 5:49
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.