Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I wish to create a concepts map from unstructured text. For example

Desired input: find "/" -name "*.txt"
Desired output: concepts-graph.dot

In other words, I want to mine my text files and create some kind of structured representation of key words/concepts. Loosely a poor-man's Google text analyser.

Is there an open source tool/API that can find relationships between terms in a plaintext file?

share|improve this question

closed as off-topic by random Sep 28 at 13:15

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – random
If this question can be reworded to fit the rules in the help center, please edit the question.

    
Sridihar, We're trying to merge [graphs] into [charts] per meta.superuser.com/questions/6841/…;. –  Kenster Sep 28 at 12:35
    
hmmm, so neither "graph" nor "chart" can be used here. And theres no tag like "directed acyclic graph" –  Sridhar-Sarnobat Sep 29 at 17:47

1 Answer 1

There are many tools you could build with:

As far as key words go, there are basic tools, like Porter stemmers, available in most programming languages, and lots more options for specific languages.

For example, there's NLTK (natural language toolkit) - a Python text classification system - which you can use for things like part-of-speech tagging (http://nltk.org/)

Also, there are various text mining packages you can use within R: http://tm.r-forge.r-project.org/, for example (also see these slides: http://www.zinkov.com/posts/2010-10-21-slides_from_larug/tm_slides.pdf).

If you can provide a clearer idea of the sort of text analysis you have in mind it would be easier to suggest specific packages that might be relevant?

share|improve this answer
    
Thanks for the reply Soz. Basically, my todo.txt contains a lot of URL + Title pairs from websites I've visited (I save them all before closing my browser window every session). I want to get a pictorial representation of what I have been spending my time reading, in the form of a spider diagram (or graph). So the graph might contain paths like: (1) root -> nosql -> cassandra (2) root -> nosql -> neo4j (3) root -> soccer -> brazilian players -> Ronaldo So instead of spending hours reading through my txt file, I can just look at a diagram and extract useful content from it. –  Sridhar-Sarnobat Jun 12 '12 at 2:33
    
Understood. Well, in that sort of instance (bespoke datasets), my experience is that the easiest way is picking your favourite of Perl, Python or other similar language and building a dot file directly. Regarding dot files: I suggest the keyword 'strict' when declaring the graph to get rid of duplicate paths, and try edge [penwidth=0.2] or so to keep the lines suitably light. Regarding title parsing, part-of-speech tagging may help to pull out likely-relevant candiddate terms. –  Soz Jun 12 '12 at 2:47
    
I guess that's all the info I need in theory. The hard part is finding a simple-to-use package. I tried maui and jate but gave up on both. –  Sridhar-Sarnobat Jun 12 '12 at 5:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.