Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 2 down vote favorite
1

What I have been basically trying to find is some tool where I can enter my folder or list of text (.txt) files and get back an output of all most common phrases(keywords) for the given file.

So as an example:

Text 1 file and has no keyword so it would give random phrases as output

Text 2 file lets say has been optimized for "dog hair color" so it would obviously give me "dog hair" "dog hair color" and "hair color" as common phrases

The outpost would give me all these results separately for each file and say how many times that keyword/phrase was used in that given text file.

So to sum up the above I need to counter engineer article text files and find out what their keywords are in bulk. Since entering them one by one is very stupid and simply wasting time (thus I still have had to do it this way so far).

Edit: If someone has an answer, please reply as I still view this daily and would like a solution of any kind, even if it is a lead where to research and what to research.

share|improve this question
1  
It isn't clear that this is a programming (hence Stack Overflow) question - if you are looking for an existing tool or how to use a tool you have, try SuperUser - if you are looking for assistance writing code to meet your needs, what languages are you familiar with? This would be a reasonably good "starter" project if you wanted to learn a programming language. – Mikeb Apr 16 '11 at 18:41
Keywords that might help your search "markov chain". Add to that a map data structure of some kind and you're home and vigorously toweling off. //If this is not a programming question, just say so and we'll move it to Super User for you. – dmckee Apr 16 '11 at 18:42

migrated from stackoverflow.com Apr 16 '11 at 20:51

2 Answers

up vote 0 down vote

Knime is probably one of the best recommendations I can think of. It's fairly straightforward to use, it has built-in stuff for doing statistical analysis and if you want to jump into data-mining it has that as well. You can also use embedded R scripts, and it has the ability to use Weka from it as well.

It has the ability to get data from text files, an sql database, and many other sources.

That said, your specific task may take some ingenuity on your part to pull off, but this seems like a good fit.

share|improve this answer
It says it open source but I can't find the source, what is it written in? – Arjang May 4 '11 at 5:31
It uses eclipse as a front-end, but in my 5 minute search I had trouble finding the source as well. They claim it's GPL, so they are required to give you the source if you request it. Maybe they don't make it easy. – Brian Vandenberg May 4 '11 at 14:49
up vote 1 down vote

I would recommend Textanz for your purposes. Multiple texts are supported. Calculates frequencies of individual words and phrases of any length. Googling for "Textanz" will bring you to the vendor's site as a first ref.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.