Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

My company is looking to create a PivotViewer visualization of a client's Wordpress 2 blog posts for the last 11 years. To do so, however, we need to edit the somewhat haphazard, incomplete, and generally poor tags for use as sortable categories. I'm looking for a tool that will analyze their blog entries and perform word counting, to give us a sense of what we're dealing with.

Ideally, it would have all of these features:

  1. Word blacklisting (ignore)
  2. Word stemming
  3. Custom synonym merging
  4. Counting all uses
  5. Counting number of posts a word appears in.

I would have thought that this sort of textual analysis would be extremely common, but I haven't been able to find any software that does this sort of thing on entire blogs. Is there software available to do this?

share|improve this question
Interesting. When in doubt, Python's got your back. – Doc Aug 11 '11 at 15:28
Yeah...I'm really hoping I don't have to roll my own on this one, though. – Brian Bauman Aug 11 '11 at 15:32
there is something that does this... I remember a friend analyzed wikipedia... I will check with him tomorrow – Keltari Aug 15 '11 at 4:58

The software you are looking for can have many titles, like "Content analysis", "Tag cloud" or "Meta Tags" and many more such as "text analysis" and "text mining".

There are very many software tools for these purposes, both free and commercial.

I do not have personal experience with such tools, but a good place to start is Text Analysis Tools that lists dozens of such tools, both free and commercial.

Another such list is Text Analysis, Text Mining, and Information Retrieval Software.

share|improve this answer
I filtered my way through the first list, but none of the free options include much more than linguistic analysis. Haven't looked through the second list yet - I may end up rolling my own. – Brian Bauman Aug 16 '11 at 13:48

Take a look at Rapidminer or Weka

Seeing as its a clients blog, you probably have database access. Download all articles as plaintext and use one of the above programs to deal with the natural language processing questions (1,2,3, and 5).

The number of uses is hard to truly automate since it has to do with automatically determining the meaning of words using the context.

share|improve this answer
Counting all uses, not users. Thanks for the suggestions, though. – Brian Bauman Aug 21 '11 at 15:46
I misread, mybad. Still you should checkout Rapidminer or Weka for natural language processing. That is, unless the dataset is huge, because both try to fit it in memory – suweller Aug 21 '11 at 16:20

one of the most content analysis software is WordStat designed by Provalis Research

WordStat is a text analysis module for QDA Miner or SimStat. WordStat combines content analysis method by using dictionary approach and many algorithms exploration or various text mining methods. WordStat can apply existing categorization dictionaries to a new text corpus. It also may be used in the development and validation of new categorization dictionaries. When used in conjunction with manual coding, this module can provide assistance for a more systematic application of coding rules, help uncover differences in word usage between subgroups of individuals and assist in the revision of existing coding using KWIC (Keyword In Context) tables. WordStat is specifically designed to study textual information such as responses to open-ended questions, interviews, titles, journal articles, public speeches, electronic communications, etc.

share|improve this answer

You might want to give Wolfram's Mathematica a try. You will have to do some programming, but all the tools you need are there:

share|improve this answer

Some of these questions could be answered in a quick and dirty way using Google Search on your blog (easiest if it has its own domain).

share|improve this answer

Zemanta does analysis and can suggest tags and links. It's a wordpress plugin too.

Only problem: as it currently stands it requires manual opening and selecting and saving of each post.

There are a huge number of auto-tag plugins for wordpress though. You should search the plugin finder and give a few a try.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .