Word Frequency for YouTube Videos

YouTube has a feature where you can browse the top viewed videos over a specific time-frame (today, this week, this month, or all time). I thought it would be interesting to see which words (if any) pop up more than others. By just glancing at the list I guessed that "justin" and "beiber" would top the list. I thought I'd write some quick Groovy code to see if I was right.

The Stats:

Here is what I found when I looked at the top 160 most viewed videos of all time (as of today):

Top 25 Words:

Word	Count	Freq
official	19	3%
music	12	1%
song	7	1%
cyrus	7	1%
miley	7	1%
version	6	0%
gaga	6	0%
lady	6	0%
bieber	6	0%
justin	6	0%
jason	5	0%
feat	5	0%
dance	5	0%
love	5	0%
baby	5	0%
high	4	0%
david	4	0%
best	4	0%
this	4	0%
sean	3	0%
nuki	3	0%
iglesias	3	0%
enrique	3	0%
goes	3	0%
like	3	0%

Other Stats:
Total words: 619
Total unique words: 424

The Code:

All the source code is located here (box.net)

Here are the guts of the program:

def html = new XmlSlurper(new SAXParser()).parse(urlString)
html.'**'.findAll{ it.@class == 'video-title'}.each {nextVideo ->
 //split the title using regex on non-word characters
 nextVideo.text().split(/\W/).each{nextWord ->
  def lowerCase = nextWord.toLowerCase()
  //limit the results to "interesting" words
  if(lowerCase.length() >= minWordLength && !(lowerCase in ignoreList)){
   wordFreq[lowerCase] = wordFreq[lowerCase] == null ? 1 : wordFreq[lowerCase] + 1
  }
 }
}

Here is what it does:
1. Parse the YouTube URL using XmlSlurper
2. Find all the titles on the page
4. Split the title to get individual words
5. Convert it to lower case to make comparisons easier
7. Limit the words by a minimum length (to get rid of stuff like "of", "and", "the") and ignore other words (like "video")
8. Update the wordFreq map

I am not very comfortable with minimum length and ignoring words, but without that the top 10 words were: video, the, official, ft, music, i, t, you, in, on. That is much less interesting that the filtered list, in my opinion.