WhatLanguage: Ruby Library To Detect The Language Of A Text

WhatLanguage: Ruby Library To Detect The Language Of A Text:

Sorry, my inner coder/pedant came out. What I'm trying to say is that using whole words isn't really optimal. Why? You suffer from a sparsity problem. The probability of seeing even a common word is very small. This means that you need more training data to get equal coverage. Actually, it's worse than that. If you take the top 100 3-grams in each language you will almost certainly see them all in even a moderately short text. What's more, the probability that some of those 3-grams appear even at the word level are very good. Here's a question to ask: how many 3-grams do you need to see before you're, say, 98% certain of the language? Conversely, the probability that a given common word will appear in a text is much smaller and you're much more likely to run into a word you've never seen before. That unknown word, however, will probably contain an n-gram you've seen before...

Full article: http://www.rubyinside.com/whatlanguage-ruby-language-dete...