13 Jun '12, 2pm

WhatLanguage: Ruby Library To Detect The Language Of A Text:

Sorry, my inner coder/pedant came out. What I'm trying to say is that using whole words isn't really optimal. Why? You suffer from a sparsity problem. The probability of seeing even a common word is very small. This means that you need more training data to get equal coverage. Actually, it's worse than that. If you take the top 100 3-grams in each language you will almost certainly see them all in even a moderately short text. What's more, the probability that some of those 3-grams appear even at the word level are very good. Here's a question to ask: how many 3-grams do you need to see before you're, say, 98% certain of the language? Conversely, the probability that a given common word will appear in a text is much smaller and you're much more likely to run into a word you've never seen before. That unknown word, however, will probably contain an n-gram you've seen before...

Full article: http://www.rubyinside.com/whatlanguage-ruby-language-dete...

Tweets

RubyConf: Natural language generation and proce...

blog.nicksieger.com 08 Jun '12, 9am

RubyConf: Natural language generation and processing in Ruby Posted by Nick Sieger Sun, 22 Oct 2006 00:07:34 GMT Speaker: ...

Robots Can Learn Language From People

Robots Can Learn Language From People

planetsave.com 17 Jun '12, 2am

The researchers, Dr Caroline Lyon, Professor Chrystopher Nehaniv, and Dr Joe Saunders have recently done new experiments a...

The Technology of Text #tYPE #Typography

spectrum.ieee.org 20 Jun '12, 5am

If you’re reading this article on your computer , there’s a good chance you won’t get all the way to the end. Not because ...

Press Release: Boosting efficiency in the desig...

premierfarnell.com 12 Jun '12, 3pm

New Würth Elektronik eiSos part library for more efficiency in the design process now available 12th June 2012, London - C...

Learning Ruby and Rails:

everydayrails.com 18 Jun '12, 3am

I live in a town dominated by Python, PHP , and Java developers, so hiring someone who can come in and write Ruby code fro...