Learning words from man pages

[Abhinav Upadhyay <er.abhinav.upadhyay%gmail.com@localhost>]

Hi All,

I came across an interesting paper from Google on machine learning[1],
where they came up with an efficient representation for words from a
corpus. These representations are called word embeddings in general,
and they have titled their method as word2vec.

It is a two layer neural network which given a corpus as input,
produces a set of word vectors as its output. These vectors represent
each word in the corpus in a vector space, where words with similar
semantics lie nearer to each other in that space.

There are two methods of training the data:
1. Bag of words: here the ordering of the words in the corpus is not
considered. It can be thought of like, given a word, what are the
other words similar to this.
2. Skip grams: It considers the ordering of the words, it can be
thought of like, if given word w1, what is the probability of word w2
appearing next.

They have shown interesting implications of this, for example,
"France" and "Italy" are closer to each other in the model that they
trained. Another interesting observation is the application of vector
algebra here, for example they show that:

vector(king) - vector(man) + vector(woman) = vector(queen).

This technique is becoming widely popular and has applications in
areas like search, question answering, summarization. I've trained
this on our man page corpus data (plus some man pages from pkgsrc) and
put a demo here: https://man-k.org/words/

Some of the interesting queries that I found:
bug: gives <defect, problem, undetected, lurk etc> in the top results
man: gives <mdoc, html, overview, readme>
netbsd: shows <freebsd, openbsd, ultrix, linux>
christos: <zoulas, cornell> in the top two

Give it a try and let me know how you like it. :)

Coming Soon: I still need to implement the interface for doing vector
addition and subtraction.

BUGS: Use single word queries in non-plural form for best experience ;)

[1] http://arxiv.org/pdf/1301.3781.pdf

Regards
Abhinav