Support for FastText word embeddings

[raphael0202]

Currently, Vectors.__getitem__ returns the vector associated with the given key (https://github.com/explosion/spaCy/blob/master/spacy/vectors.pyx#L88).
This is expected for most word vectors, but it makes the integration of FastText word embeddings difficult. Indeed, the vector of a word is generated by summing the vectors associated with its char n-grams (see the gensim implementation for more details: https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/keyedvectors.py#L1657).
This is convenient as we can generate an embedding for an OOV word if we have a vector for at least one of its char n-grams.

One way to support FastText word embedding would be to let __getitem__ behave differently given a backend attribute of the Vectors (which in this case would be fasttext).

If you think this feature is worth adding to spaCy, I can participate in the implementation.

Your Environment

Python version     3.5.2          
spaCy version      2.0.7          
Platform           Linux-4.4.0-116-generic-x86_64-with-Ubuntu-16.04-xenial

[honnibal]

We do support changing the .vector attribute using the user_hooks. Would that work for your use-case?

It wouldn't let us take advantage of the character features when using the vectors as features in models --- but then, neither would changing the __get__ in the Vectors class.

[raphael0202]

Yes it would be possible to to use user_hooks to change the way vectors are computed.
But I think spaCy would really benefit from natively supporting new word embedding techniques such as FastText.
Similarly, a team from AllenAI has recently published a very interesting paper about a new type of word representation that are computed using all layers of a bi-LSTM language model: https://arxiv.org/abs/1802.05365.

Changing the Vectors class could also allow a native support of this kind of embeddings.

[honnibal]

The ELMO vectors would naturally use the .tensor attribute though --- you can't compute a table of them for the vocabulary, so they wouldn't belong in the Vectors class, or even within the Vocab.

Perhaps we could support back-off logic within the Vocab's get_vector?

[raphael0202]

Indeed for ELMO vectors the .tensor attribute seems appropriate. But if we set Doc.tensor, wouldn't it conflict with the use of this attribute by other components in the pipeline?

If we implement the switch in Vocab's get_vector, Vocab.get_vector(key) could return a different vector than Vectors.__getitem__[key], making the API inconsistent.

What I thought of was to split the Vectors class into:

• a VectorStore, responsible for storing key-value pairs (key=integer, value=array)
• a Vectors class, that serves as an interface to either:
  • querying the VectorStore directly
  • computing a vector by getting the ngrams vectors from the VectorStore (FastText case)

so we can in each case (word2vec-like or fasttext) use the Vectors.__getitem__ method.

We can possibly add a new method to return vectors in context (something like get_context_vector(token)) that would allow an implementation of ELMO-like embeddings.
The embedding type (simple, fasttext, elmo) could be serialized along the other attributes, which would allow to simply load/save the vectors using the (de)serialization methods.

[samhardyhey]

I think having this type of flexibility would be a massive boon considering the rich variety of embedding types which now exist. Flair embeddings come to mind, alongside BERT and ELMO.

[DonaldTsang]

@samhardyhey the more word embedding models the better