Inverse word embedding: vector to word

I’m building a generative text model, and the output of one of the final layers is a word embedding (vector) of the generated word. I’m left with the task of converting this vector back to the actual word.

Is there a good algorithm for doing this inversion? I’m thinking of using a fully-connected/dense layer, but then it’s decoupled from the original (forward) embedding layer. Ideally, I’d think it’s better to make use of parameters of the embedding layer somehow for the inversion.

Answer

There is no one ‘right’ way to turn wordvectors back into words.
The issue is that the words themselves form a discrete set of points in the embedding space, and so the output of a model is very unlikely to be exactly equal to the location of any word.

Typically if your model emits a vector v then interpreting it as a word is done by finding a word w with embedding vw such that d(v,vw) is small, i.e. v is ‘close’ to the embedding of w. Choosing the distance function d is up to you, although typically the cosine similarity is used. Depending on the application, you could also consider showing the top-k similar words to your wordvector, which could offer a bit more diversity.

Attribution
Source : Link , Question Author : rishai , Answer Author : Chris Cundy

Leave a Comment