von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

[zomux]

Main Authors / Organization

Sachin Kumar & Yulia Tsvetkov

Carnegie Mellon University

PDF link

https://arxiv.org/abs/1812.04616

Hypothesis

Sequence-to-sequence model can also be trained by directly computing loss on the continuous outputs.

This means the model does not use softmax, thus can use an infinity vocabulary.

Approach

Using von Mises-Fisher loss with two kinds of regularization to encourage improving cosine distance.

Main Experimental Result

• IWSLT16 datasets
• on par with LSTM-based BPE->BPE model with both regularization
• MaxMargin loss is also as good as vMF loss

Notes