feat: add mpnet model family

[snewcomer]

https://arxiv.org/pdf/2004.09297

Huggingface cards
https://huggingface.co/microsoft/mpnet-base
https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-cos-v1

[snewcomer]

note this was copied from the Bert implementation. A few adjustments in the options but that was about it

[snewcomer]

https://huggingface.co/hf-internal-testing/tiny-random-MPNetModel

[jonatanklosko]

We want to compare against the reference values from hf/transformers:

from transformers import MPNetModel
import torch

model = MPNetModel.from_pretrained("hf-internal-testing/tiny-random-MPNetModel")

inputs = {
  "input_ids": torch.tensor([[10, 20, 30, 40, 50, 60, 70, 80, 0, 0]]),
  "attention_mask": torch.tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])
}

outputs = model(**inputs)

print(outputs.last_hidden_state.shape)
print(outputs.last_hidden_state[:, 1:4, 1:4])

#=> torch.Size([1, 10, 64])
#=> tensor([[[ 0.0033, -0.2547,  0.4954],
#=>          [-1.5348, -1.5433,  0.4846],
#=>          [ 0.7795, -0.3995, -0.9499]]], grad_fn=<SliceBackward0>)

I believe there are a few differences between MPNet and BERT, so we need to align the implementation accordingly. In particular, by a quick look some layer names differ, for example key->k, value->v, query->q, so we need to update the layer mapping as well :)

[snewcomer]

Thanks a ton! Regarding your last comment, are you looking here?

https://github.com/huggingface/transformers/blob/main/src/transformers/models/mpnet/modeling_mpnet.py

[jonatanklosko]

Yes! And the Bert implementation is https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py, which may be helpful for differences.

[snewcomer]

ok made another round of improvements! thx for the direction.

[jonatanklosko]

Nitpick: I think we want the name to be MpNet to align with our naming conventions. Basically, in acronyms we capitalize only the first letter, as in BERT -> Bert, RoBERTa -> Roberta. And we capitalize on each word, such as ResNet, ConvNext. We do this, because the reference names are often arbitrarily capitalized, and it's not ergonomic for library users to know the exact capitalization.