Support Cross encoder models

[maxdebayser]

This PR adds support for Cross Encoder models reusing most of what was done to support embedding models. It includes:

• Support for RoBERTA SequenceClassification models, tested with BAAI/bge-reranker-v2-m3
• Support for BERT SequenceClassification models, tested with cross-encoder/ms-marco-MiniLM-L-6-v2
• A score API endpoint
• A LLM.score() method
• Support for GPU and CPU

An example of an API call is:

$ curl -X 'POST'   'http://localhost:8000/v1/score'   -H 'accept: application/json'   -H 'Content-Type: application/json'   -d '{
  "model": "BAAI/bge-reranker-v2-m3",
  "text_1": "What is the capital of France?",
  "text_2" : ["The capital of France is Paris."]
}

Response:

{
  "id": "score-64dcf95966094892b0f0ccc63637c6d2",
  "object": "list",
  "created": 239128,
  "model": "BAAI/bge-reranker-v2-m3",
  "data": [
    {
      "index": 0,
      "object": "score",
      "score": [
        1.0
      ]
    }
  ],
  "usage": {
    "prompt_tokens": 0,
    "total_tokens": 0,
    "completion_tokens": 0,
    "prompt_tokens_details": null
  }
}

Instead of adding a new task type, the models in this PR are loaded with task == "embedding" but support the additional is_cross_encoder property. This allows us to reuse most of what was done the embedding support. The only changes required to the core layers between the API and LLM is the propagation the token_type_ids list. This is an additional tokenizer output that is generated for the BERT models and has to be passed to the model's embedding layer.

cc: @DarkLight1337 @flaviabeo

FIX #8022

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Can you add an example script to show how to use this API?

QQ: Why do we have to define a separate "cross_encoding" task for this? I think we can keep using "embedding" task if we just override the pooler method instead of defining a new classification_output method.

[maxdebayser]

Can you add an example script to show how to use this API?

Yes, I've added one in the PR description but it's a good idea to add it to the documentation.

QQ: Why do we have to define a separate "cross_encoding" task for this? I think we can keep using "embedding" task if we just override the pooler method instead of defining a new classification_output method.

I thought about this, and the only reason I kept it that way was that in the serving layer I need to know what task is being done because I need to call the tokenizer with tokenizer(text=text1, text_pair=text2). If instead of reusing the chat embeddings API we had a new endpoint just for cross encoding, this wouldn't be necessary. Or perhaps we can add an attribute to some of the config classes to tell the although the task is "embedding", the model is actually a "BertModelForSequenceClassification"

[DarkLight1337]

I think it may be simpler to make this a separate flag, similar to how we have a flag for multimodal models, rather than creating a new task for it. That way, we won't have to change our internals at all.

[DarkLight1337]

I think having a separate API for this would be cleaner as well - perhaps a Scoring API where we output a single score?

[maxdebayser]

I've removed the "cross_encoding" task and added a is_cross_encoder property to the ModelConfig class. I've also added support for Roberta models.

Pending TODOs:

• Add Scoring API
• Support for cross encoding in the LLM class
• Tests comparing with sentence-transformers
• CPU support

[maxdebayser]

I've added a score() method to the LLM class and added tests for it. I've also fixed the CPU support.

Pending TODOs:

• Add Scoring API
• Test Scoring API comparing with sentence-transformers

[DarkLight1337]

You should re-enable yapf afterwards.

[DarkLight1337]

Otherwise looks good, thanks for your effort and patience!

[DarkLight1337]

Wait, we still need to update the OpenAI page under /docs

[maxdebayser]

The tests that aren't failing do to hugginface.co timeout are doing so because of a JSON serialization problem. I've opened a tentative fix here: #10580

[flaviabeo]

@DarkLight1337 I have included in this last commit of mine, some documentation for the score API in the OpenAI Compatible Server page. Would this be the right place to add it?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maxdebayser.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

#10581 should fix it, please merge from main again.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @maxdebayser.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

@DarkLight1337 I have included in this last commit of mine, some documentation for the score API in the OpenAI Compatible Server page. Would this be the right place to add it?

I think we can move this up one section so that parameters are shown after all API mentions.