[Feature]: Groq - deepseek-r1-distill-llama-70b

[JamesHutchison]

The Feature

This issue is to add deepseek R1 hosted on groq cloud

Motivation, pitch

It fast and it good. Apparently.

Are you a ML Ops Team?

No

Twitter / LinkedIn details

No response

[miraclebakelaser]

As with all API providers that have OpenAI compatibility, you can get around LiteLLM model incompatibility issues by accessing the API provider through the OpenAI API format.

The litellm parameters are:

    model="openai/deepseek-r1-distill-llama-70b", # note the `openai/` prefix
    api_base="https://api.groq.com/openai/v1",
    api_key=os.getenv("GROQ_API_KEY"),

LiteLLM OpenAI Compatible-Endpoint Docs
Groq OpenAI Compatibility Docs

I've run these parameters on my end and the r1-distill works.

[ishaan-jaff]

What is the LiteLLM issue you are running into here ?

[miraclebakelaser]

What is the LiteLLM issue you are running into here?

Ah, one correction here. I assumed that the user was running into issues with the litellm.completion(model="groq/deepseek-r1-distill-llama-70b") because the model wasn't in model_prices_and_context_window.json . But there's actually no issue when running that request. Thus, my original "fix" doesn't fix anything. My original assessment of the problem was based on wonky local configs I have set up.

That said, I do an error when running litellm.cost_calculator.cost_per_token(model="groq/deepseek-r1-distill-llama-70b"):

ValueError                                Traceback (most recent call last)
File ~/GitHub/litellm/litellm/utils.py:4231, in _get_model_info_helper(model, custom_llm_provider)
   [4230](~/GitHub/litellm/litellm/utils.py:4230) if _model_info is None or key is None:
-> [4231](~/GitHub/litellm/litellm/utils.py:4231)     raise ValueError(
   [4232](~/GitHub/litellm/litellm/utils.py:4232)         "This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json"
   [4233](~/GitHub/litellm/litellm/utils.py:4233)     )
   [4235](~/GitHub/litellm/litellm/utils.py:4235) _input_cost_per_token: Optional[float] = _model_info.get(
   [4236](~/GitHub/litellm/litellm/utils.py:4236)     "input_cost_per_token"
   [4237](~/GitHub/litellm/litellm/utils.py:4237) )

ValueError: This model isn't mapped yet. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json

[JamesHutchison]

Ah, I didn't realize it would work. The documentation has a table so it wasn't clear to me that newly listed models should work "out of the box":

https://docs.litellm.ai/docs/providers/groq#supported-models---all-groq-models-supported

I just assumed it wouldn't work yet and was proactively making this issue in anticipation of using it.