Improved stability of litellm models for reasoning models. #538
Conversation
|
Hey @JoelNiklaus, since you have made most commits around LiteLLM.
|
|
Hi @satpalsr,
|
|
@JoelNiklaus Thanks. I'll just drop it as separate issue for anyone to pick. |
| kwargs["caching"] = False | ||
| logger.info("Response is empty, retrying without caching") | ||
| response = litellm.completion(**kwargs) | ||
|
|
||
| if content and "<think>" in content: | ||
| logger.debug(f"Removing <think> tags from response: {content}") |
There was a problem hiding this comment.
Why are we removing think tags from the answer here ? I think it should be done in the metric function no ?
There was a problem hiding this comment.
If we are evaluating a reasoning model the grader will look at the thinking tokens unless we remove them. We would need to remove them in every metric function otherwise.
There was a problem hiding this comment.
yeah but in that case you lose the thinking traces in the details.
What we would need:
- keep the thinking traces in the details
- allow the user to choose wether or not to evaluate with thinking tags
There was a problem hiding this comment.
True. Maybe we can open an issue for that and add that improvement in a later PR?
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
No description provided.