Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Feature]: Allow specifying fallbacks per key #8054

Open
ishaan-jaff opened this issue Jan 28, 2025 · 1 comment
Open

[Feature]: Allow specifying fallbacks per key #8054

ishaan-jaff opened this issue Jan 28, 2025 · 1 comment
Labels
enhancement New feature or request help wanted Extra attention is needed management endpoints /key, /team, /user endpoints mlops user request router

Comments

@ishaan-jaff
Copy link
Contributor

The Feature

Allow specifying fallbacks per virtual key

Desired flow:

  • Allow specifying fallbacks as an arg to /key/generate - ideally we store this in the key metadata

  • Make a /chat/completions request to a bad model, expect to use fallbacks

  • here is how we allow fallbacks on litellm proxy https://docs.litellm.ai/docs/proxy/reliability

Motivation, pitch

Are you a ML Ops Team?

Yes

Twitter / LinkedIn details

No response
@ishaan-jaff ishaan-jaff added enhancement New feature or request help wanted Extra attention is needed management endpoints /key, /team, /user endpoints router labels Jan 28, 2025
@bashketchum02
Copy link

Hello, I would like to contribute and work on this issue, if no one else is working on it 😄

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
enhancement New feature or request help wanted Extra attention is needed management endpoints /key, /team, /user endpoints mlops user request router
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ishaan-jaff @bashketchum02