Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

CreateChatCompletionRequest.max_tokens should be u32 #232

Closed
MakotoE opened this issue Jun 11, 2024 · 1 comment · Fixed by #233
Closed

CreateChatCompletionRequest.max_tokens should be u32 #232

MakotoE opened this issue Jun 11, 2024 · 1 comment · Fixed by #233
Labels
bug Something isn't working

Comments

@MakotoE
Copy link
Contributor

MakotoE commented Jun 11, 2024 •
edited
Loading

CreateChatCompletionRequest.max_tokens is a Option<u16> as of 0.23.1.

The newer models such as gpt-4o has a context window of 128,000 tokens. This context window limit is the sum of input and output tokens.

I believe the max_tokens field should be Option<u32> to allow numbers as high as 128,000.
@64bit
Copy link
Owner

64bit commented Jun 11, 2024

Thank you for reporting the bug, a PR is most welcome!

@64bit 64bit added the bug Something isn't working label Jun 11, 2024
@MakotoE MakotoE mentioned this issue Jun 11, 2024
Merged
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change max_tokens type to Option<u32>
2 participants
@64bit @MakotoE