Skip to content

Add OpenAI.rough_token_count class method #312

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 2 commits into from
Aug 20, 2023

Conversation

jamiemccarthy
Copy link
Contributor

@jamiemccarthy jamiemccarthy commented Aug 18, 2023

It can be useful to estimate the token count of some text before sending it to OpenAI. This PR adds an OpenAI.rough_token_count class method which does this.

To perform the estimation, it uses the two "helpful rules of thumb" in OpenAI's docs and takes the arithmetic average.

The empty string returns 0, and any other string returns an integer greater than 0.

This is a rewrite of #306 to remove the Tiktoken functionality and change the interface as requested.

All Submissions:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?

@jamiemccarthy jamiemccarthy mentioned this pull request Aug 18, 2023
3 tasks
Copy link
Owner

@alexrudall alexrudall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks a lot for rewriting. Just some documentation needed, would you mind adding a section to the README with a couple lines briefly explaining how to estimate tokens in a string and link to the OpenAI docs. Probably between Azure and Models sections I think?

And couple inline comments also, just with the method.

@alexrudall
Copy link
Owner

This is perfect thanks @jamiemccarthy!

@alexrudall alexrudall merged commit 51408cc into alexrudall:main Aug 20, 2023
@jamiemccarthy jamiemccarthy deleted the jm-rough-token-count branch August 20, 2023 19:51
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants