Skip to content

0.2.0
Compare
Choose a tag to compare
@tox-p tox-p released this 06 Apr 07:25
0.2.0
df17669

Features

  • Add encodeOrdinary and countTokensOrdinary methods to Encoding.
    • The existing encode and countTokens method currently throw an exception if a special token is encountered. This change introduced encodeOrdinary which simply encodes special tokens as if they were normal text.
  • Add getEncodingForModel(String) to EncodingRegistry to allow retrieving encodings for models by their string name.
  • It is now possible to call EncodingRegistry#getEncodingForModel(String) with a snapshot of a model, for example "gpt-4-0314" and receive the correct encoding.

Full Changelog: 0.1.0...0.2.0

Assets 2
Loading