Skip to content

Possibility of adding beta header in Claude #85

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
luismiguelamado opened this issue Apr 1, 2025 · 2 comments · May be fixed by #105
Open

Possibility of adding beta header in Claude #85

luismiguelamado opened this issue Apr 1, 2025 · 2 comments · May be fixed by #105
Labels
enhancement New feature or request

Comments

@luismiguelamado
Copy link

Hello,

Thank you for such a great library! I'm really enjoying using it.

I'm encountering a limitation with Claude 3.7 regarding the output token limit of 8192 tokens.

There is a beta header that can be added to increase the output tokens. According to Anthropic's [documentation](https://docs.anthropic.com/en/docs/about-claude/models/all-models), "Include the beta header output-128k-2025-02-19 in your API request to increase the maximum output token length to 128k tokens for Claude 3.7 Sonnet."

Is there a possibility of adding this header? Perhaps during the configuration, we could have a flag to enable or disable the header.

Thank you in advance!

@crmne
Copy link
Owner

crmne commented Apr 1, 2025

Hey @luismiguelamado thank you so much!

I'd love to see that added to the library! Honestly, that header should be the default and I would go without configuration on this one.

Regarding the implementation: I did something similar - changing something only for a model - for the o1 and o3 series of models here:

def complete(messages, tools:, temperature:, model:, &block) # rubocop:disable Metrics/MethodLength
normalized_temperature = if capabilities.respond_to?(:normalize_temperature)
capabilities.normalize_temperature(temperature, model)
else
temperature
end

A similar implementation would be much appreciated!

@crmne crmne added the enhancement New feature or request label Apr 1, 2025
jimjimovich pushed a commit to jimjimovich/ruby_llm that referenced this issue Apr 8, 2025
This change implements support for Anthropic's `output-128k-2025-02-19` beta header, specifically for the `claude-3-7-sonnet-20250219` model. Including this header allows users to leverage the increased 128k maximum output token limit available for this model.

This implementation attempts to follow the suggested pattern for provider-specific capabilities, similar to the existing temperature normalization logic:
- A new method `additional_headers_for_model` was added to `Anthropic::Capabilities` to encapsulate the provider-specific logic.
- The shared `Provider#post` method was updated to conditionally call this capability method via `respond_to?` and merge the resulting headers before making the API request.
- The necessary `model_id` context is now passed down through the relevant methods (`embed`, `paint`, `sync_response`, `stream_response`) to `post`.

This approach avoids overriding shared methods within the provider module and keeps the specific logic contained within the capabilities definition.

Closes crmne#85
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants
@crmne @luismiguelamado and others