[Bug]: Anthropic claude-3-7-sonnet issues pack

[mitya52]

What happened?

Hi!

I'm implementing claude-3-7-sonnet support in Refact.ai.
We faced some issues with litellm's implementation, here is the list:

• streaming mode with thinking doesn't return signature, see this PR Fix missing signature_delta in thinking blocks streamed from Claude 3.7 #8797
• by some reason in streaming mode deltas contain "signature_delta" field instead of actual "signature" in Anthropic format
• when thinking mode is enabled stream contains empty tool_call after thinking part ChatCompletionDeltaToolCall(id=None, function=Function(arguments='{}', name=None), type='function', index=-1)
• litellm.get_max_tokens("claude-3-7-sonnet-20250219") returns 8k but the new model supports 128k see https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#extended-output-capabilities-beta

BTW thanks for your work, litellm is a great project!

Relevant log output

Delta(reasoning_content='The user has sent', thinking_blocks=[{'type': 'thinking', 'thinking': 'The user has sent', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': 'The user has sent', 'signature_delta': None}]}, content='', role='assistant', function_call=None, tool_calls=None, audio=None)
Delta(reasoning_content=' a simple greeting "hi!', thinking_blocks=[{'type': 'thinking', 'thinking': ' a simple greeting "hi!', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': ' a simple greeting "hi!', 'signature_delta': None}]}, content='', role=None, function_call=None, tool_calls=None, audio=None)
Delta(reasoning_content='". I should respond in a friendly an', thinking_blocks=[{'type': 'thinking', 'thinking': '". I should respond in a friendly an', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': '". I should respond in a friendly an', 'signature_delta': None}]}, content='', role=None, function_call=None, tool_calls=None, audio=None)
Delta(reasoning_content='d welcoming manner, introducing myself and offering', thinking_blocks=[{'type': 'thinking', 'thinking': 'd welcoming manner, introducing myself and offering', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': 'd welcoming manner, introducing myself and offering', 'signature_delta': None}]}, content='', role=None, function_call=None, tool_calls=None, audio=None)
Delta(reasoning_content=' to help with anything they might need.', thinking_blocks=[{'type': 'thinking', 'thinking': ' to help with anything they might need.', 'signature_delta': None}], provider_specific_fields={'thinking_blocks': [{'type': 'thinking', 'thinking': ' to help with anything they might need.', 'signature_delta': None}]}, content='', role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content='', role='assistant', function_call=None, tool_calls=[ChatCompletionDeltaToolCall(id=None, function=Function(arguments='{}', name=None), type='function', index=-1)], audio=None)
Delta(provider_specific_fields=None, content="Hello there! It's nice to meet you.", role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=" I'm an AI assistant ready", role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=' to help with information, answer', role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=' questions, or just chat. How', role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=' are you doing today? Is there something', role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=' specific I can help you with?', role=None, function_call=None, tool_calls=None, audio=None)
Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None)

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.62.1

Twitter / LinkedIn details

No response

[krrishdholakia]

by some reason in streaming mode deltas contain "signature_delta" field instead of actual "signature" in Anthropic format

hey @mitya52 so i see Anthropic returning signature_delta - how would you want us to handle this?

[mitya52]

@krrishdholakia thanks for clarification! But what about another issues?