[Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK #8066

EnzoGehlen · 2025-01-28T19:00:41Z

What happened?

A bug happened!

Hi there, i'm using LiteLLM's proxy in a docker container to test it out. As the client, i'm using Vercel SDK client library for nodejs. Works pretty well for averall chat.

However, when trying to call some tool declared in a stream mode, the proxy return a stream without the proper tool call. I'm having this issue with the Anthropic provider only, but it not seems to be a problem on their side because when running direclty to their API, everything works fine. Trying the same on an OPENAI model works too, even with toolcall ~ streams.

For test purposes, i've run a tcpdump with the same prompt and get different stream-structure results. Here what i receive when calling Anthropic direct API asking for it to run some tool:

Anthropic API stream response

`HTTP/1.1 200 OK
Server: nginx/1.27.0
Date: Tue, 28 Jan 2025 17:57:58 GMT
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-cache
anthropic-ratelimit-requests-limit: 4000
anthropic-ratelimit-requests-remaining: 3999
anthropic-ratelimit-requests-reset: 2025-01-28T17:57:57Z
anthropic-ratelimit-input-tokens-limit: 400000
anthropic-ratelimit-input-tokens-remaining: 387000
anthropic-ratelimit-input-tokens-reset: 2025-01-28T17:57:59Z
anthropic-ratelimit-output-tokens-limit: 80000
anthropic-ratelimit-output-tokens-remaining: 73000
anthropic-ratelimit-output-tokens-reset: 2025-01-28T17:58:02Z
anthropic-ratelimit-tokens-limit: 480000
anthropic-ratelimit-tokens-remaining: 460000
anthropic-ratelimit-tokens-reset: 2025-01-28T17:57:59Z
request-id: req_xxxxxx
via: 1.1 google
cf-cache-status: DYNAMIC
X-Robots-Tag: none
CF-RAY:x76z7x6z7x67

event: message_start
data: {"type":"message_start","message":{"id":"msg_01Wi6EU3sjVax1ui9YCRvuxA","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":7284,"cache_creation_input_tokens":0,"cache_read_input_tokens":0,"output_tokens":2}} }

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""} }

event: ping
data: {"type": "ping"}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"I'm g"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"oing to find for ZTE t"} }

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"o you now"} }

event: content_block_stop
data: {"type":"content_block_stop","index":0 }

event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"toolu_01YAd3MYgH4CQ2LywHjt9pah","name":"searchDevices","input":{}} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{"desc"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"rip"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"ti"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"on": "Sea"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"rching dev"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"ice"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":" ZTE""} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":", "filters""} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":": ["} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{"acc"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"essor": ""} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"devi"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"ceInfo"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":".manu"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"facturer", "}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""operator""} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":": "equal"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"", "va"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"lue": "ZTE"} }

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":""}]}"} }

event: content_block_stop
data: {"type":"content_block_stop","index":1 }

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"tool_use","stop_sequence":null},"usage":{"output_tokens":115} }

event: message_stop
data: {"type":"message_stop" }`

LiteLLM Stream response:

`HTTP/1.1 200 OK
date: Tue, 28 Jan 2025 18:05:06 GMT
server: uvicorn
content-type: text/event-stream; charset=utf-8
transfer-encoding: chunked

event: message_start
data:{"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-3-5-sonnet-20240620", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}}

event: content_block_start
data:{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "I'm g"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "oing to find for ZTE t"}}

event: content_block_delta
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": "o you now"}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_stop
data:{"type": "content_block_stop", "index": 0}

event: message_delta
data:{"type": "message_delta", "delta": {"stop_reason": "tool_use"}, "usage": {"input_tokens": 0, "output_tokens": 0}}

event: message_stop
data:{"type": "message_stop"}`

And not matter how or how much times i ask it to run the tool, it always return just the text with a bunch of empty strings
I've dumped this direct response from Anthropic using another Nginx proxy that forwards all my http traffic (to see into the wire) to their https API, that's how i now there is something off with LiteLLM's proxy.

When working with the stream=false option, the problem does not occurs, and the tool calls are being done properly

Relevant log output

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

main-stable

Twitter / LinkedIn details

No response

krrishdholakia · 2025-01-29T01:39:02Z

hey @EnzoGehlen can you share a script i can run to repro this issue?

krrishdholakia · 2025-01-29T01:43:33Z

unable to repro @EnzoGehlen

CURL

Just ran this

curl -L -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
  "model": "anthropic-claude",
  "messages": [
    {
      "role": "user",
      "content": "What'\''s the weather like in Boston today?"
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"]
            }
          },
          "required": ["location"]
        }
      }
    }
  ],
  "tool_choice": "auto",
  "stream": true
}'

RESPONSE

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"I'll","role":"assistant"}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" help you check the current weather in Boston."}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" I'll retrieve the current weather information"}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" using the get_current_weather function"}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"."}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"id":"toolu_01GVQJwXRjHkF75kYMfCss9v","function":{"arguments":"","name":"get_current_weather"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114940,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":""},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"{\""},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"locati"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"on\":"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":" \"Boston, "},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"MA\""},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":", \"un"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"it\": \""},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"fahrenh"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"","tool_calls":[{"function":{"arguments":"eit\"}"},"type":"function","index":0}]}}]}

data: {"id":"chatcmpl-c243f6cf-768c-472b-ba10-7f6637578290","created":1738114941,"model":"claude-3-5-haiku-20241022","object":"chat.completion.chunk","choices":[{"finish_reason":"tool_calls","index":0,"delta":{}}]}

data: [DONE]

EnzoGehlen · 2025-01-29T11:35:41Z

@krrishdholakia You can reproduce using this example curl from anthropics API reference, using /v1/messages endpoint

curl -L -X POST 'http://0.0.0.0:4000/v1/messages' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-d '{
  "model": "claude-3-5-sonnet-20241022",
  "max_tokens": 1024,
  "tools": [
    {
      "name": "get_weather",
      "description": "Get the current weather in a given location",
      "input_schema": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA"
          }
        },
        "required": ["location"]
      }
    }
  ],
  "tool_choice": {"type": "any"},
  "messages": [
    {
      "role": "user",
      "content": "What is the weather like in San Francisco?"
    }
  ],
  "stream": true
}'

event: message_start
data:{"type": "message_start", "message": {"id": "msg_1nZdL29xx5MUA1yADyHTEsnR8uuvGzszyY", "type": "message", "role": "assistant", "content": [], "model": "claude-3-5-sonnet-20240620", "stop_reason": null, "stop_sequence": null, "usage": {"input_tokens": 25, "output_tokens": 1}}}

event: content_block_start
data:{"type": "content_block_start", "index": 0, "content_block": {"type": "text", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_delta
data:{"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": ""}}

event: content_block_stop
data:{"type": "content_block_stop", "index": 0}

event: message_delta
data:{"type": "message_delta", "delta": {"stop_reason": "tool_use"}, "usage": {"input_tokens": 0, "output_tokens": 0}}

event: message_stop
data:{"type": "message_stop"}

EnzoGehlen · 2025-01-31T13:04:59Z

@krrishdholakia have you been able to repro this?

reymondzzzz · 2025-02-05T07:42:48Z

Hi! I have same issue. I happened with long arguments for tool_call.

EnzoGehlen added the bug Something isn't working label Jan 28, 2025

krrishdholakia changed the title ~~[Bug]: ToolCall not working properly when stream=true with Anthropic provider~~ [Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK Jan 29, 2025

krrishdholakia self-assigned this Jan 29, 2025

krrishdholakia added unable to repro awaiting: user response labels Jan 29, 2025

devin-ai-integration bot linked a pull request Jan 30, 2025 that will close this issue

fix: tool call streaming for anthropic provider IXC-Soft/litellm-devin#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK #8066

[Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK #8066

EnzoGehlen commented Jan 28, 2025 •

edited

Loading

krrishdholakia commented Jan 29, 2025

krrishdholakia commented Jan 29, 2025

EnzoGehlen commented Jan 29, 2025 •

edited

Loading

EnzoGehlen commented Jan 31, 2025

reymondzzzz commented Feb 5, 2025

[Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK #8066

[Bug]: ToolCall not working properly when stream=true with Anthropic provider with Vercel AI SDK #8066

Comments

EnzoGehlen commented Jan 28, 2025 • edited Loading

What happened?

A bug happened!

Anthropic API stream response

LiteLLM Stream response:

Relevant log output

Are you a ML Ops Team?

What LiteLLM version are you on ?

Twitter / LinkedIn details

krrishdholakia commented Jan 29, 2025

krrishdholakia commented Jan 29, 2025

CURL

RESPONSE

EnzoGehlen commented Jan 29, 2025 • edited Loading

EnzoGehlen commented Jan 31, 2025

reymondzzzz commented Feb 5, 2025

EnzoGehlen commented Jan 28, 2025 •

edited

Loading

EnzoGehlen commented Jan 29, 2025 •

edited

Loading