Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

helicone-stream-force-format: true seems to cause JSON::ParserError when using Azure OpenAI #411

Open
rohitpaulk opened this issue Dec 19, 2023 · 1 comment
Labels

Comments

@rohitpaulk
Copy link
Contributor

Describe the bug

When working with the Azure OpenAI service, we seem to be running into occasional "JSON::ParserError" errors. These don't happen frequently, but if you run a script that runs 100 prompts, it almost always triggers the error.

We tracked this down to the "helicone-stream-force-format" parameter suggested in the README. When that's present, the bug triggers. When it isn't, the bug doesn't trigger.

The error looks something like this:

unexpected token at '{"finish_reason":null,"index":0,"deltdata: {"id":"chatcmpl-8XckshxV3cEBasKbvvUD4MDSpUgYJ","object":"chat.completion.chunk","created":1703024490,"model":"gpt-35-turbo","choices":[{"finish_reason":null,"index":0,"delta":{"content":"."}}]}' (JSON::ParserError)

To Reproduce

Run this script:

client = OpenAI::Client.new(
  access_token: ENV.fetch("AZURE_OPENAI_API_KEY"),
  uri_base: "https://oai.hconeai.com/openai/deployments/gpt-35-turbo",
  request_timeout: 240,
  extra_headers: {
    "api-key": ENV.fetch("AZURE_OPENAI_API_KEY"),
    "Helicone-Auth": "Bearer #{ENV.fetch("HELICONE_API_KEY")}",
    "Helicone-OpenAI-Api-Base": ENV.fetch("AZURE_OPENAI_ENDPOINT"),
    "Helicone-Property-Prompt": "TestUnexpectedTokenErrorsPrompt",
    "helicone-stream-force-format": "true"
  },
  api_type: :azure,
  api_version: "2023-03-15-preview"
)

100.times.peach(8) do |i|
  puts ""
  puts "Iteration #{i}"
  puts ""

  client.chat(
    parameters: {
      model: "gpt-3.5-turbo",
      messages: [
        {role: "user", content: "Write me 3 paragraphs about the topic 'OpenAI parsing needs to be fixed'"}
      ],
      response_format: {type: "text"},
      seed: i, # Default, can be overridden using parameters
      stream: proc do |chunk, _bytesize|
        if chunk.dig("choices", 0, "finish_reason") == "stop" # last message
          puts ""
          puts "Iteration #{i} finished."
          puts ""

          next
        end

        content = chunk.dig("choices", 0, "delta", "content")
        unless content.nil?
          $stdout.write(".")
        end
      end
    }
  )
end

Expected behavior

JSON::ParserError shouldn't be triggered.

@rohitpaulk
Copy link
Contributor Author

rohitpaulk commented Dec 19, 2023

Looks like the "helicone-stream-force-format" header was added in #251, and the work in #338 possibly negates the need for this?

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants