feat: add OpenAI's new structured output API

[monotykamary]

What does this PR do?:

• Add json_schema Field to ChatOpenAI and Enhance Response Format Handling
• Cleans up a handful of tests for mix test --include live_open_ai

Summary:

This pull request introduces a new json_schema field to the ChatOpenAI module, which enhances the response format handling capabilities by allowing for JSON Schema-based validation.

Changes:

1. Field Definition:

   • Added field :json_schema, :map, default: nil to the ChatOpenAI struct.
   • Updated the list of valid struct keys to include :json_schema.

2. Response Format Handling:

   • Modified the set_response_format function to account for cases when json_response is true and json_schema is provided.
     • If json_response is true and json_schema is not nil, the response format is set to include json_schema.
     • If json_response is true but json_schema is nil, the response format defaults to json_object.
     • If json_response is false, the response format is set to text.

Code Details:

1. Field Addition:

   field :json_schema, :map, default: nil

2. Response Format Modification:

   defp set_response_format(%ChatOpenAI{json_response: true, json_schema: json_schema}) when not is_nil(json_schema) do
     %{
       "type" => "json_schema",
       "json_schema" => json_schema
     }
   end
   
   defp set_response_format(%ChatOpenAI{json_response: true}) do
     %{"type" => "json_object"}
   end
   
   defp set_response_format(%ChatOpenAI{json_response: false}) do
     %{"type" => "text"}
   end

[brainlid]

Thanks @monotykamary for the PR!

From what I see, it looks like your primary goal here is to attach a json_schema to the OpenAI module. However, the schema is not sent to OpenAI, it is intended to be optionally used by the developer later. It's basically a "hold on to this because it's relevant and I may need it later." The ultimate goal seems to be to link the schema to the request so it can be used later to validate generated responses for schema compliance.

Is that right? Or did I miss something?

If that's right, then I'm inclined to have that live on the LLMChain so other models benefit as well. Or optionally in a MessageProcessor for schema compliance.

Oh, and thanks for fixing those bad tests! 😊 How did those get in there? 😅

[monotykamary]

It's essentially to implement structured outputs from OpenAI, which is more like json_object, but we specify the json schema inside the response format instead of inside the prompt: https://platform.openai.com/docs/guides/structured-outputs/examples

However, the schema is not sent to OpenAI

Wait, how? 🤔 We are using this exact code to structure our outputs for a scraper we have.

[ksanderer]

OAI Structured Output requires the response_format field. The model will use the schema during the inference phase to sample generated tokens only from the "allowed" tokens.

https://platform.openai.com/docs/api-reference/chat/create#chat-create-response_format

[brainlid]

Awesome! Thanks for walking me through that and thank you for the contribution!

❤️💛💙💜

[lud-wj]

Do you know if this is going to be released anytime soon ?

[brainlid]

Hi @lud-wj, I keep intending to make a release soon. It will probably be a new RC because there's still a couple changes I want to make. That and I'm procrastinating writing the CHANGELOG. 😆

[lud-wj]

Alright, thank you!

[vkryukov]

That and I'm procrastinating writing the CHANGELOG. 😆

Don't forget that this is a package for working with LLMs! Here is something to get you started: I asked Claude to write the summary change log for git log v0.3.0-rc.0..HEAD --pretty="%s" in the format of CHANGELOG.md. I have no idea whether that's accurate or not, but just to give you a starting point :).

## v0.3.0 (Unreleased)

**Major Features:**
* Added AWS Bedrock support for Anthropic Claude integration
* Added support for OpenAI's new structured output API
* Added support for fallback handling when LLM services are overloaded
* Improved tool/function handling across providers:
  - Strict mode for tools
  - Better support for tool calls with parameters
  - Improved error handling for function execution
  - Support for functions with no parameters in Google AI

**Improvements:**
* Enhanced Google AI Support:
  - Added safety settings configuration
  - Added system instructions support
  - Fixed handling of empty text parts and token usage
  - Better handling of finish reasons
* Improved streaming reliability:
  - Fixed streaming issues with Azure OpenAI Service
  - Fixed OpenAI stream decode issues
  - Fixed Ollama streaming response
* Better error handling and recovery:
  - Improved handling of "Too many requests" from AWS Bedrock
  - Better handling of overloaded service responses
  - Enhanced function execution failure responses

**Breaking Changes:**
* Changed return value structure of `LLMChain.run/2`
* Added "processed_content" to ToolResult struct

**Documentation:**
* Improved documentation for all LLM providers
* Added examples for tool_choice in OpenAI and Anthropic
* Updated configuration documentation for API keys
* Added new examples and notebooks for image content chats

[brainlid]

Ha! You rock @vkryukov! I used your approach as a starting point and I'm prepping an RC.1 right now.

[brainlid]

Published the new RC release

[lud-wj]

You can use git cliff to generate nice changelogs, since you are already using commit prefixes like feat:, fix, etc.