Prompt Caching

[tpaulshippy]

What this does

Support prompt caching config in both Anthropic and Bedrock providers for Claude models that support it. And report prompt caching token counts for OpenAI and Gemini which cache automatically.

Caching system prompts:

chat = RubyLLM.chat
chat.with_instructions("You are a helpful assistant.")
chat.cache_prompts(system: true)

Caching user prompts:

chat = RubyLLM.chat
chat.with_instructions("You are a helpful assistant.")
chat.cache_prompts(system: true, user: true)
chat.ask("What is the capital of France?")

Caching tool definitions:

chat = RubyLLM.chat
chat.with_instructions("You are a helpful assistant.")
chat.with_tool(MyTool)
chat.cache_prompts(tools: true)
chat.ask("What is the capital of France?")

Type of change

• New feature

Scope check

• I read the Contributing Guide
• This aligns with RubyLLM's focus on LLM communication
• This isn't application-specific logic that belongs in user code
• This benefits most users, not just my specific use case

Quality check

• I ran overcommit --install and all hooks pass
• I tested my changes thoroughly
• I updated documentation if needed
• I didn't modify auto-generated files manually (models.json, aliases.json)

API changes

• New public methods/classes

Related issues

Resolves #13

[tpaulshippy]

@crmne As I don't have an Anthropic key, I'll need you to generate the VCR cartridges for that provider. Hoping everything just works, but let me know if not.

[crmne]

@tpaulshippy this would be great to have! Will you be willing to enable it on all providers?

I'll do a proper review when I can.

[tpaulshippy]

My five minutes of research indicates that at least OpenAI and Gemini take the approach of automatically caching for you based on the size and structure of your request. So the only support I think we'd really need for those two is to populate the cached token counts on the response messages. Unless we want to try to support explicit caching on the Gemini API but that looks complex and not as commonly needed.

Do you know of other providers that require payload changes for prompt caching?

[tpaulshippy]

Realizing this might cause errors on older models that do not support caching. If it does, we could raise here, or just let the API validation handle it. I'm torn on whether the capabilities check complexity is worth it as these models are probably so rarely used.

[tpaulshippy]

@crmne As I don't have an Anthropic key, I'll need you to generate the VCR cartridges for that provider. Hoping everything just works, but let me know if not.

Scratch that. I decided to stop being a cheapskate and just pay Anthropic their $5.

[tpaulshippy]

Looking to implement this in our project and now I'm wondering if it should be an opt out rather than an opt in. If you are using unique prompts every time I guess it adds some cost to cache them but my guess is in most applications prompts will get repeated, especially system prompts.

[crmne]

Thank you for this feature @tpaulshippy, however there are several improvements I'd like you to make before we merge this.

On top of the ones made in the comments, and the most important one, I'd like to have prompt caching implemented in all providers.

Plus I have not fully checked the logic in providers/anthropic but the patch seems a bit heavy-handed with the amount of changes needed at first glance. Where all changes necessary or could it be done in a simpler manner?

[tpaulshippy]

I'd like to have prompt caching implemented in all providers.

Did you see this? Is the request to populate the cached token counts on the response messages for OpenAI and Gemini?

[crmne]

Did you see this? Is the request to populate the cached token counts on the response messages for OpenAI and Gemini?

Thank you for pointing that out, I had missed it. I think it would certainly be a nice addition to RubyLLM to have all providers have almost the same level of support of caching.

[tpaulshippy]

Did you see this? Is the request to populate the cached token counts on the response messages for OpenAI and Gemini?

Thank you for pointing that out, I had missed it. I think it would certainly be a nice addition to RubyLLM to have all providers have almost the same level of support of caching.

Ok we have a bit of a naming issue. Here's the property names we get from each provider:

Anthropic
cache_creation_input_tokens
cache_read_input_tokens

OpenAI
cached_tokens

Gemini
cached_content_token_count

My reading of the docs indicates that the OpenAI and Gemini values correspond pretty closely with the cache_read_input_tokens in Anthropic.

What should we call these properties in the Message?

[crmne]

For the naming, let's go with:

• cached_tokens - maps to the cache read values from all providers (the main property developers will use)
• cache_creation_tokens - Anthropic-specific cache creation cost (nil for other providers)

This keeps it consistent with our existing input_tokens/output_tokens pattern while handling the provider differences cleanly.

Can you update the Message properties to use these names? Thanks Paul!

[tpaulshippy]

Had to play with the length/complexity of the prompts. Finally got them to cache. Should be ready to go.

[jordanyeo]

This will be huge for me- possible to get a review and merge? Thanks!

[tpaulshippy]

I rolled this out 2 weeks ago. So far the numbers show that I am paying 55% less than I would have without this.

[crmne]

Paul, this PR is 90% VCR cassettes, but it's still a monster and I feel like the actual code is overengineered. Passing cache_control through 7 layers? No thanks.

However, how about embracing the fact that there may be moments we want to augment each message?

# Just add message-level params
chat.add_message(
  role: :user,
  content: "huge doc",
  params: { cache_control: { type: 'ephemeral' } }
)

That's it. Add params to Message.

We can then even add:

chat.with_message_params(cache_control: { type: 'ephemeral' })

In order to apply it to every message. That's it.

Then adding a provider agnostic implementation of cache control becomes trivial.

[tpaulshippy]

That could work for system and user messages. How about tools?

I still think there will be some complexity in the anthropic provider that may be unavoidable.

[tpaulshippy]

One complication is that these params have to be nested under the content array for anthropic.

So it would actually need to be something like this:

chat.add_message(
  role: :user,
  content: "huge doc",
  params: { content: [{ cache_control: { type: 'ephemeral' } }] }
)

Right?

[tpaulshippy]

I don't think adding the params to all messages with chat.with_message_params will generally work as you suggest.

Couple of reasons -

1. Anthropic limits you to 4 cache breakpoints per request.
2. You generally want your cache breakpoints on the last system or the last user message to get the maximum use of the cache.

[tpaulshippy]

I do like the suggestion of adding params to messages because that could give more fine grained control over exactly where you want the breakpoints. Would also enable passing optional keys on other providers like this one from OpenAI --

EDIT: Not seeing this option on the OpenAI responses API...

[tpaulshippy]

Would like to explore making this a per message option and maybe something like a with_tools_params option to make this more provider agnostic.

Thanks for the feedback @crmne

[tpaulshippy]

The more I think about it, the more I'm thinking it is odd to have a provider specific feature enabled like this. I wish anthropic was just more like the others in this area.

[tpaulshippy]

One complication is that these params have to be nested under the content array for anthropic.

So it would actually need to be something like this:

chat.add_message(
  role: :user,
  content: "huge doc",
  params: { content: [{ cache_control: { type: 'ephemeral' } }] }
)

Looks like to make this work, deep_merge will have to be enhanced to support arrays. Currently it just overrides the whole array rather than merging the hashes within. Will need something like this:

What do you think @crmne ?