Skip to content

feat(js/plugins/vertexai): add support to anthropic prompt caching #2886

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amondnet
Copy link

@amondnet amondnet commented May 8, 2025

Description here... Help the reviewer by:

When using Anthropic models, you can set up cache control for system messages. You can enable this feature by adding a cacheControl property to the custom field of a message. The only currently supported cache type is ephemeral.

For example, to use cache control for a specific system message, you would set it up like this:

const llmResponse = await ai.generate({
  model: claude3Sonnet, // or another Anthropic model
  messages: [
    {
      role: 'system',
      content: [
        {
          text: 'This is an important instruction that can be cached.',
          custom: {
            cacheControl: {
              type: 'ephemeral',
            },
          },
        },
      ],
    },
    {
      role: 'user',
      content: [{ text: 'What should I do when I visit Melbourne?' }],
    },
  ],
});

https://github.com/anthropics/anthropic-sdk-typescript/releases/tag/sdk-v0.33.0
anthropics/anthropic-sdk-typescript#631

Checklist (if applicable):

@github-actions github-actions bot added docs Improvements or additions to documentation js config labels May 8, 2025
@amondnet amondnet force-pushed the vertexai-anthropic-context-caching branch from 72b29ea to 20881ee Compare May 8, 2025 12:59
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
config docs Improvements or additions to documentation js
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

[JS] Add support to anthropic prompt caching
1 participant