New in v1.2.0 - ✨ Prompt Caching with Anthropic

[olimorris]

As per v1.2.0, I've now added in prompt caching for the Anthropic adapter, which they've outlined here. Thanks to everyone in #99 and #111 for raising this as a suggested enhancement.

The adapter automatically caches the large system prompt which is attached to every request and also detects any messages which are over 300 tokens and caches them. If you use the #buffer or #buffers variables in the chat buffer then this will have a huge impact.

From my testing:

• If I open up a chat buffer alongside codecompanion.utils.util.lua
• And type #buffer what does this code do? followed by Can it be refactored to be better?
• This would typically consume 3,000 tokens

With prompt caching, this is now coming out at less than 1,500 tokens. If you set the logging in the plugin to TRACE you'll see output such as:

data: {"type":"message_start","message":{"id":"msg_01Af4gwNihrYaA5YpzYwxTeM","type":"message","role":"assistant","model":"claude-3-5-sonnet-20240620","content":[],"stop_reason":null,"stop_sequence":null,"usage":{"input_tokens":348,"cache_creation_input_tokens":0,"cache_read_input_tokens":1559,"output_tokens":1}}  }

Which will give you an idea of how much caching is taking place.

The caching at over 300 tokens may be too large or too small...only our testing will find out. Equally, I know Anthropic limits cache breakpoints to 4...that may yet cause an issue.

So please try this out and report back.

[niba]

It has some problems when you try to use @rag functionality. Then you see errors like that

"Error: A maximum of 4 blocks with cache_control may be provided. Found 6."

Just open the chat buffer, ask a question with rag marker

[olimorris]

Thanks for sharing. I wasn't sure how they'd handle the limit of 4.

There's a few options here:

• Stop caching system prompts (they're generally not many tokens)
• Record what we've cached and stop once we've hit the limit of 4

I'll mull them over.

Glad you've been playing with the Agents/Tools though. Still a lot more to do on them.

[olimorris]

This is now fixed in 2.3.1.

The Anthropic adapter will initially cache user messages over the opts.cache_over token limit. If it still has some cache breakpoints left, it will then apply them to the system prompts (irrespective of whether they're over the token limit or not). The logic for this is users are more likely to share buffers and files now we've implemented slash commands and they're going to be the code that's subject to the most discussion and prompting.