AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Use prompt caching with open-weight models on Amazon Bedrock

Collected Nov 16, 2026

Open-weight models on Amazon Bedrock now support explicit prompt caching, according to an AWS Machine Learning Blog post that walks through both caching modes using the Chat Completions API. Prompt caching lets Bedrock reuse the computed state of a prompt prefix that matches an earlier request, so the shared content is read from cache and only the remainder is processed. That can cut input costs and response latency when an application resends the same long context, such as a system prompt, reference documents, tool definitions or the conversation so far.

Bedrock manages the cache itself, and which modes a model offers depends on the model and the API. The post describes two approaches. Implicit caching requires no request changes: Bedrock automatically reuses eligible prefixes on a best-effort basis. Explicit caching asks the developer to mark the end of each reusable prefix with a cache breakpoint. In a multi-turn conversation the previous request naturally becomes the cached prefix of the next one, so static material should sit at the start of the prompt and changing material at the end. If an early part of the prompt changes, the cache is rebuilt from that point.

Under explicit mode, content up to and including each breakpoint is written to cache once and read on later requests, while anything after the last breakpoint is processed every time at the standard input rate. Models support a maximum number of breakpoints, a minimum token count before a breakpoint, and a time to live that resets on each cache hit; those values live on the model card. Cache reads are billed below standard input, and on some models cache writes cost more than standard input, which is the main reason to be deliberate about what gets written. With implicit caching, Bedrock can also write content that later requests never reuse, such as a question that differs each time. Explicit mode confines writes to the shared prefix, so on models with a premium write rate you avoid paying it for text that is never read back.

The worked example is a support assistant for a fictional water filter, the AnyCompany AquaPure 500. The system prompt holds a short instruction plus roughly 4,000 tokens of Markdown owner's manual, and three customer questions follow. The samples use the OpenAI Python SDK pointed at the Bedrock OpenAI-compatible endpoint, authenticated with a short-term Bedrock API key produced by provide_token() from the aws-bedrock-token-generator package, valid for up to 12 hours, so no long-lived key needs storing. Prerequisites include an AWS account with model access, an IAM identity permitted to invoke the model, Python 3.12 with openai 3.26.0 or later and aws-bedrock-token-generator 1.1.0 or later, and npm for the coding agent portion. Model availability and caching support vary by Region.

Cache activity shows up in each response's usage.prompt_tokens_details, specifically cached_tokens and cache_write_tokens, with the uncached remainder derived by subtraction. In the implicit run, the first call writes the prompt to cache; calls two and three read the instruction and manual back from cache and write a few tokens each because implicit mode extends the cached prefix over each new question. A few trailing tokens stay uncached. Re-running within the TTL means every call is a read and nothing is written. The post notes that putting the manual and the question as parts of a single user message produced no implicit hits, so the system/user split matters in practice.

For explicit mode, the request carries prompt_cache_options with mode set to explicit, sent through the SDK's extra_body because it is a Bedrock-specific field, and a prompt_cache_breakpoint object attached to the text content part that ends the reusable prefix. Both pieces must appear together: explicit mode without a breakpoint caches nothing, and a breakpoint without explicit mode falls back to implicit behavior. An optional ttl field takes values documented on the model card. With multiple breakpoints you can cache layered prefixes, for example shared instructions and tool definitions followed by a per-customer document. In the explicit run only the first call writes; repeats read the full prefix and leave only the question uncached. To force a cold start, the post suggests prefixing the system prompt with a random run identifier.

The second example applies implicit caching to OpenCode, an open source coding agent, installed at version 1.18.35 and configured through opencode.json to reach the Bedrock endpoint as an OpenAI-compatible provider via @ai-sdk/openai-compatible. The configuration points both the main model and small_model, used for session titles, at the same Bedrock model, and reads the key from a BEDROCK_API_KEY environment variable. Coding agents are a natural fit because each step resends a large stable prefix of system prompt, tool definitions and prior conversation. In a three-turn session against a small repository, the first request is written to cache, and each later request reads everything up to the end of the previous one, processing only the new tail. This required no caching configuration. One caveat: OpenCode 1.18.35 does not record cache_write_tokens for an OpenAI-compatible provider, so its export shows cache reads but reports zero writes.

The closing guidance generalizes across models: keep static content first and identical between requests, put timestamps and request IDs at the end, and place breakpoints after the static blocks they cover. Prefixes must meet the model's minimum token count to be eligible. Developers should confirm hits by reading the usage fields per response and tracking the CacheReadInputTokenCount and CacheWriteInputTokenCount runtime metrics across a workload. Steady traffic keeps a prefix warm because the TTL resets on each hit, whereas an idle gap longer than the TTL forces a rewrite. Cache only what gets reused, particularly where writes cost more than standard input, and expect some extra writes under cross-Region inference during high demand. The examples create no AWS resources, and cache entries expire once unread for the TTL. AWS also states that prompts and outputs are not shared with model providers or used to train models, matching its standard Bedrock data-protection posture.

My reading is that the practical decision is mostly about prompt shape rather than API surface. If an application already leads with a fixed block and appends variable content, implicit caching should capture most of the benefit with no code change. Explicit breakpoints earn their complexity in two situations: prompts with layers that change at different rates, and workloads where a large, always-different tail would otherwise be written to cache under a premium write rate. The reported detail that splitting manual and question across one user message killed implicit hits is a useful warning that small structural choices can silently disable caching, which is why per-response usage fields are worth logging from day one.

Why it matters: teams running open-weight models on Bedrock through the OpenAI-compatible API can now control exactly which prompt segments are cached, which mainly affects cost and latency for retrieval-augmented assistants, agent loops and coding tools that resend large stable prefixes. Developers building on Bedrock should expect to restructure prompts so static text leads, and to instrument cache reads and writes rather than assume caching is happening. My inference is that applications with long, frequently reused system prompts stand to gain the most, while those whose prompts shift near the start will see little benefit until they are reorganized.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Many generative AI applications resend the same long context on every request. Prompt caching lets open-weight models on Amazon Bedrock reuse it to reduce input cost and latency. This post shows implicit and explicit caching through the Chat Completions API and applies caching to an OpenCode coding agent session.