How to Use Prompt Caching to Cut Costs

When you send a prompt, the model processes every input token before it generates a single output token. If your prompt begins with the same 4,000-token system instruction and reference document on every call, you pay to reprocess those same tokens every single time. Prompt caching breaks that cycle: the provider stores the processed prefix and, on later calls that start with the identical prefix, reuses it instead of recomputing it.

The savings come from two places. First, cached input tokens bill at a reduced rate compared to standard input tokens — check the live cached-read rate on the Anthropic pricing page or the OpenAI pricing page, since the exact discount varies by provider. Second, you cut latency, because the model skips the work of re-encoding the prefix. For high-volume apps, the input side of the bill is often the dominant cost, so trimming it is where the real money is.

The catch: caching only helps when the prefix is genuinely stable and reused within the cache's lifetime. A cache entry has a limited time-to-live; if calls are too far apart, the entry expires and you pay full price again. That makes prompt caching ideal for bursty, repeated workloads and less useful for one-off requests.

Anthropic exposes prompt caching explicitly: you add a cache control marker at the boundary of your stable prefix, and the API reports how many tokens were a cache write versus a cache read. The full behavior — minimum cacheable length, time-to-live, and how prefixes are matched — is in the Anthropic prompt caching documentation.

OpenAI applies prompt caching automatically for sufficiently long prompts on supported models, with cached input billed at a reduced rate; the details are on the OpenAI models and pricing pages. Google's Gemini offers context caching as a managed feature for reusing large contexts — see the Gemini pricing page for the current cached-content rate.

The shared principle across all of them: put the stable content first, the variable content last, and keep the prefix byte-for-byte identical across calls. Even a one-character change near the top of the prompt invalidates the cache for everything after it.

Cache efficiency is an ordering problem. Arrange your prompt so the most stable content is at the very top and the least stable at the bottom, in this order: (1) system instructions and persona, (2) tool and function definitions, (3) large reference documents or knowledge-base chunks that stay constant, (4) few-shot examples, then finally (5) the user's variable input.

The reason is that caching matches on the longest identical prefix. If your reusable document sits below the user's question, the question changes every call, the prefix no longer matches, and the document never gets cached. Move the variable text to the end and the whole expensive prefix above it becomes reusable.

Keep the prefix deterministic. Avoid injecting timestamps, request IDs, or randomized ordering into the cached region. If you template the prompt, make sure the template renders identical bytes for the stable section every time. Our structured output schema design patterns guide covers keeping schemas stable, which doubles as good cache hygiene.

Here is a common anti-pattern that defeats caching — the variable user question sits above the large stable reference, so nothing reusable can be cached:

``` USER QUESTION: What's our refund window for EU customers? Knowledge base (8,000 tokens of policy docs that never change)... System: You are a support assistant. Follow the policies above. ```

Now the cacheable version — stable content first, variable content last, with the cache boundary at the end of the reusable prefix:

``` System: You are a support assistant. Follow the policies below exactly. Knowledge base (8,000 tokens of policy docs that never change)... [--- cache the prefix up to here ---] USER QUESTION: What's our refund window for EU customers? ```

The 8,000-token policy block now sits in the stable prefix. The first call writes it to the cache; every subsequent question reuses it at the cached rate, and only the short user question is billed at the full input rate. See the Anthropic prompt caching guide for the exact marker syntax.

Do not assume the cache is hitting — verify it. Both Anthropic and OpenAI return usage fields that distinguish cached input tokens from fresh input tokens. After each call, check those fields: a cache write on the first call followed by cache reads on later calls means it is working. If you only ever see cache writes and never reads, your prefix is changing between calls or the calls are spaced beyond the cache's time-to-live.

Track the ratio of cached reads to total input tokens over a representative window. A high cached-read ratio on a repeated-prefix workload is the signal that your restructuring paid off. If the ratio is low, audit the prefix for hidden variability — a templated date, a reordered list, trailing whitespace differences.

Finally, watch for cache thrash. If you have many distinct stable prefixes (one per customer, say) competing for cache space, you may write more than you read. In that case, consolidate shared instructions into one common prefix and push the per-customer bits into the variable tail.

**Putting variable content too high.** The user's question, a timestamp, or a session ID near the top invalidates everything below it. Push all volatility to the end.

**Prefixes shorter than the minimum.** Providers only cache prefixes above a minimum token threshold. A short system prompt may not be cacheable at all — check the provider docs for the current minimum.

**Calls spaced beyond the time-to-live.** Caches expire. Low-frequency endpoints may never get a hit; batch or warm them if caching matters.

**Non-deterministic templating.** Map iteration order, locale-dependent formatting, or unstable JSON key order can change the rendered bytes. Serialize the stable region deterministically.