LLM Caching Strategies 2026: Prompt Cache vs. KV Cache vs. Semantic Cache

**Mechanic:** Mark a prompt prefix (system prompt + few-shot examples + long context) as cacheable. The provider stores the computed state on their infrastructure. Subsequent requests with the same prefix hit the cache and skip the redundant compute.

**Anthropic's implementation:** Per Anthropic's prompt caching docs at docs.anthropic.com, cache writes cost 25% more than base input tokens; cache reads cost 10% of base input tokens. Cache TTL: 5 minutes (refreshes on each hit) or 1 hour (separate price tier). Break-even: 2-3 cache hits.

**OpenAI's implementation:** Per OpenAI's prompt caching at platform.openai.com, automatic caching for prompts ≥1024 tokens. Cached tokens cost ~50% of normal input. No explicit cache mark needed; the API caches identical prefixes automatically.

**Google's Gemini context caching:** Per Google's caching docs at ai.google.dev, explicit cache creation via API with minimum 4,096 tokens. Cached tokens at ~25% of normal input. Configurable TTL.

**Best for:** Workloads with long static system prompts + few-shot examples + repetitive large contexts (RAG retrievals that appear in many queries). Typical savings: 60-80% on input cost for cache-eligible workloads.

**Mechanic:** During LLM inference, the engine computes 'KV cache' — the keys + values for each attention layer. Per vLLM's KV cache architecture documentation at docs.vllm.ai, efficient inference engines reuse this across requests when prompt prefixes overlap (prefix caching) and across tokens within a single request (the default behavior).

**vLLM specifics:** Per vLLM's automatic prefix caching docs at docs.vllm.ai, prefix caching can be enabled with a flag. Common system prompts across requests get their KV cache reused. Significant latency reduction (often 2-5× faster first-token time) when prefix-cache hits.

**SGLang specifics:** Per SGLang's RadixAttention documentation at sgl-project.github.io, automatic radix-tree-based prefix sharing across requests. Designed for workloads with structured prompt patterns (chatbots with same system prompt, RAG with same context).

**When this layer matters:** If you're running self-hosted inference (open-source models on your own infrastructure), KV cache strategy is a major lever. Per the vLLM blog at blog.vllm.ai, well-tuned KV caching can 3-5× the throughput of a given GPU pool.

**When it doesn't:** Closed-API users (OpenAI, Anthropic) don't control this layer directly — but the provider's prompt caching is conceptually doing the same thing on their side.

**Mechanic:** Embed incoming user queries. Compare against embeddings of cached query-response pairs. If similarity > threshold, return the cached response directly without calling the LLM. Per Redis Labs' semantic caching guide at redis.io, semantic caching skips the LLM entirely on cache hits — the strongest cost lever of the three layers.

**The trade-off:** Cache hit = zero LLM cost + millisecond latency vs. quality risk if the cached response doesn't perfectly fit the new query. Threshold tuning is critical: too loose → wrong responses; too tight → low hit rate.

**Implementation patterns:** Redis as a vector store with vector similarity search, Pinecone vector DB at pinecone.io for managed semantic cache, LangChain's semantic cache integration at python.langchain.com, or custom Postgres + pgvector. The vector-store choice matters less than the threshold tuning + cache invalidation strategy.

**Best for:** High-volume customer-facing workloads (FAQ-style chatbots, product Q&A, support routing) where the query distribution is heavy-tailed and many users ask substantively the same question. Worst for: workflows where the LLM must reason over user-specific data — semantic caching doesn't fit.

**Typical savings:** 30-70% of LLM call volume eliminated entirely at cache-hit threshold ~0.95 similarity. Quality impact: usually negligible at properly-tuned thresholds, occasionally noticeable on edge cases.

**The reality:** All three layers can stack. Each one addresses a different inefficiency. Use the matrix:

**Repetitive system prompts + long static context** → Provider-side prompt caching (Layer 1). Highest ROI for closed-API workloads.

**Self-hosted inference with prefix patterns** → Engine-level KV cache (Layer 2). Highest throughput multiplier for self-hosted.

**High-volume customer queries with significant semantic repetition** → Application-side semantic cache (Layer 3). Eliminates LLM calls entirely on cache hits.

**All three:** A production stack with long static system prompt + RAG context + high-volume FAQ-style usage can stack all three layers. Per Redis Labs' production caching patterns at redis.io and Anthropic's prompt caching documentation at docs.anthropic.com, the layers compose: semantic cache catches duplicates; prompt cache handles the long prefix; KV cache (provider-managed for closed APIs) speeds the rest.

No caching layer in production: Every request pays full inference cost. Latency is constant regardless of repetition. At scale, 60-80% of LLM spend is wasted on redundant compute.
Right caching layer for the workload: 30-90% cost reduction depending on workload structure. Latency improvements often 2-10×. Engineering cost: 1-4 weeks per layer. ROI typically positive within 30-60 days at production volume.