If you have ever put a generative AI application into production, you know the feeling: the solution works beautifully in the POC, and when volume grows, the token bill becomes the topic of the meeting. The good news is that a large part of this cost is avoidable waste — you are paying, repeatedly, for the model to “reread” the same context on every call.

Prompt caching solves exactly that. In this article I explain what it is, how it works under the hood, show the architecture, give real examples on the three platforms I use most (Anthropic, Azure OpenAI, and GitHub Models), and close with the cost math and best practices that separate those who save money from those who only think they are saving.

The problem: LLMs have no memory between calls

Language models are stateless. Each API request is independent: the model does not “remember” the previous call. To continue a conversation or provide context (a document, system instructions, few-shot examples), you need to send everything again on every request.

In practice, a typical production prompt is composed like this:

[ System prompt + instructions      ]  <- large, stable, always repeated
[ Documents / knowledge base        ]  <- large, changes little
[ Conversation history              ]  <- grows with each turn
[ User's current question           ]  <- small, always different

The problem becomes clear with numbers. Imagine an assistant with 30,000 tokens of fixed context (instructions + documentation) and a 200-token question. With every message, you pay for 30,200 input tokens — even though 30,000 are exactly the same as in the previous call. Multiply that by thousands of requests and you understand where the bill comes from.

In one of my real consumption analyses, an AI tool accumulated ~195 million input tokens across 638 calls — about 305,000 tokens per call. That is only possible when the same huge context is resent on every interaction. It is the perfect scenario for caching.

What is prompt caching?

Prompt caching is the provider’s ability to store the internal processing state of a section of the prompt (the prefix) and reuse it in subsequent calls, instead of reprocessing it from scratch.

Technically, what gets cached is not the text — it is the result of the model’s computation over that text: the attention states (KV cache) of the transformer layers for those tokens. When you resend the same prefix, the provider recognizes the hash and “skips” the heavy work of recomputing those activations.

The effect is twofold:

1. Lower cost — tokens read from cache cost a fraction of the normal price (typically ~10%).
2. Lower latency — skipping the prefill of tens of thousands of tokens noticeably speeds up time-to-first-token.

The golden rule: the cache is based on the exact prefix

This is the part most people get wrong. The cache works on the prompt prefix, from left to right, and requires a byte-for-byte match. If the first character changes, the entire cache from that point onward is invalidated.

Call 1:  [A][B][C][pergunta_1]     -> processes everything, writes cache for [A][B][C]
Call 2:  [A][B][C][pergunta_2]     -> cache HIT on [A][B][C], processes only the question OK
Call 3:  [X][A][B][C][pergunta_3]  -> prefix changed at the beginning, total cache MISS

Practical corollary: put stable content at the beginning and variable content at the end.

Architecture

The diagram below shows the path of a request with caching and where the gain happens.

              User / App
                   |
             (stable prefix + variable input)
                   v
              Gateway / SDK
                   |
                   v
            LLM provider API
                   |
           Is the prefix already cached?
             /                            HIT                    MISS
           |                      |
   Reads KV-cache          Processes from scratch
   ~10% of input           writes to cache
   lower latency           ~125% of input
           \                      /
            \                    /
             v                  v
        Response generation (output = normal price)
                   |
                   v
                Response -> User

Anatomy of a cache-optimized prompt:

|============ CACHEABLE PREFIX (does not change) ============|== DYNAMIC SUFFIX ==|
[ System prompt ] [ Documents / RAG ] [ Few-shot ] || [ History ] [ Question ]
                                          ^
                                   cache breakpoint (cache_control)

The central idea: you mark a breakpoint right after the stable block. Everything to the left becomes a cache candidate; everything to the right is always recomputed.

Practical examples

1. Anthropic (Claude) — explicit control

In the Anthropic API, caching is explicit: you mark blocks with cache_control. It is the most transparent model and the one that gives the most control.

import anthropic

client = anthropic.Anthropic()

DOC_GRANDE = open("manual_produto.md").read()  # e.g.: 30k tokens

response = client.messages.create(
    model="claude-opus-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a technical support specialist. Cite the manual section.",
        },
        {
            "type": "text",
            "text": DOC_GRANDE,
            # everything UP TO HERE becomes a cacheable block
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[
        {"role": "user", "content": "How do I roll back a release?"}
    ],
)

print(response.usage)
# cache_creation_input_tokens : tokens written to cache (1st time, ~1.25x)
# cache_read_input_tokens     : tokens read from cache (~0.1x)
# input_tokens                : new tokens processed normally

Anthropic key points:

• You can have up to 4 cache breakpoints.
• Default TTL of 5 minutes (there is an extended 1-hour option with different pricing).
• The first write costs ~1.25× normal input; subsequent reads cost ~0.1×.
• Requires a minimum number of tokens to activate (e.g., ~1024 for Opus/Sonnet).

2. Azure OpenAI — automatic caching

In Azure OpenAI (and OpenAI directly), caching is automatic and transparent for prompts above ~1,024 tokens. You do not mark anything; the service detects repeated prefixes.

from openai import AzureOpenAI

client = AzureOpenAI(
    api_version="2024-10-21",
    azure_endpoint="https://SEU-RECURSO.openai.azure.com",
)

SYSTEM = "Long, stable instructions... (>1024 tokens to activate the cache)"

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": SYSTEM},   # stable prefix first
        {"role": "user", "content": "User's variable question"},
    ],
)

# The gain appears in usage.prompt_tokens_details.cached_tokens
print(resp.usage.prompt_tokens_details.cached_tokens)

Azure OpenAI key points:

• Automatic — activates from 1,024 prompt tokens, in blocks of 128.
• Cached tokens usually receive a significant discount on input.
• You do not control the breakpoint: that is why the discipline of “stable first, variable later” is even more important.
• Metric in usage.prompt_tokens_details.cached_tokens.

3. GitHub Models — prototyping and light production

GitHub Models exposes models (including the OpenAI family and others) through an OpenAI-compatible API, which is great for prototyping. Caching follows the behavior of the underlying provider (automatic for OpenAI models).

from openai import OpenAI

client = OpenAI(
    base_url="https://models.github.ai/inference",
    api_key="SEU_GITHUB_TOKEN",   # PAT with models scope
)

resp = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {"role": "system", "content": "Long and stable context here..."},
        {"role": "user", "content": "User question"},
    ],
)
print(resp.choices[0].message.content)

Pay attention to the GitHub Models billing model: the free plan is based on rate limits (great for experimentation), and paid usage is measured in requests/tokens. For high-volume production, the natural path is to migrate to Azure OpenAI while keeping the same code (compatible API).

The cost math

Let’s look at the numbers with a concrete example. Suppose the Opus model with reference prices of $15 per million input tokens and $75 per million output tokens, and a workload of 638 calls, ~195M input tokens (dominated by repeated context), and ~1.12M output tokens.

Scenario A — without caching

Scenario B — with 90% cache hit

Cache read at ~$1.50/M (10% of input). 90% of input becomes cache read, and 10% is processed normally:

Result: from ~$3,009 to ~$640 — a ~79% reduction in the bill, without changing a single line of business logic. In workloads with very large and stable context, real savings often exceed 80%.

General formula

custo_input = tokens_normais     * preço_input
            + tokens_cache_write * (preço_input * 1.25)
            + tokens_cache_read  * (preço_input * 0.10)

custo_total = custo_input + tokens_saída * preço_saída

The break-even point: because the write costs 1.25× and the read costs 0.10×, the cache “pays for itself” after ~2 reads of the same prefix. In other words, any context reused more than once within a few minutes is already worth it.

Production best practices

1. Stable first, variable last. System prompt, instructions, documents, and few-shots at the beginning; history and question at the end. This is requirement #1 for any caching to work.
2. Do not inject volatile content into the prefix. Timestamps, session IDs, “current time,” user names in the middle of the system prompt — all of that breaks the cache. Move it to the end or remove it.
3. Standardize the prefix across calls. Whitespace, field order, JSON serialization — any byte-for-byte difference causes a miss. Generate the prefix deterministically.
4. Respect the TTL. The cache is ephemeral (~5 min). Sparse requests do not benefit. If traffic is intermittent, consider the extended TTL (where available) or “warm” the cache with a periodic ping.
5. Measure for real. Instrument cache_read_input_tokens / cached_tokens and track the cache hit rate as a FinOps KPI. Without measurement, you are guessing.
6. Segment by version. When you change the system prompt, you invalidate the entire cache. Deploy prompts consciously and group requests by the same version.
7. Combine with other techniques. Caching reduces input cost, but it does not replace well-designed RAG, history truncation, and appropriate model selection (not everything needs Opus).
8. Be careful with sensitive data. The cache is isolated by account/organization, but treat context with PII under the same governance policies as the rest of the pipeline.

Where caching does NOT help

• Short prompts that are always different. Without a significant stable prefix, there is nothing to cache.
• Prefixes below the minimum (e.g., fewer than 1024 tokens): caching does not even activate.
• Very sparse traffic, with intervals longer than the TTL between similar calls.
• Output token reduction: caching acts on input; generation still uses the full price.

Implementation checklist

• Is the stable block (system + docs + few-shot) at the beginning of the prompt?
• Has all volatile content (timestamps, IDs) been moved to the end?
• Is the prefix generated deterministically (same byte for byte)?
• Does the stable prefix have more than ~1,024 tokens?
• Are you measuring the cache hit rate in production?
• Do similar calls happen within the TTL (~5 min)?
• (Anthropic) Are the cache_control markers in the right places?
• (Azure/OpenAI) Do you rely on automatic caching and have you organized the prompt for it?

Conclusion

Prompt caching is one of the few generative AI optimizations that delivers large gains with little effort: you reorganize the prompt, mark (or trust) the breakpoint, and start paying a fraction for repeated context. In the real example in this article, it was the difference between a bill of ~$3,000 and ~$640 for the same work.

The secret is not magic — it is architecture discipline: stable context first, variable content last, and constant measurement. Do that, and caching stops being a technical detail and becomes a real FinOps lever in your AI platform.