The AI bill goes up, the team enables prompt caching, lowers max_tokens, swaps in a smaller model — and cost per resolved request keeps rising. When I open the architecture, I usually find the same pattern: overnight processing running as synchronous calls, provisioned capacity sized for the peak and idle outside it, retries hidden behind a successful response, and a router escalating simple requests to more expensive models without anyone measuring the quality gain. The problem is not a lack of optimization. It is optimizing the wrong lever.

The missing decision Caching solves reuse. AI FinOps starts when you separate four questions that look like one: when to run, how to buy capacity, how to absorb variation, and which model should answer. Batch, PTU, pay-as-you-go, and Model Router are not direct competitors. They operate on different dimensions — and sustainable cost appears when each workload lands in the right lane.

What is AI FinOps?

AI FinOps is the operating discipline that connects consumption, capacity, quality, and business outcome. It is not merely tracking spend by subscription or comparing price per million tokens. It is being able to answer, for each use case: how much an accepted task costs, what quality was delivered, what latency was required, and how much purchased capacity sat idle.

The contrast with traditional FinOps matters. In a deterministic API, one call tends to represent the same kind of work. In generative AI, two requests to the same endpoint may use different contexts, call tools, trigger retries, land on different models, and finish with incompatible quality levels. That is why cost per token is an infrastructure metric; cost per approved task is a product metric.

A simple equation exposes what the invoice hides:

Cost per approved task = inference + retrieved context + tool calls + retries + idle capacity, divided by the responses that passed the quality threshold.

The problem it solves

Without this view, every local optimization creates a side effect somewhere else:

1. A smaller model without evaluation lowers the token rate but increases fallback and rework.
2. PTU purchased too early stabilizes capacity but turns uncertain adoption into contracted idle time.
3. Ignoring Batch leaves workloads that could wait competing with interactive traffic for quota and price.
4. Model Router without telemetry changes the model mix, but nobody knows whether the quality gain justifies the selection.
5. Treating caching as the full strategy saves repeated prefixes but does not fix a workload running in the wrong execution mode.

How it works — step by step

1. Fix the processing boundary and latency contract. Start with residency: Global Standard for most workloads without a specific boundary, Data Zone when processing must stay in the zone, and a regional deployment when the requirement is a single region. Then separate second-level responses from work that can wait.
2. Measure a pay-as-you-go baseline. Before committing capacity, record input and output tokens, calls per hour, peaks, retries, latency, and approval rate for each use case.
3. Move work that can wait to Batch. Bulk classification, catalog enrichment, embedding generation, and offline summarization should use an asynchronous queue when the operating window allows it.
4. Separate the stable baseline from the burst. Predictable demand may justify PTU; uncertain growth, seasonality, and spikes remain on pay-as-you-go. A mature architecture often uses both.
5. Use routing only where complexity varies. If every request requires the same reasoning level, a fixed model is simpler to govern. If complexity varies, Model Router can select among supported models — provided quality, selected model, and cost remain observable.
6. Apply caching as a cross-cutting layer. Stable prefixes, system instructions, and repeated context should be reused, but caching does not decide deadline, capacity, or model.
7. Close the loop with cost per outcome. A FinOps dashboard must connect consumption and technical telemetry to the evaluation result: approved task, grounded answer, processed document, or resolved interaction.

Technical deployment example

In production, I would keep the modalities in separate deployments and place the selection policy in the application. The YAML below is pseudoconfiguration for the operating policy — it is not an ARM/Bicep schema and should not be sent directly to the API:

deployments:
  batch: llm-batch
  ptu: llm-ptu
  payg: llm-payg
  router: llm-router

policy:
  contracts-nightly:
    execution: batch
    latency_slo: 24h
  service-copilot:
    execution: online
    baseline: ptu
    spillover: payg  # where supported
  public-assistant:
    execution: online
    capacity: payg
  variable-complexity:
    execution: online
    model_selection: router

The application can resolve that policy before calling the SDK. The example is intentionally simple: in production, the decision must also consider residency, model availability, quota, evaluation, and circuit state.

def select_deployment(workload):
    if workload["offline"] and workload["deadline_hours"] >= 24:
        return "llm-batch"
    if workload["stable_baseline"] and workload["ptu_benchmark_passed"]:
        return "llm-ptu"
    if workload["complexity_varies"]:
        return "llm-router"
    return "llm-payg"

Every call should carry, in the same trace, use_case, prompt_version, requested deployment, executed model, cache hit, tokens, retries, TTFT/TTLT, and evaluation outcome. Before reserving PTU, deploy and test with representative traffic; before enabling the router, compare it with a fixed policy; before moving a workload to Batch, verify that queues, retries, and later completion fit the business process.

Four decisions that should not be mixed — and the layer that cuts across them

Decision	Best starting point	What it optimizes	Main risk
Batch	Asynchronous work with a known deadline	Unit economics and separation from online traffic	Queuing a workload that actually requires interaction
PTU	Stable baseline with predictable model and volume	Capacity and operational predictability	Buying before utilization is proven
Pay-as-you-go	Uncertain adoption, bursts, experiments, and variable tail	Elasticity and low initial commitment	Growing without quota and unit-cost governance
Model Router	Requests with heterogeneous complexity	Model mix across quality, latency, and cost	Opaque routing, excessive fallback, and unmeasured quality
Prompt caching (cross-cutting layer)	Large, repeated prefixes	Input tokens and latency	Mistaking reuse for a complete capacity strategy

These choices are independent in architecture logic, but combinations, quotas, and availability vary by model, region, and deployment type.

Batch changes when the work runs

Batch fits when the result can arrive within a window. Microsoft documents Global Batch at 50% less cost than Global Standard, with separate quota and a 24-hour target — without expiring the job if it takes longer. Model and region remain constraints.

PTU and pay-as-you-go change how capacity is purchased

PTU is billed on deployed capacity, not on tokens used. Quota does not guarantee deployable capacity: size and deploy before buying the reservation. Pay-as-you-go absorbs uncertainty and burst. A healthy pattern is baseline on PTU, tail on consumption, with observable spillover where supported.

Model Router changes which model serves the request

Routing fits variable-complexity traffic, but it does not guarantee savings. Record the model disclosed in the response, restrict the allowed subset, and compare it with a fixed policy. The effective context is constrained by the smallest underlying model; without evaluation, the router becomes a black box.

Caching changes how much context must be recomputed

Caching acts on reuse. The current implementation considers cache from 1,024 tokens, and newer models can add cache-write charges. Check current pricing: a high cache-hit rate does not fix a workload running in the wrong modality.

A hybrid architecture that closes the gap

Imagine a financial institution with four workloads:

• An internal service copilot, used during business hours, with predictable volume and governed context: a candidate for a provisioned baseline after benchmarking.
• A public assistant, exposed to campaigns, incidents, and seasonality: pay-as-you-go to absorb burst, with limits by channel and use case.
• Overnight analysis of contracts and communications, with no immediate-response requirement: Batch, isolated from online traffic.
• Questions with very different complexity, from a customer-record lookup to deep policy analysis: routing, but only with evaluation and model-mix telemetry.

Prompt caching cuts across all four when prefixes repeat. Observability does too: each call carries use_case, environment, prompt version, requested model, executed model, cache hit, tokens, retries, and evaluation outcome.

The point is not to find one winning option. The capacity portfolio must reflect the workload portfolio.

The economics that matter

An AI dashboard showing only spend and tokens explains the invoice, but it does not guide a decision. The minimum metric set is:

Metric	What it reveals
Cost per approved task	Whether the savings preserved the expected outcome
Evaluation pass rate	Whether a smaller model, router, or prompt change degraded quality
TTLT, TTFT, and P95 by use case	Where you are paying for unnecessary urgency
Cache hit and reused tokens	How much repeated context stopped being processed
Retry and fallback rate	Hidden cost from failure, throttling, or low quality
Model mix	What the router or application actually executed
PTU Utilization V2 and spillover	Whether the commitment has a real baseline and where excess traffic went
Completion within the Batch window	Whether savings still satisfy the business process

The executive number is cost per approved task. The others explain why it changed.

Production best practices

• Set budgets and limits by use case, not only by Azure resource.
• Version the prompt and routing policy with the code; an economic change is also a product change.
• Use an evaluation set before switching models, enabling a router, or shrinking context.
• Use Azure OpenAI-specific metrics, including Prompt Tokens, Generated Tokens, TTLT, Prompt Token Cache Match Rate, PTU Utilization V2, Service Tier, and IsSpillover; do not rely on the generic legacy latency metric.
• Bound tool loops and retries with explicit limits and visible failure.
• Benchmark PTU with representative traffic, including real context, concurrency, and request distribution.
• Keep burst outside the provisioned baseline and review utilization before expanding commitment.
• Validate price and availability at decision time. Use the live pricing page, regional tables, and Azure Cost Management; model, region, deployment type, and commercial agreement change the economics.

What has to be in place

Foundation — attributable cost and measurable quality. Every call belongs to a use case and an owner. Prompt, model, tokens, latency, retries, and evaluation outcome live in the same trace. There is a pay-as-you-go baseline and a minimum task set that defines “good enough”.

It is in place when you can explain the cost of an approved task without dividing the invoice equally across products.

Production with context — each workload in the right lane. Offline work uses Batch; variable traffic remains elastic; the stable baseline has a PTU benchmark; routing enters only where complexity varies. Caching and context limits remove repetition, and every change passes a quality evaluation.

It is in place when a cost reduction no longer needs to be defended with “quality seems to be fine”.

Scale and efficiency — capacity as a portfolio. PTU, pay-as-you-go, Batch, and routing are managed as one portfolio. Commitments have an owner and entry/exit criteria; bursts do not contaminate the baseline; cost per task and quality drive model and capacity reviews.

It is in place when the team can move a workload across modalities without losing traceability, SLA, or financial accountability.

The order is causal, not chronological. There is no efficient commitment without a baseline, and no efficient routing without evaluation.

Official references

• Deployment types in Microsoft Foundry Models
• Global Batch with Azure OpenAI in Microsoft Foundry Models
• Model Router concepts and behavior
• Provisioned throughput and cost management
• Prompt caching with Azure OpenAI
• Azure OpenAI monitoring metrics
• Azure OpenAI Service pricing

Frequently asked questions (FAQ)

Is Batch always the cheapest option?

Global Batch is documented at 50% less cost than Global Standard with a 24-hour target turnaround, but it only pays off when the process accepts the window and is designed for queues, retries, and later completion. A job does not expire merely because it exceeds the target; availability varies by model and region.

When does PTU start to make sense?

When there is a sustained, predictable baseline measured with the actual model. Do not rely only on a monthly average: observe concurrency, context size, hourly distribution, seasonality, and utilization. PTU is a capacity decision, not an automatic discount.

Does Model Router guarantee savings?

No. It can improve the model mix, but it must be compared with a fixed policy using quality, latency, and cost per task. Without evaluation, a router can select more expensive models or trigger fallback without measurable benefit.

Can I combine PTU and pay-as-you-go?

Yes. A common architecture keeps the predictable baseline on provisioned capacity and leaves burst, experimentation, new models, and the variable tail on consumption. The boundary should be reviewed with utilization data.

Does prompt caching still matter?

Very much. It reduces recomputation of repeated context and can improve input cost and latency. It simply does not replace the other decisions: when to run, how to buy capacity, and which model to use.

Conclusion

The most common AI FinOps mistake is looking for a single savings lever. Batch, PTU, pay-as-you-go, Model Router, and caching solve different problems. When treated as substitutes, the architecture becomes a chain of local optimizations and the invoice still cannot explain the value delivered.

A mature decision starts with the workload contract: what can wait, what is stable, what varies, and where complexity changes. Then it measures cost per approved task — not only per token. That is how FinOps stops reacting to the invoice and becomes product architecture.

👉 If your team has already reduced tokens and switched models but still cannot explain cost per outcome — especially across enterprise workloads with SLAs, bursts, and governance — this is the decision point. Want to compare notes on AI FinOps and Microsoft Foundry? Reach out on LinkedIn.