Cached-token accounting for AI gateways: why one input-token total is not enough

A typical model bill has at least input and output usage, but caching introduces more classes. Ordinary input is processed without a cache discount. Cache reads reuse an eligible prefix and may be charged at a reduced rate. Cache writes create or refresh reusable state and may carry a different rate or lifetime. Output tokens are usually priced separately. Some providers expose additional categories, including reasoning tokens and tool charges, so the complete schema must be versioned by provider and model.

OpenAI's current prompt-caching documentation explicitly distinguishes cached reads and cache writes for supported models and explains that its reported input_tokens total includes the component categories. OpenTelemetry's GenAI semantic attributes similarly define cache-read and cache-creation input-token counts as included in the overall input-token count. This is why adding 'input tokens + cached tokens' without checking the provider's semantics can double-count usage.

Consider a purely illustrative response with 12,000 input tokens: 1,000 ordinary, 4,000 cache reads, 3,000 short-lived cache writes and 4,000 longer-lived cache writes. Suppose ordinary input costs $1 per million tokens, cache reads cost $0.10, short-lived writes $1.25 and longer-lived writes $2.50 per million. The input charge is (1,000 × 1 + 4,000 × 0.10 + 3,000 × 1.25 + 4,000 × 2.50) / 1,000,000 = $0.01515. This is an arithmetic illustration of categories, not a quoted price for every model.

If the gateway instead prices all 12,000 input tokens at $1 per million, it records $0.012 and understates this example. In a different workload dominated by cache reads, the same shortcut can overstate cost. Both directions matter: overestimation can reject requests prematurely, while underestimation can allow more provider spending than a customer intended. A credible gateway needs model-specific rates and actual token classes rather than a fixed input multiplier.

One user-visible task may trigger multiple upstream attempts. The first can time out, a second can be rate-limited, and a third can succeed. Store each attempt with a unique ID and link it to the logical task. The provider may return final usage only after a stream completes; a connection failure can leave an ambiguous outcome. Never silently treat missing usage as zero, and do not invent a final cost when the provider has not supplied enough evidence.

At admission time, a gateway can reserve an estimated maximum amount against a budget. After the provider reports usage, the gateway can settle the reservation and record the difference. For ambiguous outcomes, it needs a bounded reconciliation state rather than replaying paid calls indefinitely. Keep original events append-only or otherwise auditable so later corrections do not erase the historical record. A correction should say what was estimated, what new evidence arrived and who approved the change.

LiteLLM documents spend tracking by virtual key, user and team, backed by its database. Its documentation describes the cost calculation function and warns that some budget behaviors depend on database availability and key/team configuration. Cloudflare AI Gateway documents spend limits scoped by model, provider or metadata, and explicitly says its cost tracking is a best-effort estimate based on token counts and model pricing. Neither description means an application-side budget is identical to an upstream provider's final invoice.

OpenRouter documents distinct credit and BYOK billing paths, including a marketplace policy for errored fallback attempts. This makes reconciliation more complex if a team uses both direct provider invoices and marketplace credits. For AetherGate, the public pricing page says customers pay upstream providers separately and the platform does not mark up tokens. Buyers should still verify how each provider's cache classes, retries and prices are represented in the actual request history before trusting any budget or savings report.

Start with a fixed UTC reporting window and immutable identifiers for each logical task and upstream attempt. Aggregate input, output, cache-read and cache-write counts by provider, model, tenant and application. Join the counts to the rate schedule that applied at request time. Compare the calculated amount to the provider's usage export or invoice where available, while accounting for discounts, taxes, rounding, credits and billing delays. Do not infer customer overbilling merely from a dashboard discrepancy.

Investigate exceptions rather than automatically rewriting them. An error may arise from a missing cache class, a changed model price, a retry counted twice or a provider adjustment posted later. Record the discrepancy range when exact historical counts are unavailable. Keep the original event and attach a reversible adjustment with its reason, evidence and approver. If the dashboard controls spending, review the policy impact separately; a financial correction does not automatically justify relaxing an operational cap.

Ask the vendor to show one real request record with a cache hit, one with a cache write, one with a fallback and one whose final usage arrived after streaming. The record should show model identity, raw token categories, applied rates, estimated and settled amounts, and the source of truth for each field. A pretty cost dashboard that cannot explain a single disputed request is not sufficient for enterprise budgeting.

For an initial proof of concept, compare a week's gateway totals against your upstream provider export, using non-sensitive sample workloads. Measure the absolute discrepancy and the share of requests missing complete usage. That evidence is more useful than a generic claim of 60% token reduction, because token savings, cache discounts, quality and final invoice cost are different measurements.