Why one task can create several cost rows
An end user may click 'Generate' once while the application sends several model requests. A gateway can retry a timeout, route to a backup provider, evaluate a guardrail or run a tool-calling continuation. The billable unit at the upstream provider may therefore be an attempt, not the visible user action. Conversely, a gateway response cache can serve a result without a new upstream inference call. A spend dashboard that only counts successful user-visible tasks misses both cases.
Define three identifiers: a logical task ID created by the application, a gateway request ID and a unique upstream attempt ID. Record the parent-child relationship and an outcome for each attempt. A request that fails locally before provider submission should not be priced as though inference definitely occurred. A timeout after submission may have unknown provider-side billability until the provider's records arrive. Never replace unknown with zero simply because the client did not receive a response.
- Task: customer-visible unit used for product analytics.
- Gateway request: policy decision, authenticated tenant, routing and retry context.
- Upstream attempt: provider, model, status, reported usage, estimated cost and invoice match state.
Use cost categories that match the provider's bill
Input and output token totals alone may be insufficient. Some providers price cached input reads differently from uncached input; other categories can include cache writes, audio, images, tools or regional processing. LiteLLM's spend-tracking documentation explicitly recommends aligning time ranges, token categories including cache, and model price maps when debugging a cost discrepancy. Its per-request cost is a software calculation, not an immutable copy of a provider invoice.
Preserve the pricing version or rate table used when the gateway calculated the request. When a provider changes pricing, do not silently reprice history in place. Store a new adjustment or reconciliation record linked to the original request and the evidence used. If historical logs lack the exact cached-token count, calculate a justified range rather than inventing a precise correction. Finance needs a reproducible bridge between operational telemetry and the actual payable provider charges.
- Capture reported input, output, cached-read and other priced units when supplied.
- Track currency, pricing effective date, model variant and provider billing account.
- Keep estimated cost, provider-billed cost and approved adjustment separate.
Attribute requests to teams without leaking customer data
Cloudflare documents custom AI Gateway metadata that can tag a request with team or user context, but it also documents a limit of five custom entries per request and reserved fields. LiteLLM documents key, team and customer tracking with spend logs. These features help, but an untrusted client-provided 'team' string is not authorization. Derive the billing owner from an authenticated identity or a server-side mapping to the gateway key; use client tags only for additional context.
Keep tags low-cardinality and non-sensitive: tenant ID, environment, cost center, application and experiment are often sufficient. Avoid emails, prompts, customer names or raw API keys in cost labels. Separate access to aggregate finance reports from access to request payloads. Cloudflare's logging guidance explicitly distinguishes metadata-only records from payload retention; this illustrates why a useful cost audit need not preserve every prompt or response.
- Bind cost center to a trusted authenticated principal or key mapping.
- Record unknown attribution in an explicit bucket rather than assigning it to the wrong team.
- Test that one tenant cannot query another tenant's request-level spend.
Reconcile daily with a three-way ledger
Start with a daily aggregate by provider account, model, currency and UTC usage interval. Sum gateway attempts, compare the provider's usage export or invoice where available, then investigate differences. A gap can come from time-zone boundaries, delayed usage reporting, retry attempts, cached tokens, price-tier selection, taxes, provider credits or a missing gateway event. Do not force two numbers to match by changing historical rows without an explanation.
A simple reconciliation row should contain the provider account, date range, gateway estimated total, provider-reported total, known adjustments, unexplained difference, evidence source and reviewer. Establish a materiality threshold and a deadline for unresolved items. Some provider invoices aggregate across multiple applications or use billing windows that do not align to a gateway's UTC day; reconcile at the common level first, then allocate only what the evidence supports.
- Match time windows and model identities before comparing dollars.
- Classify every variance as timing, price, usage category, missing event or unresolved.
- Never auto-unlock customer budgets because a historical estimate was high.
Evaluate vendor observability as evidence, not just charts
Cloudflare AI Gateway documents request, token, cost and error analytics, custom metadata and OpenTelemetry trace export. Its OpenTelemetry guidance describes estimated costs and optional prompt/completion attributes, which means teams must review payload exposure before enabling exporters. LiteLLM documents individual spend-log APIs and paginated export; it warns that older spend-log retrieval can truncate large results. These details matter when a buyer needs a full audit rather than an attractive dashboard.
Ask each candidate gateway, including AetherGate, Portkey, TrueFoundry, Kong, Helicone and Bifrost, to produce an export for a synthetic 24-hour test: every upstream attempt, model, usage category, estimate, tenant mapping and retry outcome. Compare the export with the upstream provider's usage report. Mark unsupported or untested exports as unknown. A vendor's marketing claim about cost visibility is not evidence that its data is invoice-grade.
- Test pagination and completeness at realistic request volumes.
- Verify whether cost data is delayed, sampled, estimated or directly provider-reported.
- Check exporter permissions and whether tracing accidentally includes prompt text.
A purchasing acceptance test for finance and engineering
Create a small synthetic scenario with two tenants, two models, a cacheable repeated prompt, a deliberately failed request and a retry. Record the expected logical tasks and actual upstream attempts independently. Export the gateway's usage and the provider's usage for the same period. Produce a table showing attributable cost, unattributed cost, timing differences and any unresolved gap. Repeat after changing the gateway's model price table in a test environment to verify that old records remain explainable.
The acceptance criterion is not a magical zero discrepancy at every moment. It is a complete, privacy-aware audit trail and a documented process for resolving differences without inventing usage or relaxing spend controls. Read the gateway pricing guide alongside this article: subscription fees, inference charges and operational costs are different cost pools, and only some belong in per-request showback.