AI gateway batch API support in 2026: async jobs, discounts and failure recovery

OpenAI's Batch API accepts a JSONL input file, creates a batch with a 24-hour completion window, exposes status and cancellation operations, and provides result and error files. Its guide says batch processing costs 50% less than synchronous calls and uses a separate rate-limit pool. This is not simply a normal chat request with a cheaper model name. A gateway must support the file and batch lifecycle, or the application must deliberately call the provider's batch endpoints directly.

Anthropic's Message Batches API accepts independently identified Messages requests, processes them asynchronously, and returns results after processing ends or the 24-hour window expires. Anthropic also documents a 50% price reduction. The input structure and result API differ from OpenAI's JSONL file workflow. A vendor claiming 'multi-provider API compatibility' has not thereby demonstrated batch parity across the two providers.

Kong's AI Proxy documentation describes RESTful batch routes in newer gateway versions and native-format Anthropic batch support in its provider table. That is useful evidence of a supported architecture, not proof of every feature on every deployment. Cloudflare's dynamic routing documentation explicitly limits dynamic routes to OpenAI chat-completions-shaped requests, even though Cloudflare's broader AI REST API exposes additional formats. Treat these as different products and request paths, not as a single binary 'batch supported' field.

LiteLLM and other gateways document multiple supported endpoint families, but the only reliable procurement evidence is a working test of the exact batch operation on the selected version and plan. For OpenRouter, Bifrost/Maxim, Portkey, TrueFoundry, Helicone and AetherGate, mark batch support unknown until you see a documented route and a passing end-to-end test. Never assume a unified base URL automatically proxies provider-specific file and batch management.

OpenAI's batch guide warns that output lines may arrive in a different order than input lines and instructs clients to join using custom_id. Anthropic also identifies each request by custom_id and reports succeeded, errored, canceled and expired outcomes. A robust application stores a durable record for each logical item before submission and joins the provider result to that record. Treating output line 100 as input line 100 is a correctness bug.

Design a state machine with queued, submitted, running, succeeded, failed, expired and canceled states. Save the provider batch ID, individual custom_id, submission time, model and prompt-version hash. Keep a separate record for each resubmission attempt so an expired or ambiguous item is not blindly billed or processed twice. This is especially important when a batch produces customer-facing content or writes to downstream business systems.

A simple illustration: if a representative workload would cost $1,000 at the provider's synchronous token rates, a 50% batch discount would make the same billable tokens $500 before gateway charges, additional attempts and operational work. This is arithmetic, not a benchmark or a promise of a $500 invoice reduction. If ten percent of items need an additional full-price retry, or the workflow requires expensive preprocessing and quality checks, the realized difference changes.

Build the cost ledger at the attempt level. Track provider, model, input and output tokens, cache categories, discount type, attempt status, and the provider's actual bill. Separate batch inference from storage, file uploads, gateway subscriptions and any resubmission. When using a BYOK gateway, the upstream provider remains the source of truth for actual inference charges. A dashboard estimate should not be presented as a settled invoice.

Batch workflows create durable input and output artifacts, so a synchronous API's privacy configuration may not apply. Anthropic's current batch-processing documentation explicitly labels Message Batches as not eligible for zero data retention or HIPAA use. Its results are available for a documented retention period. OpenAI's guide describes uploaded files and output artifacts with their own lifecycle. A buyer must review each provider's current data-processing terms for the specific batch endpoint, not infer them from a gateway's marketing.

Use synthetic data for the first tests. Before real production data, establish who can upload and download batch files, whether the gateway logs bodies, where files are stored, how they are deleted and whether a provider's region or contractual terms permit the workflow. If a task requires an unsupported privacy guarantee, use an approved alternative rather than routing around the restriction.

Run a controlled batch with twenty synthetic items: twelve valid, four intentionally invalid, two long-running, and two designed to exercise a retry or cancellation path. Use the same fixture set on direct-provider APIs and through the candidate gateway. Do not assume every provider allows identical error injection; where behavior differs, document the limitation and test the closest supported case. Measure submission success, result reconciliation, job completion, retries, auditability and invoice accuracy.

Pass the gateway only if every accepted result maps to one intended logical item, all partial failures are visible, the intended access policy applies to the batch lifecycle, and the observed provider cost can be reconciled. If the gateway does not support batch natively, an explicit, documented direct-provider bypass may be operationally acceptable, but it changes the security, observability and spending-control boundary. Add that bypass to the enterprise gateway RFP rather than hiding it.