AI gateway failover during streaming: the point where a backup cannot be invisible

First, a gateway may fail before it reaches the model provider: authentication, policy admission or network resolution. Second, the provider may reject the request before generating output, such as a rate-limit response or unavailable model. Third, a provider may begin streaming a response and then disconnect or stall. These are different states, even if a dashboard collapses them into one 'failed request' count.

Before the application has received any output, a gateway can often retry or select a compatible fallback within the caller's deadline. After output has been emitted, switching models can produce a new answer that contradicts or repeats the earlier text. In a tool-using workflow, the partial output might already have caused the client to prepare an action. A safe design makes this boundary explicit and exposes an error or resumable application state rather than pretending a second model continued the same generation.

Portkey's fallback documentation describes a prioritized target list and configurable trigger status codes. It warns that alternate models may differ in capabilities, latency and cost, and that multiple invocations can occur. LiteLLM's architecture separates retries within a model group from fallbacks to another group; its key and team router settings can also vary routing policy. Neither statement alone should be interpreted as a universal guarantee that a partially delivered response can be stitched into a coherent stream.

Cloudflare's October 2026 dynamic-routing documentation describes versioned routes with model nodes, timeout/retry properties and fallback edges. Its JSON configuration says a model node's success output is reached when streaming successfully starts. That is an especially useful distinction to test: what happens if the stream starts successfully but later fails? Confirm actual behavior for your request format and gateway version. Cloudflare also documents that its dynamic routes accept the OpenAI chat-completions request shape, not every provider-native API.

Retries are not free merely because the final user sees one answer. Depending on provider and billing rules, a failed or timed-out attempt may still consume tokens or bill for work already performed. OpenRouter's published pricing currently says errored fallback attempts are not charged for model tokens on its marketplace path, with caveats for auxiliary services. That is a vendor-specific policy, not a universal statement about BYOK providers or every gateway.

Set an attempt ceiling, a total deadline and a maximum estimated cost for each logical task. Record every upstream attempt with its provider, model, status, usage class and elapsed time. A dashboard that only stores the successful attempt can understate latency and cost. For a user-facing application, one slow fallback may be worse than a prompt explicit error; measure task completion rate alongside p95 response time and actual invoice impact.

Prepare a representative set of tasks: plain text, a strict JSON schema, a tool-call request and a streaming answer. Run each directly against the primary model and then through the gateway with the same settings. Introduce controlled failures before the provider response and after the first streamed chunk. Verify that clients receive the documented status and that they never silently combine incompatible answers.

Repeat under concurrency with an intentionally tight deadline and budget. Check whether retry requests are bounded, whether fallback models are allowlisted, whether provider identity is visible, and whether the gateway returns a complete accounting of attempts. For structured output, validate the actual schema; a backup that returns fluent prose instead of required JSON is a failed task even if the HTTP request returned 200.

AetherGate's public guides state that it does not silently replay a stream on another model after a live answer has started. This is a narrower and more defensible claim than 'seamless streaming failover.' Its pricing page describes compatible failover as part of the Team offering, but buyers should test the exact route, supported provider endpoints, response framing and error semantics in their own environment.

Ask any vendor, including AetherGate, for the current implementation and a repeatable test run. An architecture diagram, feature checkmark or internally reported latency figure is not enough. The buying decision should turn on what the caller actually receives when an upstream request fails at the least convenient moment.

For chat-only interfaces, a visible interruption with a retry button may be acceptable. For financial analysis, structured extraction or tool-driven automation, partial responses and duplicate actions can create much larger problems. Define the failure contract with the application team before choosing a gateway, including whether the client can restart a logical task and what state must be persisted.

The strongest production gateway is not necessarily the one that attempts the most retries. It is the one that applies the team's authorized policy, preserves output correctness, keeps spending bounded and makes failure diagnosable. Put those criteria in the acceptance test and compare vendors on observed results.