How an LLM gateway works
The application sends a request to the gateway using a stable API contract. The gateway authenticates the caller, resolves the requested model or routing policy, applies controls such as budgets or redaction, then forwards the request to the selected upstream provider.
When the provider responds, the gateway records telemetry and returns a normalized response. If the preferred provider is unavailable, the gateway can apply bounded retries or a configured fallback path.
- Application authenticates to the gateway.
- Gateway resolves model, provider or routing policy.
- Security, budget and rate-limit rules are applied.
- Request is sent to the upstream provider.
- Latency, cost, provider and failure metadata are recorded.
- Fallback is attempted only when policy allows it.
LLM gateway versus direct provider APIs
Direct provider APIs are often simplest when one provider handles nearly all traffic. A gateway becomes valuable when a team needs multiple providers, centralized credentials, spend controls, failover, common telemetry or a stable API surface that survives provider changes.
The gateway adds another system to the request path, so it should earn that complexity with reliability, governance or portability that the application would otherwise need to build.
LLM gateway versus model router
A model router is the decision component that selects a model or provider for a request. An LLM gateway is broader: it may include routing, but also authentication, key custody, rate limits, budgets, observability, redaction and fallback.
Some products use the terms interchangeably. When evaluating a product, inspect the actual request path and control surface instead of relying on the label.
LLM gateway versus multi-agent orchestration
A gateway controls individual model calls. Multi-agent orchestration coordinates a larger objective across multiple agents, tasks and model calls over time.
AetherGate combines an OpenAI-compatible gateway with a separate durable mission layer, so teams can use the gateway by itself or place longer-running orchestration above it.