Start with the four ceilings that matter
Requests per minute (RPM) counts how many calls can enter during a time window. Tokens per minute (TPM) measures token throughput, often with provider-specific accounting for input, output and reserved output capacity. A concurrency limit controls how many calls may be active simultaneously. A spending budget limits estimated or recorded monetary usage over a longer period. A system can be below three ceilings and still hit the fourth.
OpenAI's rate-limit documentation distinguishes RPM, TPM, daily limits and model-specific shared pools. A large prompt can exhaust TPM before RPM; a slow streaming call can occupy a concurrency slot for much longer than a short completion. Record the quota's owner—organization, project, model, gateway key or tenant—and its time window. Never interpret a gateway dashboard's green budget indicator as proof the upstream provider has available capacity.
- Track requests, estimated input tokens, actual output tokens and in-flight calls separately.
- Record whether the provider uses shared model pools or separate deployment quotas.
- Distinguish admission denial from a request that reached the provider and then failed.
Understand fixed windows, sliding windows and short bursts
Two gateways both advertising 100 RPM can behave differently at the boundary of a minute. A fixed-window counter can admit a burst just before the boundary and another just after it. A sliding window counts activity across the preceding interval and may reject the second burst. Cloudflare documents both methods for its AI Gateway rate limiting, with 429 responses when its configured gateway limit is exceeded.
Provider enforcement may also happen over shorter intervals than the headline minute. OpenAI warns that rapid bursts can trigger throttling even when the aggregate per-minute total appears acceptable. For buyers, this means testing a smooth 60-second workload is insufficient. Include clustered arrivals, long-running streams, and traffic from several tenants sharing one upstream credential. Record the precise request timestamps and which limit reported the denial.
- Test a smooth load, a two-second spike and traffic straddling a window boundary.
- Record Retry-After and rate-limit headers when the provider supplies them.
- Do not treat a 429 as proof of a billing or authorization failure without inspecting its origin.
Multi-replica enforcement is a distributed-systems question
Per-process counters are not a global policy. Suppose two gateway replicas each independently allow 50 calls per minute under a supposed tenant limit of 50. A load balancer can spread requests across both and admit more than the intended allowance. LiteLLM's production guidance recommends Redis for shared rate-limit counters and routing state across replicas; its router documentation also notes that some max-parallel limits are enforced per proxy process.
Ask each vendor whether counters are centrally coordinated, eventually consistent, or process-local. Test a quota with one replica, then repeat while two or more replicas serve requests. Kill a replica during the test and check whether limits reset, block safely, or temporarily over-admit. An enterprise evaluation should include failure behavior of the backing datastore, not only the happy-path limit configuration.
- Document counter scope and synchronization delay.
- Test restart and failover without assuming global quotas persist.
- Check whether the platform fails open or closed if budget or rate-limit state becomes unavailable.
Retrying a 429 can make the incident worse
An application, SDK and gateway can each retry the same request. If each layer makes several attempts, the number of upstream calls can multiply, while users see only one logical task. Set one accountable retry policy with a maximum number of attempts, an overall deadline and bounded exponential backoff with jitter. Respect Retry-After where present. Separate provider throttling from gateway admission denial so the correct layer decides when to retry.
Fallback to a different provider may restore availability, but it is not free capacity. The alternate model may have different token limits, tool behavior, quality and pricing. Portkey's fallback documentation explicitly cautions that multiple model invocations can occur and change cost and latency. Before allowing a fallback, decide whether the user approved the alternate model and whether any already-started stream or external tool action makes replay unsafe.
- Log logical request ID and every upstream attempt ID.
- Cap retries by elapsed time as well as attempt count.
- Never automatically replay a tool action that may already have produced an external side effect.
Run a capacity acceptance test before procurement
Build a synthetic workload with a small request, a long-context request, a streaming request and a tool-calling request. Assign two authenticated tenants with different quotas and one deliberately constrained upstream provider account. Increase concurrency gradually, then introduce a brief spike. For each request, capture acceptance, rejection layer, latency, provider attempts, token usage and the tenant whose quota was charged.
A passing result is not simply that some calls receive 429. The configured tenant limit should behave as documented across replicas, the unaffected tenant should continue within its own quota where upstream capacity permits, and the application should surface a recoverable error. If a retry reaches the provider, it must appear in cost and trace records. Test the same sequence after a restart and during a datastore interruption.
- Publish a pass/fail matrix for RPM, TPM, concurrency, budgets and upstream throttling.
- Verify tenant identity comes from authenticated context, not a client-editable label.
- Compare recorded provider attempts against upstream usage reports.
How rate limits affect the gateway buying decision
LiteLLM documents key, user and team rate limits, but buyers operating the proxy must provision shared state and validate the exact release. Cloudflare documents fixed or sliding gateway limits, while its separate Unified Billing rate ceiling does not apply to BYOK requests. Those are different controls and should not be conflated. Managed services may reduce maintenance but cannot guarantee that the upstream model provider will never throttle.
Ask AetherGate or any competing gateway for a live test of per-key quotas, concurrent requests, provider 429 propagation and retry accounting. Do not assume a published spend-cap feature proves exact hard-cap behavior during simultaneous in-flight calls. The right gateway is the one whose actual admission and failure behavior matches your application, provider contract and risk tolerance—not the one advertising the largest nominal RPM.