Build a workload that resembles the application
Start with a corpus of at least 100 representative request shapes for a pilot: short chat, long context, structured JSON, tool calls, and streaming where relevant. The number is a practical starting point, not a statistically guaranteed sample size. Tag each request by model, provider, input size, expected output constraints and sensitivity. Avoid real customer secrets in the test data.
Choose a direct-provider baseline and two gateway configurations. The first gateway run should use identical provider and model targets without optimization or fallback; the second can enable optional policies. This separates proxy overhead from changes caused by routing, caching, prompt trimming or cheaper models. A model substitution may save money while changing quality, so it must not be counted as a pure gateway latency improvement.
- Record request IDs and a reproducible version of the corpus.
- Keep model IDs, decoding parameters, regions and processing tiers aligned.
- Include both warm and cold connection scenarios.
- Define pass/fail for structured output and tool-call schemas.
Measure the right latency numbers
Capture client-observed end-to-end duration, time to first token for streaming, and gateway-added overhead. For a paired request, overhead is the gateway path duration minus the comparable direct path duration; it can be negative in noisy samples, so report the distribution rather than clipping values to zero. Report p50, p95 and p99 with sample counts, not only a mean.
Run multiple repetitions in randomized order. Avoid benchmarking one product at peak traffic and another during a quiet interval. Record DNS, TLS, connection reuse, network location, provider response time, gateway CPU and memory, and concurrency. Where the gateway uses an upstream simulator, label those results separately from live-provider measurements.
Treat reliability as a workload, not a checkbox
Inject a provider timeout, a 429 rate limit, a transient 5xx, a malformed response and a stream interrupted after output has begun. For each case, capture logical request success, upstream attempt count, duplicate output risk, elapsed time and whether the gateway respected its retry budget. Do not repeatedly send live failure traffic to production providers without a controlled test environment and provider permission.
A gateway that retries twice can turn one user action into three billable upstream calls. Calculate both the user-visible success rate and provider-attempt success rate. Show the cost of retries and fallbacks; do not advertise 'zero failures' while hiding increased attempt volume. Portkey's own fallback documentation warns that multiple model invocations can affect price and latency.
Reconcile token cost with provider accounting
Separate uncached input, cached input, cache writes when applicable, output tokens, tools and other billable units. OpenAI's prompt-caching documentation shows why multiplying all prompt tokens by one price can misstate costs. Provider invoices may also depend on processing tier or contract terms. A gateway estimate is useful for operational alerts, but it should not be presented as an exact provider bill without reconciliation.
Compare the gateway's per-request estimated cost with provider usage exports or billing summaries at a matching granularity. Where provider exports cannot be matched to individual requests, report the difference as an aggregate estimate and state uncertainty. For caps, test concurrent reservations and delayed provider usage reporting; distinguish application-side limits from hard provider-side budgets.
Quality, privacy and observability gates
Run the same evaluation rubric against outputs from direct and gateway paths. For deterministic tasks, measure exact schema validity, citation correctness or unit-test pass rates. For open-ended tasks, use blinded human evaluation or a documented rubric with reviewer disagreement. A shorter prompt that changes the answer is not a cost win unless it remains acceptable for the product.
OpenTelemetry's GenAI conventions provide shared vocabulary for model, provider and token usage telemetry. They also warn that some content attributes may contain sensitive information. Confirm that tracing can be useful without indiscriminately storing prompts and responses, and verify log export and retention controls in the actual deployment.
A report template that makes claims auditable
For each gateway, publish: software version, hardware, region, direct baseline, model list, request corpus categories, number of successful logical requests, total provider attempts, p50/p95/p99 latency, time to first token, cost estimate and reconciliation variance, and a separate quality score. Show confidence intervals or raw distributions when possible. Mark every vendor claim that has not been reproduced.
Make the go/no-go decision based on requirements, not a single ranking. A team with strict data residency may reject the fastest option; a low-volume prototype may not justify enterprise subscription fees. AetherGate should be evaluated using this same protocol, including current operational and security limitations, rather than receiving a favorable score by default.