Start the RFP with the actual architecture
Document the application's request path from client to gateway to model provider, including proxies, observability services, storage, caches and support access. Identify every place that can see prompts, responses or provider credentials. Ask whether the gateway is managed SaaS, customer-hosted, VPC, hybrid or an agent inside your network.
A vendor's home country is not a substitute for a data-flow diagram. Ask for regions of processing, log retention, subprocessors and upstream model locations. If you need a US-only processing commitment, obtain it in the contract and verify the selected model endpoints.
- Provide a current architecture diagram and data-flow map.
- List subprocessors, regions, default retention and deletion SLAs.
- Specify encryption boundaries and administrative access.
- Identify deployment modes that are generally available versus custom or planned.
Credential custody: ask what one compromised system can spend
An encrypted API key stored in the same runtime that can decrypt it is not isolated from compromise of that runtime. Ask which process holds upstream bearer credentials, which identities can authorize requests, and whether the app can bypass approval. A meaningful threat model names which independent component must be compromised before the attacker can spend.
Require a live test: revoke a provider credential, revoke an app key, rotate signing material and attempt to reuse a captured authorization. Inspect whether the system fails closed. A separate broker architecture is only useful when deployed with independent authority, isolation and controls; a design document alone does not establish that protection.
- Show who can read, decrypt or use provider keys at runtime.
- Demonstrate tenant-wide revocation and expiration under concurrent traffic.
- Demonstrate request binding, replay rejection and bounded authorization where claimed.
- Explain provider-enforced limits, residual exposure and how to contain a compromised gateway.
Budget controls: force a race condition
Ask each vendor to enforce a small spending ceiling while dozens of parallel requests are admitted. Determine whether the budget is checked before dispatch, after usage reports arrive, or only through asynchronous alerts. A displayed budget number does not prove a hard stop under concurrency.
Check how token costs are estimated, reserved, settled and reconciled with provider invoices. Model price catalogs can lag; caching, tool calls and retries complicate accounting. A strong answer describes the residual overshoot bound and how disputes are resolved.
- Pass: requests beyond a declared hard ceiling are denied under the stated concurrency and error model.
- Pass: cost telemetry identifies requested and served models, retries and cache-token categories.
- Pass: alerts distinguish gateway estimates from provider-billed actuals.
- Fail: a dashboard warning is represented as an enforced spending cap.
Reliability: break the primary provider on purpose
Inject rate limits, 5xx responses, malformed model outputs and network timeouts. The gateway should make retry and failover rules observable and bounded. Do not assume that a fallback model has the same tool-call schema, context length, privacy terms or quality. Test your exact structured-output contract.
Measure end-to-end p50, p95 and p99 latency, time to first token, success rate and cost under both normal and degraded conditions. Run identical tests direct to provider and through the gateway. Vendor-published proxy microseconds are not a substitute for these application-level measurements.
- Trace every upstream attempt and the served model.
- Prevent infinite retries and runaway bills during an outage.
- Verify stream semantics and avoid replaying a stream after output has started.
- Document incident ownership, support response and rollback procedures.
Governance: prove separation between teams
Create two application teams and two environments. Verify that keys, routing policies, logs, budgets and model allowlists cannot cross boundaries without authorization. If SSO, SCIM, audit logs or role-based access are required, verify the exact product edition and the required identity provider. Do not infer enterprise identity features from a generic 'governance' badge.
Portkey publicly lists RBAC and service account keys on its Production tier, with custom enterprise options. TrueFoundry describes SSO, RBAC and deployment into customer infrastructure. LiteLLM documents virtual keys, budgets and role-related access controls. These are starting points for RFP questions, not independent certification of a specific deployment.
Observability, retention and evidence
Ask for request-level traces, cost attribution by team and application, structured error classification, export formats, redaction behavior and retention controls. If the vendor offers audit evidence, ask who can rewrite or delete it and whether customers can independently verify continuity.
Run an export-and-delete rehearsal with non-sensitive test traffic. Confirm what remains in backups and third-party observability systems. Data retention and log redaction must be tested against your actual contracts and upstream model policies.
Commercial evaluation and exit rehearsal
Compare total costs, not only a subscription quote: inference, gateway fees, storage, support, migration and the engineering work required to run a self-hosted alternative. Define a pilot with real representative traffic and a rollback plan. Include a contract clause requiring prompt access to configuration and logs on termination.
Before procurement closes, move one working workflow away from the vendor and measure the disruption. Export routing rules, keys where permissible, telemetry and policy definitions. If leaving requires rewriting large portions of your application, account for that lock-in explicitly.
- Request a documented trial plan, success criteria and named owner.
- Ask for service commitments and support terms in writing.
- Perform a real configuration export and alternate-gateway test.
- Score security, cost, latency, quality and operational burden separately.
Where AetherGate belongs in an enterprise evaluation
AetherGate publicly offers BYOK provider billing, app-key limits, routing and Foreman orchestration. Its pricing page lists Team $249.99/month, Enterprise Growth $999/month and Enterprise Scale $2,499/month. Custom VPC deployment, service-level commitments and compliance reviews are scoped separately; they are not automatically included.
VARDEN-7 describes stronger independent credential custody and authorization mechanisms, but its guarantees depend on an actual isolated broker deployment and verification. Do not treat proposed architecture as a completed certification. For a security-sensitive buyer, the correct next step is a deployment-specific technical review, not a promise based on marketing copy.