Credential custody and authorization
Map where upstream credentials are created, encrypted, decrypted and transmitted. Encryption at rest reduces some storage threats but does not, by itself, stop a compromised runtime that is authorized to use the keys.
Test least privilege and revocation. A scoped application credential should not be able to manage provider vaults, view root secrets or raise its own budget. Examine service-to-service trust boundaries and independent controls.
- Identify every service that can read or use an upstream key.
- Rotate and revoke a compromised application credential during a live test.
- Check whether upstream provider budgets and restrictions remain effective if the gateway is compromised.
- Verify separation of administrative and request-execution permissions.
Prompt data, logs and retention
Decide whether your organization can send prompts, attachments and responses through the gateway under its data policy. Review storage, masking, retention, subprocessors and deletion guarantees.
Telemetry should make provider routing, attempts, cost, status and latency observable without leaking credentials. Audit dashboards and raw service logs separately; reducing secrets in the UI is not enough.
Budget controls under failure
Exercise concurrent requests approaching the same spending limit, provider timeouts and multi-provider retry storms. Document the difference between an application-side estimated limit and a hard provider-enforced ceiling.
If a product claims independent signing, witness authorization or guaranteed economic containment, request threat models and reproducible evidence, not a diagram alone. A single compromised component may still create costs if the upstream provider allows it.