The five gateway categories buyers are actually comparing
Managed model marketplaces optimize for breadth and time-to-first-request. Self-hosted gateways optimize for control and deployment ownership. Observability-first products optimize for traces and debugging. Enterprise control planes emphasize policy, SSO, auditability and deployment controls. Orchestration products go above routing and coordinate work across multiple models.
Those categories overlap, but they are not interchangeable. A team that mainly wants one key for hundreds of models has a different buying problem from a team that wants provider-direct BYOK, strict budgets, or a durable multi-agent workflow.
- Marketplace: broad model catalog and unified credits.
- Self-hosted proxy: maximum infrastructure control.
- Observability gateway: logging, traces, cost and evaluation.
- Enterprise gateway: governance, RBAC, audit and deployment controls.
- Orchestration control plane: persistent multi-model task execution above the gateway.
What to compare before you look at model count
Model count is easy to advertise and often a weak predictor of production fit. Start with billing and ownership: who bills inference, whether BYOK adds a percentage fee, whether logs are metered, and whether the gateway can be moved or self-hosted later.
Then test reliability controls. A useful failover system should make the routing rule explicit, preserve request context, bound retries and timeouts, and expose what actually happened. A status badge that says “fallback enabled” is not enough when you are debugging an incident.
- Provider-direct billing vs prepaid gateway credits.
- BYOK fee thresholds or percentage markups.
- Fallback policy, retries, timeouts and attempt traces.
- Per-key or per-team budgets and rate limits.
- Retention rules and whether prompts/responses are stored.
- Deployment and data-residency requirements.
When orchestration becomes the deciding factor
Routing one prompt to one model is a gateway problem. Breaking an objective into workstreams, assigning different models, preserving intermediate state, reviewing contradictions and verifying a final answer is an orchestration problem.
If your application already has an agent framework, you may only need the gateway layer. If you want the infrastructure itself to own durable execution across models, evaluate whether the product has persistent mission state, restart recovery, explicit review stages and observable model handoffs.
A simple evaluation process
Shortlist two or three products and run the same workload through all of them. Use the same provider accounts and model IDs where possible. Measure setup time, proxy overhead, failover behavior, logging quality, billing clarity and how easy it is to understand a failed request.
Avoid choosing from feature tables alone. Gateway products sit directly in the request path, so operational behavior matters more than landing-page parity.