Start with the same task, not a model-count headline
A realistic gateway evaluation uses the exact model endpoints, response formats, traffic mix and failure cases your application needs. Portkey's official gateway page lists conditional routing, fallbacks, retries, load balancing, multimodal routes and trace visibility. Helicone's gateway documentation describes a unified API for multiple providers with automatic logging and failovers. Those claims make both credible candidates; they do not establish that every model, parameter or modality is interchangeable.
Create a fixture set of at least twenty sanitized requests: normal chat, structured output, a tool call, a streamed response, a rate-limit error, a timeout and a long prompt. Keep provider, model version, temperature, prompt and expected output criteria constant across both products. Report differences as observed on the tested plan and date. If a route fails because the provider or model lacks a capability, do not attribute that limitation to the gateway without reproducing the direct-provider call.
- Separate gateway compatibility from upstream model capability and account entitlements.
- Record the actual model and provider that answered, not only the alias the application requested.
- Compare end-to-end quality and retries before comparing raw response latency.
Routing policies: Portkey configurations and Helicone gateway rules
Portkey's documented fallback configuration takes a prioritized target list, and its documentation allows error-code-specific triggers such as a 429. Its load-balancing documentation shows weighted distribution across virtual-key targets. Portkey also describes a trace that links failed and successful attempts. This matters when a failed primary call can still incur provider usage and the buyer needs to know which route actually served the request.
Helicone's AI Gateway is described by its own launch materials as a routing, caching and observability product, with multi-provider support and automatic failover. Its current quickstart uses the ai-gateway.helicone.ai endpoint, while older provider-specific proxies remain documented as legacy integrations. Do not compare Portkey's newest managed configuration with a Helicone legacy proxy and call the result a fair test. Use the current recommended integration for each.
- Inject a 429 and a timeout separately; inspect which errors trigger fallback and how many attempts are billed.
- Check whether the same logical request can be traced through every attempted provider.
- Disable automatic fallback for operations where different models would change output contracts or approved data destinations.
Observability: can you explain a customer-visible answer and its cost?
Portkey advertises logs, traces, feedback, custom metadata and alerting on paid plans. Its fallback documentation shows filtering by configuration and trace IDs to inspect an individual chain of attempts. Helicone's core positioning is detailed AI request analytics, debugging and monitoring integrated with its gateway. Both should be evaluated on whether one customer action can be reconstructed without exposing private prompts to unauthorized operators.
Ask each vendor to demonstrate three joins: application request ID to every upstream attempt; upstream attempt to model and token-cost category; and customer or project ID to an authenticated dashboard view. Verify log retention and export, including what happens when storage or log quotas are exceeded. A pretty trace is insufficient if it silently omits failed attempts or uses stale prices for cached tokens.
- Compare request-level timestamps, time to first token, error codes, cache accounting and per-attempt costs.
- Test access boundaries with two tenants; one must not see the other's request details.
- Use synthetic payloads in the test; inspect raw infrastructure logs separately from product dashboards.
Published plan prices: compare the unit of metering
Portkey's current pricing page lists a free prototype-oriented plan, a Production plan at $49 per month and custom Enterprise pricing. It describes 100,000 recorded logs per month on Production, overages for additional request volumes, and plan-specific retention and security features. These are published terms, not a quote for every deployment. Portkey also offers a self-hosted open-source option with a different operations burden.
Helicone's pricing page lists a free Hobby plan with 10,000 free requests and 1 GB of storage, Pro at $79 per month plus usage, Team at $799 per month plus usage, and custom Enterprise. Storage and request charges must be modeled alongside the subscription. Both products sit in front of provider inference, so the model provider's charges or any separate credits must also be counted. Do not compare $49 with $79 and declare a winner without the same usage, retention and features.
- Worksheet: provider inference + gateway subscription + metered requests/logs + storage/export + operations.
- Model a low-volume pilot and a high-volume month with retries and debug logging enabled.
- Confirm current plan limits, regional requirements and procurement terms in writing before purchase.
Key custody, data controls and deployment are separate decisions
Portkey's pricing page distinguishes Production from Enterprise for custom security controls and data residency requirements. Helicone offers hosted gateway and observability services and documents an open-source gateway implementation. Self-hosting a gateway can move operational responsibility but does not automatically make downstream provider processing private or compliant. Request the actual data-flow diagram and list of subprocessors for your selected deployment.
Test provider-key rotation, gateway-key revocation, least-privilege project access, retention deletion and incident recovery. Ask whether logs contain prompts or responses by default and whether that behavior can be disabled at every layer. Treat published compliance claims as vendor statements to verify with current contractual evidence. AetherGate should be held to the same scrutiny; no comparison here establishes that its own production security is certified.
- Identify which runtime can decrypt or forward provider credentials and who can administer it.
- Review support for SSO, audit export, region restrictions and data deletion on the exact paid plan.
- Check whether the application can bypass the gateway during an incident without changing model behavior.
A practical choice rule and 30-minute demonstration
Choose Portkey when its documented policy configuration, governance and team-oriented operations solve a problem you can reproduce in a pilot. Choose Helicone when its gateway-plus-observability workflow gives your team the trace quality and developer ergonomics it needs at an acceptable total cost. Those are conditional fits, not independent endorsements or measured performance rankings.
Have both vendors demonstrate the same sequence: create a restricted application key; route ten successful calls; force a primary-provider 429; show the fallback attempt chain; export a sanitized trace; revoke the key; and reconcile reported costs with the upstream provider's usage. Score correctness, auditability, operational work and total cost separately. Use AetherGate's enterprise RFP checklist as a neutral test worksheet, and compare any AetherGate proposal against the same evidence.
- Pass only when the required behavior is shown on the proposed plan and deployment.
- Record unknowns instead of assuming missing documentation means a feature is absent.
- Keep the raw test results and pricing snapshots so a later procurement review can reproduce the decision.