AI gateway compatibility tests for tool calling and structured outputs

A gateway can accept a familiar chat-completions request while translating it into another provider's native protocol. That translation is useful for switching models, but the upstream model may use different tool-choice options, message roles, image formats, parallel tool-call semantics or structured-output capabilities. An SDK connecting successfully proves transport compatibility; it does not prove the business workflow survives a provider change.

Inventory every request your application actually sends. Include Chat Completions versus Responses-style endpoints, system and developer instructions, tool schemas, JSON constraints, streaming, images and file attachments. Then record each required response field: tool-call ID, function name, arguments, finish reason, refusal indicator, usage details and provider identity. An omitted field can break a downstream state machine even when the natural-language answer looks acceptable.

OpenAI's documentation distinguishes JSON mode, which targets syntactically valid JSON, from Structured Outputs, which enforces a supported JSON Schema subset when configured correctly. A successful JSON parse is therefore not sufficient if your application requires a fixed enum, all mandatory properties or no unexpected fields. Anthropic separately documents structured JSON outputs through output_config.format and strict tool inputs. Their API request shapes are not interchangeable.

Test a schema with nested objects, required fields, enum values, an array and additionalProperties restrictions. Include an intentionally unsupported schema constraint and record whether the provider or gateway rejects it clearly. Validate every returned object on your server even when the provider offers constrained generation; application authorization and business rules are not delegated to a schema. Never assume a model that supports ordinary JSON also supports your exact strict-mode configuration.

Tool calling has at least three stages: the model emits a proposed tool call, your application validates and executes it, and the application returns a result for the next model turn. The gateway generally mediates the model request; it does not automatically establish that a tool invocation was authorized. A valid tool-call JSON object can still request a prohibited operation or contain an identifier belonging to another tenant.

Use a synthetic tool that looks up a fictional order and another that simulates an irreversible action without actually changing anything. Test tool names, argument types, required fields, multiple calls in one turn, missing tool results and tool-result correlation. Check that the same logical tool call is not executed twice after a timeout or fallback. Make tool execution idempotent where feasible and require explicit application-side permission checks.

In a streaming response, tool-call arguments may arrive as partial fragments rather than one complete JSON object. An application must accumulate and validate them before execution. Different gateways may normalize chunk boundaries, usage events or finish markers differently. Test whether a gateway emits a partial function name, changes a tool-call identifier, drops final usage information or closes the stream without an explicit terminal state.

Introduce a network disconnect before the first token, halfway through tool arguments and after the model has completed a tool call. A fallback that begins after partial output can create duplicate user-visible text or conflicting tool requests. The safe response may be to surface an interrupted state rather than silently switching models. Verify cancellation propagation and count any upstream attempts that still incur provider charges.

Create a fixture set with ten or more representative cases: simple text, strict JSON, invalid schema, one tool, parallel tools, a tool refusal, nested arguments, multi-turn tool results, streaming tool fragments and a deliberate provider error. Run the same fixtures directly against the provider and through each candidate gateway using pinned model versions. Compare not only HTTP status but schema adherence, tool-call identity, execution safety and task-specific answer quality.

Use deterministic mock tools so the test never contacts customer systems. Record the request and response protocol shape, latency, token usage and gateway route version. Where outputs are nondeterministic, repeat the task and use a rubric instead of demanding identical wording. Separate hard protocol failures from differences in model reasoning. A model that cannot satisfy a required strict schema should not be a fallback target for that workflow.

LiteLLM describes an OpenAI-compatible proxy that translates requests across providers, while Cloudflare's dynamic routing documentation explicitly limits that routing mode to OpenAI chat-completions request shapes. That restriction does not mean every Cloudflare AI Gateway endpoint has the same limitation; evaluate the particular product feature and route being used. Portkey's fallback documentation also warns that target models can differ in capabilities, so a configured fallback is not proof of functional parity.

Ask vendors, including AetherGate, to demonstrate your exact tool schemas, structured outputs and streaming traces on the models you plan to deploy. Request an unsupported-feature matrix and an upgrade regression policy. If a provider-native capability is essential and cannot be represented faithfully, keep a direct provider integration for that workflow rather than weakening the application's contract to fit a generic gateway.