A common base URL does not imply a common multimodal contract
Text chat often hides integration differences. A text-only smoke test exercises an authentication header, a model string and a completion endpoint; it says almost nothing about how the gateway handles image URLs, base64 payloads, files, speech, transcription, video or embeddings. OpenRouter documents separate dedicated endpoints for images, video, speech, transcription and embeddings, while several input modalities use chat completions. LiteLLM separately enumerates supported completion, response, embedding, image, transcription and speech interfaces. Those are product-specific interfaces, not a universal protocol guarantee.
Build a capability matrix with one row per application operation. Include the exact HTTP endpoint, upstream model identifier, content type, maximum payload, supported transport, expected response schema, streaming requirements and any required preprocessing. A model that accepts images for analysis is not necessarily an image-generation model. Likewise, an embedding endpoint returning vectors is not interchangeable with a chat completion returning text.
- Image understanding: URL versus inline encoded image, multiple images and resolution handling.
- PDF: direct file, remote URL, extraction method, page limits and image-only scans.
- Audio: transcription, speech synthesis, live or chunked streaming and file formats.
- Embeddings and generated media: dedicated endpoints, dimensions, asynchronous jobs and artifact storage.
Trace what happens to a PDF rather than treating it as a file upload
A PDF can follow materially different routes: native document understanding by the upstream model, text extraction before inference, or OCR plus embedded-image processing. OpenRouter describes its PDF support as a routing layer that may select native PDF handling or an extraction engine. Its public PDF documentation describes a paid OCR path and a text-only path. These are not interchangeable for scanned contracts, diagrams, tables or forms, and a gateway may need to send document contents to an additional processor.
Create a representative document set: a digital text PDF, a scanned page, a chart-heavy report and a document with confidential identifiers replaced by synthetic examples. For each gateway, record the actual processors involved, output fidelity, page and token charges, upload retention and whether the result includes citations to the correct page. Compare both extraction errors and the final answer quality. A low-cost text-only extraction that misses an embedded chart can be more expensive operationally than a correct native document workflow.
- Identify every document-processing subprocessor and applicable retention policy.
- Measure whether tables, diagrams and scanned text survive preprocessing.
- Verify that the gateway's bill and upstream bill account for any separate extraction service.
Audio and streaming need their own latency and failure tests
Audio APIs differ in whether they accept encoded audio in a chat message, raw audio bytes at a dedicated endpoint, or a realtime connection. OpenRouter's multimodal guidance notes that audio input uses base64 in its documented unified API and that some media endpoints are separate from chat completions. Do not infer that a gateway supporting streaming text also supports streaming speech, live transcription or interruption handling. Verify the specific product and model version in current documentation.
Use a short recorded utterance, a longer noisy recording and a partial-stream interruption in the test suite. Track upload time, time to first useful transcript or audio chunk, final completion time, recognition accuracy on the test set and whether cancellation actually stops upstream work. For voice applications, a successful HTTP response after several seconds can still fail the user experience. A retry may repeat a transcription or speech-generation charge even if the application only displays one result.
- Measure first useful output and end-to-end latency separately.
- Test client disconnects and cancellation without assuming provider billing stops.
- Check media file retention and whether request logs capture payload content.
Compare modality support by vendor, not by model-count headline
OpenRouter documents a broad unified API for multiple media types, but its own guidance acknowledges endpoint-specific and provider-specific limitations. LiteLLM exposes many endpoint families, with deployment and model mapping controlled by the operator. Cloudflare AI Gateway supports provider-native routing and a compatible interface for qualifying requests; its custom-provider documentation explicitly distinguishes compatible chat calls from provider-specific paths. These are meaningful architectural differences, not proof that every model in each catalog accepts every media format.
Ask Portkey, Bifrost/Maxim, TrueFoundry, Kong, Helicone and AetherGate to demonstrate your exact operations on the specific plan and deployment you are considering. Mark untested support as unknown. Do not turn a vendor's general model-support statement into a verified claim about PDF OCR, live audio, embedding dimensions or generated-video lifecycle. Some vendors specialize in observability or gateway policy rather than offering their own unified media processing.
- Test the specific route and upstream provider, not merely the vendor's brand.
- Require an explicit capability and unsupported-parameter error instead of silent data loss.
- Document the supported model/endpoint matrix and the release date of the tested software.
A compact acceptance test for procurement
Use five fixtures with synthetic data: an image containing small printed text, a mixed text-and-chart PDF, a spoken sentence with background noise, a fixed embedding input and a short generated-media task only if your application needs it. Run each directly against the selected provider and through the candidate gateway. Hold model, region and content constant. Compare output structure, factual correctness, total latency, upstream attempts and costs. Include one deliberately unsupported modality and verify that it fails explicitly rather than being silently converted or dropped.
Repeat with the application's actual authorization and observability configuration. Record the gateway key used, provider route, effective model, cost category and any secondary processor without storing sensitive payloads in logs. A good procurement result is a pass/fail table with known gaps and a documented workaround, not an assertion that one vendor supports 'all modalities.'
- Pass: required payload and output fields survive the gateway unchanged or through an approved documented transform.
- Pass: unsupported media is rejected predictably with actionable errors.
- Pass: costs and processors can be explained per fixture and no unapproved fallback occurs.
How this changes the AI gateway buying decision
If the workload is only text chat, prioritize routing reliability, budgets and key controls. If it handles documents, verify extraction quality and the privacy boundary. If it serves voice, test real-time behavior and cancellation. If it produces embeddings or images, verify endpoint shape, output artifacts and how those operations are metered. These priorities can favor different gateways even for the same company.
Use the existing AetherGate enterprise RFP checklist and gateway comparison as the procurement framework, then append this modality matrix. Do not assume AetherGate or any alternative has verified production support for a modality until the requested endpoint, model and account tier have passed your own test. A vendor that candidly documents a gap can be a safer choice than one that advertises broad compatibility without an executable example.