Two caches, two very different request paths
A gateway response cache stores a previously generated answer and can return it for a later eligible request. On a hit, the provider may receive no new inference request. A provider prompt cache instead reuses a matching input prefix within the model service; the model still processes the changing portion of the request and generates a fresh completion. These paths have different cost, latency and correctness properties.
Cloudflare's AI Gateway caching documentation describes response caching for identical requests and notes that the feature is disabled by default. OpenAI's prompt caching documentation describes prefix matching, cache breakpoints and model-dependent cached-token accounting. Do not assume a gateway advertising caching implements both mechanisms, or that a response-cache hit means the upstream provider reported cached input tokens.
- Response-cache metric: provider calls avoided and response-cache HIT/MISS.
- Prompt-cache metric: input tokens read from provider cache versus newly processed tokens.
- Track generated output tokens on provider prompt-cache hits; output generation still occurs.
Understand the cache key before enabling reuse
Cloudflare documents a default response-cache key built from provider, endpoint, model, provider authentication and the full request body. That design distinguishes different prompts and credentials. It also supports custom cache keys, which can deliberately group requests more broadly. An incorrectly scoped custom key can return one user's answer to another user's request even when the text, permissions or source data differ.
For any shared or multi-tenant application, the key should account for tenant identity, authorization context, prompt template version, relevant retrieved-document version, model and settings when these affect the answer. Never put raw secrets into an externally visible key. Test whether key overrides, middleware and retries can accidentally bypass isolation. If a vendor cannot explain its key construction and tenant boundary, keep response caching off for sensitive traffic.
- Negative test: same prompt text from two tenants with different private context must not share an answer.
- Negative test: changed tool schema, model or authorization scope must not reuse stale output.
- Record cache-key inputs in a privacy-safe way for debugging.
Freshness and nondeterminism are correctness requirements
A cached answer to a static FAQ may be useful; a cached account balance, appointment availability or live operational status can be wrong seconds later. Time-to-live alone is insufficient when the underlying record changes early. Define invalidation triggers for knowledge-base revisions, permission changes, data deletions and model or prompt updates. If the application cannot identify those triggers, it should not cache the final answer.
Temperature and other generation settings also matter. Replaying an old completion may be acceptable for a fixed taxonomy label but not for an agent expected to make a new decision using current tool results. Treat response caching as a product behavior decision with a correctness contract, not a free infrastructure optimization. For state-changing tools, do not replay a previous result as though a new action occurred.
- Classify endpoints as static, bounded-staleness, personalized or state-changing.
- For each cacheable endpoint, specify TTL, invalidation and expected response equivalence.
- Include time-sensitive and permission-change fixtures in acceptance tests.
Provider prompt caching has its own eligibility and costs
OpenAI documents prefix-cache eligibility, explicit and implicit breakpoints on supported models, different cache-write and cache-read rates and usage fields that identify cached tokens. Anthropic documents cache-write and cache-read pricing for its own prompt-caching system. Both mechanisms depend on the model, endpoint and request shape; neither should be described as a universal percentage reduction in total inference bills.
Stable system instructions, tool definitions and repeated context can improve prefix reuse, but changing content at the start of a prompt can destroy an expected hit. Measure the actual provider usage fields and apply the correct model-specific rates. A long context that was cheap to read from cache may still produce expensive output, and a cache write can cost more than an ordinary uncached input token on some models.
- Separate ordinary input, cached input, cache writes and output tokens.
- Record the provider model ID and price schedule used for each calculation.
- Verify cache accounting under streaming and non-streaming responses.
Run a four-part cache acceptance test
First, send two identical safe, non-personalized requests and verify the gateway's response-cache HIT/MISS behavior and whether the second call reaches the provider. Second, vary one field at a time: prompt text, tool definitions, model, parameters, tenant and retrieved-data version. Third, test expiry and explicit invalidation. Fourth, repeat with response caching disabled to measure provider prompt-cache reuse independently.
Collect end-to-end latency, model success rate, provider request count, provider usage tokens, gateway cache status and effective cost. Use a direct-provider baseline with equivalent input. The test should also confirm that logs do not expose raw prompts or credentials beyond the agreed policy. Report results by cache type rather than merging all saved tokens or calls into one misleading savings number.
- Pass: no cross-tenant response reuse or stale privileged output.
- Pass: cache status and provider request count reconcile for every test case.
- Pass: cost report distinguishes avoided calls from discounted input-token reads.
Choose a cache policy by workload, not marketing percentage
For static classification with identical requests and acceptable staleness, response caching may avoid repeated provider calls. For long, stable instructions followed by changing user requests, provider prompt caching is often the more appropriate mechanism. For personalized, time-sensitive or tool-executing workflows, consider no response caching even when prefix caching is available. Different endpoints in the same application can require different policies.
Before purchasing a gateway, ask for a demonstration of cache-key construction, tenant separation, opt-out controls, invalidation, cache observability and provider usage reconciliation. The most useful comparison is a safe production-like workload, not a vendor's best-case cache-hit claim. Revisit the policy when models, prompt templates, retrieval indexes or privacy requirements change.
- Buying criterion: observable cache type and explicit bypass control.
- Buying criterion: documented isolation and invalidation semantics.
- Buying criterion: cost accounting that does not equate cache hits with guaranteed savings.