Proving a Prompt Cache Hit Actually Happened on the Aggregator Instead of Assuming It

Why Cache Hits Matter

Prompt caching can reduce latency and cost when the same prefix is reused across requests. On an aggregator that offers an OpenAI-compatible endpoint, you may see faster responses and wonder if a cache hit occurred. But speed alone is not proof—network variance, server load, or model routing can also cause quick replies. This guide shows how to instrument requests to verify a cache hit deterministically.

Understand What the API Exposes

OpenAI-compatible endpoints typically return a usage object in the response. For caching, look for fields like:

• prompt_tokens
• completion_tokens
• total_tokens
• prompttokensdetails (may include cached_tokens)

Not all providers or aggregators populate cached_tokens. If present, it is a direct signal. If absent, you must rely on indirect evidence.

Note: The exact fields depend on the upstream provider and the aggregator's implementation. Always inspect a sample response to see what is available.

Design a Controlled Experiment

To prove a cache hit, you need a baseline and a test case with minimal confounding variables.

1. Craft a long, unique prompt prefix (e.g., a few thousand tokens of random but fixed text). This prefix will be reused.
2. Send a first request with this prefix and a simple completion instruction. Record the full response, including usage and latency.
3. Send a second request with the identical prefix but a different completion instruction (e.g., "Summarize the above" vs. "List key points").
4. Compare the two responses for differences in usage fields, especially cached_tokens, and for latency.

If cached_tokens is present and greater than zero on the second request, that is strong evidence. If not, look for other signals.

Indirect Signals of a Cache Hit

When explicit fields are missing, you can infer a cache hit from:

• Latency reduction: The second request is significantly faster, but only if you control for other factors (same model, same region, same time window).
• Token accounting: Some providers bill cached tokens at a lower rate, which may be reflected in the aggregator's cost breakdown. Check if the aggregator exposes per-request cost or a cost field.
• Consistency: Repeat the experiment multiple times to rule out transient speedups.

Instrument Your Requests

Add logging around your API calls to capture:

• Request timestamp
• Request body (hash the prompt to avoid storing sensitive data)
• Response usage object
• Response latency (measured client-side)
• Model name and any request IDs

Use a structured log format so you can analyze patterns. For example, log JSON lines with fields like prompthash, cachedtokens, latency_ms.

Verify with a Negative Control

To be sure a cache hit is real, include a negative control:

• Send a request with a different long prefix (not seen before) and compare its latency and usage to the cached case.
• If the new prefix yields similar speed but no cached_tokens, the speed alone was not indicative.

Caveats

• Aggregator variability: The aggregator may route requests to different backend instances, which could affect caching. Cache hits are more likely if requests are sent close together.
• Model and provider differences: Not all models support prompt caching, and the behavior may vary.
• Cost implications: Even with caching, you pay the aggregator's multiplier (×1.3). Cache hits reduce the underlying token count, which can lower your cost, but the multiplier still applies.

Practical Checklist

• [ ] Inspect a sample response for cached_tokens or similar fields.
• [ ] Run a two-request experiment with a fixed long prefix.
• [ ] Log usage and latency for both requests.
• [ ] Repeat with a new prefix as a control.
• [ ] Check if the aggregator provides any cache-related headers or metadata.

By following these steps, you can move from assuming a cache hit to proving one—or discovering that the speedup was coincidental.