Tracking Cache Hits Across Six Backends on One Key: Which Prefixes Are Actually Saving You USDC

Why cache hits matter when you pay per token

When you call multiple models through one aggregator key, you pay the official price × 1.3 for each request. Many backends offer prompt caching: if you reuse the same prefix, the cached portion is billed at a lower rate. But caching behavior varies by provider, and some may not report cache hits clearly. To control costs, you need to see which prefixes are actually being served from cache and which are re-billed at full price.

Understanding cache-hit data from backend responses

Most APIs that support caching return usage details in the response. Look for fields like cachecreationinputtokens and cachereadinputtokens (or similar). The aggregator passes through these per-backend metrics. Your job is to collect and compare them.

• Cache read tokens indicate a hit: those tokens were served from cache and billed at a reduced rate.
• Cache creation tokens indicate a miss: the prefix was stored for future use, billed at a higher rate (often 1.25x or more).
• Regular input tokens are billed at full price when no caching applies.

If a backend does not report cache metrics, you must assume no caching benefit and plan accordingly.

Step-by-step: reading per-backend cache hits

1. Log the full response usage object for every request, along with the model name and timestamp.
2. Segment by backend (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) to compare cache behavior.
3. Calculate effective cost per request using the reported token counts and the known multiplier (×1.3 for users).
4. Identify prefixes that consistently produce cache reads across multiple calls.
5. Flag prefixes that never show cache reads—these are silently re-billed each time.

Which prefixes are likely to be cached?

Caching is most effective for long, stable prefixes that appear in many requests. Good candidates:

• System instructions or role definitions that rarely change.
• Few-shot examples or templates repeated across calls.
• Document context that multiple queries reference.

Avoid dynamic content at the start of your prompt. Place variable parts later so the stable prefix can be cached.

Detecting silent re-billing

A prefix is silently re-billed when you repeat it but no cache read tokens appear. This can happen because:

• The backend does not support caching for that model or endpoint.
• The prefix is too short to meet the minimum cacheable length.
• The prefix changes slightly (e.g., timestamps, user IDs) and invalidates the cache.
• The cache expired between requests due to time-to-live limits.

Regularly audit your logs for these patterns. If a prefix is never cached, consider shortening it, removing it, or accepting the full cost.

Using cache data to reduce USDC spend

Once you know which prefixes are cached, you can:

• Standardize prompts to maximize the reusable prefix length.
• Batch similar requests to increase cache hit rates within the TTL window.
• Switch models for tasks where caching is ineffective on one backend but works on another.
• Monitor cost per backend and allocate workloads accordingly.

Remember that even with caching, you still pay the ×1.3 multiplier on the cached rate. The savings come from the backend's reduced cache-read price, not from the aggregator.

What about contributor credits?

If you contribute keys, you earn credits at official price × 1.1 (or × 1.2 for premium models) in USDC. Cache hits affect the official price calculation, so accurate reporting benefits both users and contributors. Always verify that cache metrics are correctly reflected in your billing.

Tools and practices for ongoing tracking

• Store raw API responses in a structured log (JSON) for later analysis.
• Build a simple dashboard that aggregates cache read/write tokens by model and prefix hash.
• Set alerts for prefixes that stop showing cache hits after a model update.
• Periodically review model documentation for changes in caching behavior.

By treating cache hits as a first-class metric, you can make informed decisions about prompt design and model selection, ensuring you get the most value from your USDC balance.