prompt-cache-per-backend-comparison · EN · 2026-11-03

Cache Behavior Differs by Backend: Testing Whether Claude, GPT and DeepSeek All Honor Your Cached Prefix

This guide explains how to empirically test whether different LLM backends (such as Claude, GPT, and DeepSeek) honor cached prefixes by running the same cacheable workload across multiple models with a single API key. It covers the conceptual differences in cache implementations, a step-by-step testing method using token usage and latency metrics, and how to interpret results to optimize costs and performance.

Why Cache Behavior Varies by Backend

Prompt caching is not a standardized feature across LLM providers. Each backend implements caching differently:

  • Scope: Some cache only the system prompt, others cache any prefix of the conversation.
  • TTL and eviction: Cache lifetimes and eviction policies differ.
  • Granularity: Caching may operate at token, block, or message level.
  • Pricing: Cached tokens may be billed differently (e.g., discounted input price).

Even when using a single API key through an aggregator, the underlying backends remain distinct. Assuming uniform cache behavior can lead to unexpected costs or latency.

Designing a Cacheable Prefix Workload

To compare backends fairly, construct a workload that stresses prefix caching:

  • Long, static prefix: Use a substantial system prompt or document (e.g., a few hundred tokens) that remains identical across requests.
  • Variable suffix: Vary only the final user message to prevent full-request caching.
  • Multiple repetitions: Send the same prefix with different suffixes several times in sequence.
  • Controlled environment: Run tests within a short time window to avoid cache expiry.

Example request structure:

{
  "model": "claude-3-5-sonnet",
  "messages": [
    {"role": "system", "content": "<long static prefix>"},
    {"role": "user", "content": "<variable query>"}
  ]
}

Repeat for each backend (e.g., GPT, DeepSeek) with the same prefix and a new suffix each time.

Collecting Cache Metrics

Most providers expose cache-related metrics in the API response. Look for fields such as:

  • cachecreationinput_tokens
  • cachereadinput_tokens
  • prompttokensdetails.cached_tokens

The exact field names vary, but they indicate whether the prefix was written to cache or read from it. If the aggregator normalizes responses, inspect the raw usage object or documentation for cache indicators.

Also record latency: cached requests often return faster. A significant drop in response time for repeated prefixes suggests cache hits.

Running the Comparison

  1. Warm the cache: Send an initial request with the static prefix to each backend.
  2. Measure cache hits: Send a few follow-up requests with the same prefix but different suffixes. Record cache read tokens and latency.
  3. Repeat for each backend: Use the same prefix content and similar suffix complexity.
  4. Tabulate results: For each backend, note whether cache reads appear, the magnitude of cached tokens, and latency differences.

Example table:

| Backend | First request cache tokens | Second request cache read tokens | Latency drop |
|---------|----------------------------|----------------------------------|--------------|
| Claude | 0 (creation) | 500 | 40% |
| GPT | 0 | 0 | 5% |
| DeepSeek| 0 | 200 | 20% |

Interpreting Results

  • Cache honored: Cache read tokens > 0 and latency improves.
  • Cache not honored: No cache read tokens despite repeated prefix; latency unchanged.
  • Partial support: Some backends may cache only certain message roles or require explicit cache control parameters.

If a backend does not honor cached prefixes, you may need to adjust your prompt structure or accept higher costs for that model.

Practical Implications

  • Cost: Cached input tokens are often billed at a lower rate. Knowing which backends cache helps predict expenses.
  • Performance: Caching reduces latency, which matters for interactive applications.
  • Portability: Code that relies on caching behavior may need per-backend adjustments.

When using an aggregator, remember that you pay the official price × 1.3 for usage, and contributors are credited official price × 1.1 (or × 1.2 for premium models) in USDC. Cache behavior affects the official price component, so testing helps you choose the most cost-effective backend for your workload.

Best Practices

  • Test with realistic prefix lengths; very short prefixes may not trigger caching.
  • Re-test periodically; providers update caching policies.
  • Document per-backend cache characteristics in your codebase.
  • Use a single API key to simplify switching between backends during tests.

By empirically validating cache behavior, you avoid assumptions and optimize both cost and performance across multiple LLM backends.