Retrying After a Timeout on One Key Without Double-Billing: Idempotency Tactics Across Backends

When an LLM API call times out, the request may have completed on the server. Retrying blindly can lead to double billing (if the first call succeeded) and two divergent answers. This guide explains how to probe, deduplicate, and retry safely across different backends.

Understand the timeout ambiguity

A timeout means your client stopped waiting for a response—not necessarily that the server failed. The request might still be processing, or it might have completed after your client gave up. Without safeguards, a retry could:

• Incur a second charge for the same logical request.
• Produce a different answer (LLM outputs are non-deterministic), leading to inconsistency.

Idempotency keys: your first line of defense

An idempotency key is a unique identifier you attach to a request. If the server receives two requests with the same key, it processes only one and returns the same result for the other. This prevents double execution and double billing.

• Generate a key per logical operation (e.g., a UUID).
• Include it in a header like Idempotency-Key if the backend supports it.
• Reuse the same key when retrying the same operation.

Not all backends support idempotency keys. Check the API documentation for your provider.

Probing: check if the request already completed

If idempotency keys are unavailable, you can probe the server before retrying:

• List recent completions: Some APIs allow you to query recent requests or usage logs. Look for a request matching your parameters and timestamp.
• Use a status endpoint: If the backend offers a way to check the status of a request by ID, use it.
• Client-side tracking: Record the request ID (if returned asynchronously) and poll for its result.

Without such mechanisms, you must rely on deduplication at the application level.

Deduplication strategies

When retrying, ensure you don’t create duplicate logical requests:

• Unique request fingerprints: Hash the request payload (model, messages, parameters) and store it with a timestamp. Before retrying, check if an identical request was made recently and if it succeeded.
• Distributed locks: Use a lock keyed by the request fingerprint to prevent concurrent identical requests.
• Result caching: Cache the response for a short period. On retry, return the cached result if available.

These tactics reduce the risk of double billing and divergent answers.

Backend-specific considerations

Different LLM providers handle idempotency and timeouts differently:

• OpenAI-style APIs: Some support idempotency keys via headers. Others may not; rely on client-side deduplication.
• Anthropic (Claude): Check for official idempotency support. If absent, use request fingerprints and caching.
• Open-source / self-hosted models: You control the server, so you can implement idempotency at the gateway level.
• Aggregators (like this platform): The aggregator may not expose idempotency keys per backend. In that case, deduplicate on your side before sending the request.

Always consult the specific API documentation for the model you’re calling.

Retry logic with backoff and limits

Implement retries with exponential backoff and a maximum attempt limit. This avoids hammering the server and gives time for the first request to complete.

• Start with a short delay (e.g., 1 second) and double it each attempt.
• Set a maximum number of retries (e.g., 3–5).
• After exhausting retries, treat the operation as failed and handle it accordingly (e.g., alert, fallback).

Handling divergent answers

Even with idempotency, if a retry does occur (e.g., due to a missing key), you might get a different answer. To mitigate:

• Use deterministic parameters if possible (e.g., set temperature to 0).
• Compare responses and flag differences for review.
• For critical operations, store the first successful response and reuse it.

Putting it together

1. Generate an idempotency key for each logical request.
2. Send the request with the key (if supported).
3. On timeout, wait briefly and probe for completion (if possible).
4. If probing is unavailable, check your local deduplication cache.
5. If no completion is found, retry with the same idempotency key and exponential backoff.
6. Limit retries and handle failures gracefully.

By combining idempotency keys, probing, and deduplication, you can retry safely without double-billing or forking answers.