Idempotency on One Key: Making Every Call Safe to Replay Across Six Backends

Why client-side idempotency matters here

An aggregator that routes one API key to several backends (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) is a convenience layer, not a transaction manager. Routing across providers means a single logical operation may be retried against a different backend than the one that first saw it. Because the aggregator does not deduplicate across backends, the responsibility for making a call safe to replay sits with your client.

Billing compounds the concern: you pay official price × 1.3 per successful call. A duplicated call is a duplicated charge, and a crash between "response received" and "response persisted" leaves you unable to tell whether the work was already billed.

What "idempotent" actually means in this context

• Exactly-once effect, at-least-once transport. Networks fail; clients retry. You cannot guarantee one delivery, but you can guarantee one observable effect.
• Server-side dedup is absent by design. Do not assume the aggregator remembers your request. Assume every HTTP call that reaches a backend may be executed and billed.
• The key is the unit of identity. Two requests are "the same" if and only if they carry the same idempotency key and the same request fingerprint.

Designing the idempotency key

Generate the key before the first attempt and persist it before sending anything.

Good keys are:

• Deterministic from intent — derive from a stable hash of the logical operation (user task ID, conversation turn ID, content hash, model family), not from a timestamp or random value chosen at send time.
• Scoped — include a namespace (tenant, project, workflow) so keys from different contexts never collide.
• Opaque and bounded — a UUIDv4 or a hash digest is fine; avoid embedding raw prompt text or PII.
• Stable across retries — the same logical call must reuse the same key even if the prompt is re-serialized identically.

A workable pattern: key = sha256(namespace + logicalopid + canonicalrequestfingerprint).

The fingerprint should cover model identifier, message content, sampling parameters, and any tool definitions. If the fingerprint changes, it is a new operation and deserves a new key.

Dedup storage: what to record and when

You need a durable store (database table, KV store, or object store) keyed by the idempotency key. Write the intent before dispatch, and the outcome after completion.

For each key, record:

• State — pending, succeeded, failed_permanent.
• Request fingerprint — to detect key reuse with a different payload and reject it.
• Attempt metadata — timestamps, a monotonically increasing attempt counter, and optionally which backend the attempt was routed to.
• Result — the full response body (or a pointer to it), token usage, and any billing-relevant identifiers the aggregator returns.
• Lease/expiry — a timestamp so stale pending entries can be recovered rather than blocking forever.

Use an atomic conditional write ("insert if not exists" or compare-and-set on state) to claim a key. Two concurrent retries must not both proceed.

The state machine to implement

1. Claim — attempt to insert the key in pending. If it already exists in succeeded, return the stored result. If pending and the lease is live, back off.
2. Dispatch — send the request with the key attached as a header or metadata field.
3. Persist outcome — on success, atomically transition to succeeded and store the result in the same transaction if possible.
4. Recover — on client crash or expired lease, re-read the key. If succeeded, replay the stored result without calling again. If pending and the lease expired, you cannot know whether the backend executed; treat this as an ambiguous case and decide explicitly (see below).

Handling the ambiguous window

There is an unavoidable window between "request sent" and "outcome recorded." Options, in increasing cost:

• Accept a bounded double-call risk for cheap, idempotent-by-nature operations (classification, extraction, short completions with no side effects).
• Resolve by reconciliation — if the aggregator exposes usage or billing records tied to your key, query them before retrying a stale pending entry.
• Resolve by side-effect design — keep the LLM call itself free of external side effects; apply side effects (writes, messages, payments) only after the succeeded transition, using the idempotency key again downstream.

Whichever you pick, document it. Silent ambiguity is what produces surprise charges.

Retries: classify before you replay

Not every failure should be retried with the same key:

• Transport timeouts / connection resets — retry with the same key; the request may or may not have executed.
• Explicit provider errors (rate limit, overload) — retry with the same key and exponential backoff plus jitter.
• Deterministic client errors (malformed request, invalid parameters) — do not retry; mark failed_permanent.
• Cross-backend failover — changing the model identifier changes the fingerprint. Decide whether failover is the same logical operation (keep the key, update an attempt log) or a new one (new key). Keeping the key is usually correct when the user asked for "a completion for this task," because the result is still stored once.

Crash-resume on the client

Resuming means reading the dedup store, not re-deriving state from memory:

• On startup, scan for keys in pending older than your lease.
• For each, re-check for a stored result before issuing any new call.
• Replay succeeded results to the rest of your pipeline exactly as if they had just arrived, so downstream code paths are identical.
• Never regenerate a key on resume for an operation that already has one.

Payload design that makes replay safe

• Canonicalize requests before hashing: sort keys, normalize whitespace, pin defaults explicitly.
• Store the canonical request alongside the key so a replay sends byte-identical input.
• Version your fingerprint scheme so a change in canonicalization doesn't silently orphan existing keys.
• Bound result size — store full bodies for small responses; for large ones, store a digest plus a pointer in object storage.

Operational hygiene

• Retention — keep succeeded entries long enough to cover your retry horizon and crash-recovery window, then expire them; keep failed_permanent longer if you want audit trails.
• Observability — log key, state transitions, attempt count, and backend per attempt. Alert on high pending-expiry rates, which signal timeouts or lease misconfiguration.
• Cardinality control — keys should be per logical operation, not per token or per message chunk; runaway key creation defeats dedup and inflates storage.
• Cost awareness — dedup reduces duplicate billing at ×1.3, but it does not change the multiplier. If your team also contributes keys and receives the ×1.1 (or ×1.2 premium) credit in USDC, keep those accounting records separate from request dedup; they answer different questions.

A minimal checklist

• Key derived deterministically from namespace + logical operation + canonical payload hash.
• Persisted before first dispatch, in an atomic claim.
• State machine with lease-based recovery.
• Stored results replayed on resume and on duplicate-key hits.
• Retry policy distinguishes transport, transient, deterministic, and failover cases.
• Documented decision for the ambiguous mid-flight window.
• Retention and observability configured.

None of this requires the aggregator to cooperate. It requires only that your client treat the key as the source of truth and never dispatch a logical operation whose outcome it has not first checked.