Rotating an Aggregator Key With Zero Downtime: Draining Old Keys While New Calls Flow

Why key rotation needs a drain, not a switch

Rotating an API key looks like a one-line change: replace the old secret with a new one. But with LLM calls — especially long-running agent sessions like Claude Code — an abrupt swap can kill requests that are already in flight. The goal is zero downtime: new calls use the successor key, while older streams are allowed to finish before the predecessor is revoked.

This checklist walks through that pattern. It is conceptual; exact tooling depends on your stack.

Step 1: Issue a successor key first

Never revoke before the replacement exists. Create the new key while the old one remains active, so both are valid for a short overlap window.

• Confirm the new key can authenticate and call a cheap model.
• Store it in the same secret manager or environment path as the old key.
• Tag it with a rotation date so you can audit later.

With a single-key aggregator, you can hold both keys simultaneously — the old key is not invalidated by creating a new one.

Step 2: Make routing switchable without redeploying

If the key is baked into a deployment, rotation forces a restart, which is exactly what you want to avoid mid-session. Instead, load keys from a source you can change live:

• Environment variables injected at boot are fine for short-lived jobs, but awkward for long streams.
• A config service or secret store that workers poll gives you a switch without restarts.
• A thin proxy layer in front of the aggregator can map logical key names to real secrets, so clients never hold the secret directly.

Whichever you choose, the switch should flip new requests to the successor key while leaving existing connections untouched.

Step 3: Route new traffic to the successor key

Once both keys are valid, change the routing so that new requests use the successor. Existing connections continue on the old key. This is the core of zero-downtime rotation: the cutover applies to the next request, not the current one.

• Verify with a fresh request that the new key is in use (for example, through logs or a metering dashboard).
• Keep the old key valid for the entire drain window.
• Watch error rates and latency for a short period after the switch.

Step 4: Let in-flight streams finish on the old key

Streaming responses and agent loops can run for minutes. During this window:

• Do not revoke the old key.
• Do not restart workers that hold open streams.
• Monitor active sessions per key so you know when the old key has no traffic.

The drain ends when the old key has no active requests — not when the clock says so. If you use a proxy, it can report connection counts per key.

Step 5: Revoke the predecessor

Only after the old key shows zero in-flight calls should you revoke it. Revocation should be deliberate and logged.

• Revoke from the aggregator dashboard or API.
• Remove the old secret from config stores and CI variables.
• Keep the rotation record: issue date, cutover time, revoke time, and who performed it.

Claude Code sessions and other long-lived agents

Claude Code and similar tools hold a session open across many model calls. A rotation that swaps the key under a live session can surface as mid-task authentication errors. The drain pattern avoids this by treating the session as in-flight until it completes.

Practical notes:

• If your agent client re-reads credentials per request, a config-driven switch is enough.
• If it caches credentials for the session lifetime, start new sessions on the successor key and let existing sessions finish.
• Avoid revoking the old key during a known long-running task.

A minimal rotation checklist

• [ ] Create successor key; test authentication.
• [ ] Make key routing live-switchable (config store or proxy).
• [ ] Switch new requests to the successor.
• [ ] Monitor old key until in-flight streams reach zero.
• [ ] Revoke the old key; remove it from secret stores.
• [ ] Record the rotation for audit.

Why this matters for metered aggregators

On a pay-per-call aggregator, every request is metered against a key. A botched rotation can mean failed streams, retries, and duplicate spend. Draining before revoking keeps billing clean and sessions stable. It also gives you a reversible step if something looks wrong: while both keys are valid, you can switch back without user-visible impact.