Wiring a Multi-Key Failover Pool Into a CLI That Only Accepts One Base URL

The problem: one URL, one key

Many CLI tools and SDKs expose exactly two configuration knobs for model access: a base URL and an API key. That is fine until you want redundancy. If the key behind that URL is rate-limited, misconfigured, or the upstream call fails, the tool has no concept of a second provider to try.

Editing the tool's config or environment on every failure is fragile and manual. A better pattern is to insert a tiny local process — a failover shim — between the tool and the upstream APIs. The tool keeps pointing at a single base URL and a single placeholder key. The shim owns the real key pool and decides which upstream request to make.

This works especially well with an API aggregator that lets you hold several keys across accounts or teams, since the shim can rotate among keys that all speak the same OpenAI-compatible surface.

What the shim is responsible for

Keep the scope narrow. The shim should:

• Listen on 127.0.0.1 at a fixed port and expose an OpenAI-compatible path (for example /v1/chat/completions and /v1/models).
• Accept whatever key the CLI sends — often a static dummy value — and ignore it, or use it only as a routing hint.
• Select a real key from a local pool, attach it as the Authorization: Bearer ... header, and forward the request to the upstream base URL.
• Retry or fail over to the next key when the response indicates a retryable failure.
• Stream responses back to the CLI unchanged, including server-sent events.

It should not transform prompts, cache completions, or reinterpret model names unless you explicitly want aliasing.

Designing the key pool

Store the pool in a local file that is git-ignored and readable only by your user. Each entry needs, at minimum:

• The key value or a reference to a secret store.
• The upstream base URL if keys can target different aggregator endpoints.
• An optional label for logs.
• A mutable health state kept in memory (not in the file).

A flat list is usually enough. Avoid over-engineering with weights or tiers unless you have a concrete need; deterministic ordering plus health tracking handles most cases.

Failover rules that stay predictable

Failover logic is where shims go wrong. Adopt explicit rules:

• Retry before rotating. A single 5xx or connection reset may be transient. Try the same key once more before moving on.
• Rotate on auth and quota signals. A 401 or 403 means the key will not work for this request; jump to the next key immediately.
• Rotate on 429, but respect backoff. A rate-limited key should be marked temporarily unavailable with a cooldown rather than discarded.
• Do not retry non-idempotent surprises blindly. For streaming requests, only fail over if no bytes have been sent to the client yet. Once you have started streaming a response, a mid-stream failure must surface to the caller rather than silently restarting.
• Cap total attempts. A small fixed number of attempts per request prevents a full-pool outage from turning into a retry storm.

Return the last error to the client when every key has been tried, so the CLI sees a normal error rather than a hang.

Tracking health without a database

An in-memory health map is sufficient for a CLI workflow. For each key, track:

• Consecutive failures and the timestamp of the last failure.
• A cooldown expiry for rate-limit responses.
• A simple success counter that clears failure state.

When selecting a key, skip entries whose cooldown has not expired. Prefer the first healthy key in a stable order so behaviour is reproducible. Periodically reset counters so a key that recovers is not penalised forever.

Keeping streams intact

Streaming is the most common compatibility failure. The shim must:

• Forward the request body unmodified, including stream: true.
• Relay response headers that matter (Content-Type, Cache-Control) and let the runtime handle chunked transfer.
• Pass through SSE frames byte-for-byte without buffering the whole stream.
• Flush promptly so the CLI sees tokens as they arrive.

If your language's HTTP client buffers by default, disable that for streaming paths.

Pointing the CLI at the shim

Once the shim runs, configuration is trivial:

• Set the CLI's base URL to http://127.0.0.1:<port>/v1.
• Set its API key to any non-empty placeholder.
• Leave the tool's other settings untouched.

Because the shim speaks the same protocol, the CLI cannot tell the difference. Run the shim in the foreground while debugging and as a background service or launchd/systemd unit when you want it always available.

Billing and cost awareness

If you use an aggregator that charges a transparent multiplier over official prices, the shim does not change that. Requests still settle at the aggregator's rate, and any contributor credit arrangements remain between you and that service. The shim only changes which key pays for a given call; it does not alter per-call pricing. Keep that distinction clear when estimating spend across a pool.

Observability and guardrails

A few small touches make the shim safe to leave running:

• Log the chosen key label, retry count, and final status for each request, but never log key values.
• Emit a lightweight /health endpoint reporting the number of healthy keys.
• Fail closed: if the pool file is missing or empty, return a clear 503 rather than forwarding with no key.
• Bind to loopback only. This process holds real credentials and should never be exposed on a network interface.
• Restrict the pool file's permissions and keep it out of version control.

When this pattern fits

A local failover shim is a good match when a tool's URL and key are fixed, when you already hold several keys for the same aggregator, and when you want failures to degrade gracefully without manual edits. It is less useful when the tool natively supports provider fallback or when you need cross-provider routing with different request schemas — in that case, a proper gateway with model mapping is the better tool.