Hitting a Wall Mid-Batch: Concurrency, Rate Limits and Queueing on One Aggregator Key

The Symptom: Failures That Appear Only Under Load

A batch job that runs fine with one worker starts throwing errors once you fan out to ten. The prompts didn't change. The model didn't change. What changed is the shape of the traffic arriving at the provider.

When many workers share one API key through an aggregator, those workers are not independent tenants. They look like a single noisy client to every upstream model provider behind the key. Providers enforce their own concurrency and rate limits, and the aggregator has to enforce some discipline on your behalf so one key can't starve every other caller.

Why Shared Keys Fail Differently Than Per-Worker Keys

• One bucket, many consumers. Rate limits are usually counted per key or per account. Ten workers don't get ten budgets; they split one.
• Bursts beat averages. A pool that averages a safe request rate can still spike above the limit when several workers unblock at the same moment.
• Errors compound. A timeout triggers a retry, the retry adds load, the added load causes more timeouts. Without jitter and a cap, retries become the outage.
• Retries are invisible in your metrics. If your worker counts requests but not attempts, you will underestimate real traffic by the retry multiplier.

Sizing a Worker Pool

The right pool size is not a round number you pick once. Derive it from observed behavior:

1. Start with one worker and measure. Record latency percentiles per request and completion time per batch. This is your baseline.
2. Double the pool and re-measure. Watch both throughput and error rate. Throughput should rise roughly linearly while errors stay flat.
3. Find the inflection point. Keep increasing until error rate or p95 latency begins to climb. The pool size just below that point is your practical ceiling for that workload and model.
4. Re-measure per model and per workload. A long-context generation workload and a short classification workload will saturate at different pool sizes. Store the ceiling alongside the model name.

A useful rule of thumb: if raising concurrency by a step increases total completed work by less than the step size, you are past the useful range.

Distinguishing the Failure Modes

Not every error means "slow down." Classify before you react:

• Rate-limit rejections are explicit. Back off and retry after the indicated delay.
• Timeouts may mean the provider is slow, your prompt is large, or your own network is saturated. Retrying immediately usually makes it worse.
• 5xx from upstream are often transient but can also indicate a request the provider will never accept. Cap retries.
• 4xx are almost always your payload. Do not retry these; fix the request.

Handle each class with its own policy rather than a single generic retry wrapper.

Backoff That Actually Helps

• Exponential backoff with full jitter. Randomize the delay across the whole backoff window, not just add a small jitter. This spreads retries from many workers so they don't reconverge into a second burst.
• Cap the delay and the attempts. Three to five attempts is usually enough; beyond that the batch is better off parked and revisited later.
• Respect retry hints. If a response tells you how long to wait, use it as the floor for your delay.
• Circuit-break on sustained failure. If a model keeps failing across many workers, stop sending it traffic for a cooling period and route new work elsewhere or pause the queue.
• Count attempts, not just requests. Track retry rate as a first-class metric. A rising retry rate is your earliest warning that the pool is too big.

Buffering Work Locally

The alternative to shrinking the pool is to keep the pool small and feed it from a durable local queue.

• Decouple submission from execution. Write every unit of work to a local queue (a table, a file-backed queue, or an embedded broker) before any worker touches it.
• Let workers pull. Push-based fan-out creates bursts; pull-based workers take only as much as they can finish.
• Checkpoint aggressively. Mark each item done as soon as the response is persisted, so a crash costs you one item, not one batch.
• Persist partial results. If a batch of ten items fails on item seven, the first six should not be re-run.
• Add idempotency keys on writes downstream so a retried item can't create duplicates.

This design turns a concurrency problem into a throughput problem, which is far easier to reason about.

A Practical Sequence

1. Measure baseline throughput and latency with a single worker.
2. Classify errors by type and record attempt counts.
3. Introduce a pull-based local queue with checkpoints.
4. Add per-class retry policies with capped, jittered backoff.
5. Raise worker count stepwise until errors or p95 latency rise, then step back one.
6. Record the ceiling per model and workload and use it as the default.
7. Re-tune when you switch models, change prompt sizes, or shift the batch profile.

Cost and Accounting Notes

Aggregator pricing is transparent and fixed: users pay the official model price multiplied by 1.3, and key contributors are credited the official price multiplied by 1.1 (or 1.2 for premium models) in USDC. Practically, that means retries and wasted calls cost you real money at the same multiplier as successful ones. Reducing retry storms is one of the few levers that improves both reliability and spend at once.