Choosing Per-Backend Timeouts on One Aggregator Key: Why the Same 30s Setting Fails Differently Across Six Backends

Why one timeout value is a trap

When you use a single API key to access multiple model backends—Claude, GPT, DeepSeek, Qwen, GLM, Kimi—each backend has its own latency profile. A 30-second timeout might be generous for one backend and dangerously tight for another. Worse, a timeout that is too long lets a slow backend tie up resources and stall your whole pipeline.

The solution is to derive timeout and retry budgets per backend from your own observed latency data, then configure your OpenAI-compatible client accordingly.

Gather latency data first

Before setting any timeout, you need real numbers. Log the response time for every request you make, grouped by backend. After a few hundred calls, compute the p50 (median) and p99 (near-worst-case) latency for each backend.

• p50 tells you the typical experience.
• p99 tells you how slow things get in the tail.

Use separate buckets for different prompt sizes or model variants if they differ significantly.

Derive timeout and retry budgets

A good starting point is to set the timeout to a multiple of your observed p99—say 1.5× to 2× the p99. This gives headroom for normal variance while still cutting off truly stalled requests.

For retries, consider how many times you are willing to retry a failed request and what total time budget you can afford. A common pattern is:

• Timeout per attempt: 2 × p99 for that backend.
• Max retries: 1 or 2 for idempotent operations.
• Overall deadline: sum of all attempt timeouts plus a small buffer.

Avoid fixed retry counts that ignore the backend’s typical latency—a backend with a high p99 may need fewer retries to stay within your total budget.

Configure per-backend settings in an OpenAI-compatible client

Most OpenAI-compatible clients allow you to set a default timeout and override it per request. Use the default for fast backends and override for slower ones.

Example pseudo-code:

client = OpenAIClient(api_key="YOUR_AGGREGATOR_KEY", base_url="https://api.aggregator.example/v1")

backend_timeouts = {
    "claude-3-opus": 45.0,   # derived from p99
    "gpt-4": 30.0,
    "deepseek-chat": 60.0,
    "qwen-max": 55.0,
    "glm-4": 40.0,
    "kimi": 50.0,
}

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[...],
    timeout=backend_timeouts["deepseek-chat"],
)

If your client supports a global timeout and per-request overrides, set a conservative global timeout and always pass the model-specific value.

Prevent a slow backend from stalling the pipeline

Even with per-backend timeouts, a slow backend can still block if you process requests sequentially. Use asynchronous calls or a thread pool with a maximum concurrency limit per backend. Combine this with circuit breakers: if a backend’s error rate or timeout rate spikes, temporarily route around it.

• Concurrency limits: cap the number of in-flight requests per backend based on its typical latency and your throughput needs.
• Circuit breakers: after N consecutive timeouts, skip that backend for a cooldown period.
• Separate queues: give each backend its own queue so a backlog on one does not delay others.

Monitor and adjust

Latency changes over time. Re-evaluate your p50/p99 monthly and after any major model update. Adjust timeouts and retry budgets accordingly. Log timeout occurrences and track how often they happen per backend—this is your early warning system.

Cost and reliability trade-offs

Remember that you pay official price × 1.3 for all calls, and key contributors get credited official price × 1.1 (premium × 1.2) in USDC. Timeouts that are too aggressive may cause retries, increasing cost. Timeouts that are too lax may stall your pipeline and hurt user experience. Balance both by using data-driven per-backend settings.