Long-Running Calls Through the OpenAI-Compatible Endpoint: Tuning Client Socket Timeouts and Keep-Alive Against Six Backends

Why timeouts matter with an aggregator

When you point your OpenAI SDK at an aggregator, a single base URL fronts multiple upstream providers. Each backend (e.g., Claude, GPT, DeepSeek, Qwen, GLM, Kimi) can have different typical latency, especially for long generations or reasoning-heavy tasks. A socket timeout that works for a fast chat completion may be too short for a slow, complex request routed to a different model.

The aggregator handles routing and billing, but your client still owns the connection to the aggregator’s endpoint. If your client socket times out mid-response, you may lose tokens you already paid for. Tuning timeouts and keep-alive at the client is therefore a practical reliability concern, not just a performance tweak.

Understand the two timeout layers

Most HTTP clients expose at least two distinct timeouts:

• Connection (or socket connect) timeout – how long to wait for the TCP (and TLS) handshake to complete.
• Read (or socket read/socket timeout) timeout – how long to wait between bytes after the connection is established.

Some clients also have a total request timeout that covers the entire request lifecycle. Aggregated backends can vary in both handshake and generation time, so setting both layers explicitly is safer than relying on defaults.

For streaming responses, the read timeout should be interpreted as the maximum gap between chunks, not the total stream duration. Non-streaming responses require a total timeout that accommodates the full generation.

Set timeouts per request, not globally

A single global timeout forces you to choose the lowest common denominator. Instead, derive the timeout from the expected workload:

• For short, interactive completions, a shorter read timeout is usually fine.
• For long-form generation or tool-calling loops, increase the read timeout or switch to streaming.
• For streaming, keep the read timeout small (e.g., enough to cover typical inter-chunk latency) and rely on the overall connection staying alive; if a backend pauses longer than that, you may need a higher value.

Many OpenAI-compatible clients accept a timeout parameter per request. Use it to override the default for calls that you know may be slow.

Tune keep-alive and connection pooling

Keep-alive reduces handshake overhead but introduces state. When the aggregator routes to different backends, an idle pooled connection to the aggregator is still reusable; the aggregator handles the upstream connection. That means keep-alive against the aggregator is generally beneficial and does not need to be disabled per backend.

However, long-lived idle connections can be closed by intermediaries (load balancers, NATs) without notice. To avoid using a dead connection:

• Set a reasonable keep-alive expiry (e.g., a few minutes) at the client.
• Enable TCP keep-alive probes if your HTTP client supports it.
• Limit the connection pool size to avoid exhausting file descriptors when concurrency is high.
• For streaming requests, avoid reusing a connection that has been idle for a long time; a fresh connection may be more reliable.

Streaming changes the timeout model

With non-streaming calls, the client waits for the full response. With streaming, the client reads chunks as they arrive. This shifts the risk from “total request time” to “time between chunks.”

• Set the read timeout to cover the maximum expected pause between chunks for the slowest backend you use. If a reasoning model emits a long internal deliberation before the first token, the time to first byte can be large even if subsequent chunks are fast.
• Consider a separate, larger timeout for the initial response (headers) versus the streaming body, if your client allows it.
• Handle graceful cancellation: if you abort a stream, you may still be billed for tokens already generated. Design retries and cancellation carefully.

Practical client configuration checklist

• Use explicit timeouts for connect and read; avoid infinite timeouts except for debugging.
• Override per request when calling models known to be slower or when using long prompts.
• Prefer streaming for long outputs to keep the socket active and detect stalls early.
• Set keep-alive expiry shorter than typical intermediary idle timeouts.
• Limit pool size and reuse connections for sequential calls to the same aggregator endpoint.
• Log timeout events with the model name and request shape so you can adjust thresholds over time.
• Retry idempotent requests with backoff, but be aware that retries may incur additional charges if the first request partially succeeded.

Example: Python OpenAI client

from openai import OpenAI

client = OpenAI(
    base_url="https://your-aggregator.example/v1",
    api_key="sk-...",
    timeout=60.0,  # total request timeout in seconds
)

# Per-request override for a potentially slow call
resp = client.chat.completions.create(
    model="some-model",
    messages=[{"role": "user", "content": "..."}],
    timeout=120.0,
)

For streaming, the same timeout applies to the overall request in many clients; check your client’s documentation for how it handles read timeouts during streaming. Some clients expose separate connect and read timeouts via a custom httpx client.

Example: Node.js OpenAI client

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://your-aggregator.example/v1",
  apiKey: "sk-...",
  timeout: 60000, // milliseconds
});

// Per-request override
await client.chat.completions.create({
  model: "some-model",
  messages: [{ role: "user", content: "..." }],
}, { timeout: 120000 });

Node’s default HTTP agent reuses connections; you can tune keepAlive and maxSockets via the http.Agent if needed.

Monitoring and iterating

Because you cannot assume fixed latency across six backends, treat timeout values as tunable parameters. Instrument your client to record:

• Time to first byte
• Total request duration
• Inter-chunk gaps for streaming
• Timeout/abort frequency per model

Adjust thresholds based on observed distributions, not guesses. A timeout that is too aggressive increases retries and cost; one that is too lax can hang workers.

Billing note

The aggregator charges official price × 1.3 for usage. Key contributors are credited official price × 1.1 (premium × 1.2) in USDC. Timeouts and retries affect how many tokens you are billed for, so reliable client configuration indirectly protects your balance. There is no claim here that tuning reduces your rate—only that avoiding premature disconnects and unnecessary retries helps you get the tokens you pay for.