Kicking Off Long Jobs Through the Aggregator: Timeouts, Retries and Where to Check Progress

Why Long Jobs Break Synchronous Calls

Many LLM tasks—summarizing large documents, multi-step agent workflows, batch classification—can exceed the default timeout of a single HTTP request. When you call a model through an aggregator, the request passes through additional network layers, so the round-trip time may be longer than a direct provider call. Treating every job as a synchronous request is fragile: a timeout does not mean the model never processed the input, only that your client gave up waiting.

Design for Asynchronous Workflows

Instead of holding a connection open, structure long jobs as asynchronous tasks:

• Submit and poll: Send the job to your own queue or worker, return an immediate acknowledgement, and store the aggregator's response when it arrives.
• Use streaming where supported: Streaming keeps the connection alive and delivers tokens incrementally, which can reduce perceived latency and help avoid idle timeouts.
• Break work into chunks: Split large inputs into smaller requests that complete within typical timeout windows, then assemble the results.

Set Timeouts Deliberately

A single global timeout is rarely optimal. Consider:

• Connect timeout vs read timeout: Set a shorter connect timeout to fail fast on network issues, and a longer read timeout for generation.
• Per-model expectations: Different models have different latency profiles; tune timeouts per model family rather than using one value everywhere.
• Client-side and server-side limits: Your HTTP client, any proxy, and the aggregator may each enforce timeouts. Align them so the shortest limit does not cut off a valid request prematurely.

Retry with Care

Retries are essential but can amplify load and cost if misused.

• Retry only on transient failures: Network errors, 5xx responses, and rate-limit responses are candidates. Do not retry on 4xx client errors such as malformed requests.
• Use exponential backoff with jitter: Avoid synchronized retry storms. Increase delay between attempts and add randomness.
• Cap attempts: A small maximum (e.g., a few tries) prevents infinite loops.
• Make retries idempotent: If your job has side effects—writing to a database, sending an email—ensure a retry does not duplicate them. Use idempotency keys or deduplication logic on your side.
• Respect rate limits: If the aggregator returns a rate-limit signal, back off and consider queuing rather than immediate retry.

Where to Check Progress

The aggregator is an API gateway, not a job orchestrator. It does not provide a built-in dashboard for your long-running tasks. Progress tracking is your responsibility:

• Log request IDs: Capture any request or trace identifiers returned by the aggregator so you can correlate logs.
• Store state in your own system: Record job status (submitted, processing, completed, failed) in your database or queue.
• Use your provider dashboard for usage: The aggregator's console typically shows API key usage and billing, not per-job progress. For execution status, rely on your application's logs and metrics.
• Monitor latency and error rates: Instrument your client to track timeouts and retries. A rising retry rate often signals that timeouts need adjustment.

Practical Patterns

• Worker queue: Enqueue jobs, let workers call the aggregator with appropriate timeouts, and update job status on completion.
• Webhook or callback: If your architecture supports it, have the worker notify your system when a job finishes, rather than polling.
• Dead-letter queue: Route jobs that exhaust retries to a separate queue for manual inspection.
• Circuit breaker: If a model or endpoint repeatedly fails, pause requests temporarily to avoid cascading failures.

Cost Awareness

Long jobs can generate many tokens. Since you pay the official price multiplied by 1.3 for usage, and contributors receive a credit of official price × 1.1 (or × 1.2 for premium models) in USDC, it is worth estimating token usage before launching large batches. Break jobs into measurable units so you can track spend per job and avoid runaway costs from retries or oversized inputs.

Summary

Treat long jobs as asynchronous tasks with deliberate timeouts, cautious retries, and your own progress tracking. The aggregator handles model access and billing; your application handles orchestration and observability. This separation keeps long-running workloads reliable and cost-effective.