Async Jobs Through One Aggregator Key: Simulating Webhook Callbacks With Only a Synchronous API

Why Simulate Webhooks

Many AI model APIs are synchronous: you send a request and wait for the full response. Some long-running tasks (e.g., large generations, fine-tuning) can be handled asynchronously, but if the API doesn't support webhooks, you need a workaround. A poll-and-notify shim lets you submit a job, poll for completion, and fire a local callback when done—all with a single API key from an aggregator.

How the Shim Works

The shim sits between your application and the aggregator API. It performs three steps:

• Submit: Send the job request to the API and receive a job ID (or a token to poll).
• Poll: Periodically check the job status using the job ID until it's complete.
• Notify: Once complete, trigger a local callback (e.g., HTTP request to your server, message queue, or function call).

This pattern works with any synchronous API that provides a way to query job status. If the API doesn't expose a job ID, you can use idempotency keys or store the request and re-poll.

Implementing the Polling Loop

Use a simple polling loop with backoff to avoid overwhelming the API:

• Start with a short interval (e.g., 1 second) and increase gradually (exponential backoff) up to a maximum.
• Set a timeout to avoid infinite polling.
• Handle errors gracefully: retry on transient failures, fail on permanent errors.

Example pseudo-code:

def poll_job(job_id, callback, initial_interval=1, max_interval=30, timeout=300):
    start = time.time()
    interval = initial_interval
    while time.time() - start < timeout:
        status = api.get_job_status(job_id)
        if status == 'completed':
            callback(api.get_job_result(job_id))
            return
        elif status == 'failed':
            callback(None, error='Job failed')
            return
        time.sleep(interval)
        interval = min(interval * 2, max_interval)
    callback(None, error='Timeout')

Firing Local Callbacks

Your callback can be any local action:

• An HTTP POST to your own server endpoint.
• A message to a queue (e.g., Redis, RabbitMQ).
• Direct invocation of a function in your application.
• A database update.

Ensure the callback is idempotent to handle duplicate notifications.

Using One Aggregator Key

With an aggregator like the one described, you use a single API key for all models. The shim can submit jobs to different models (Claude, GPT, DeepSeek, etc.) and manage them uniformly. Billing is straightforward: you pay the official price × 1.3, and key contributors get credited (official × 1.1, premium × 1.2). This simplifies cost tracking across models.

Handling Failures and Retries

• Implement retries with backoff for transient API errors.
• Log all job submissions and status checks for debugging.
• Set a maximum retry count and a dead-letter queue for failed notifications.

Best Practices

• Respect rate limits: Even without official limits, avoid aggressive polling.
• Use persistent storage: Store job IDs and statuses in a database to survive restarts.
• Monitor: Track job success rates and latency.
• Secure your key: Never expose it in client-side code.

Conclusion

Simulating webhooks with a poll-and-notify shim is a practical way to handle long-running jobs when the API lacks webhook support. With a single aggregator key, you can manage jobs across multiple models and maintain consistent billing. Implement robust polling, idempotent callbacks, and error handling to build a reliable system.