Same Key, Backup Model: Automatic Fallback When One Model Is Down or Rate-Limited

Why Fallback Matters

When you rely on a single LLM provider, any downtime or rate limit can break your application. Even the most reliable APIs occasionally return 429 (Too Many Requests) or 5xx (Server Error) responses. Instead of showing an error to your users, you can automatically retry the request with a different model. This pattern is called fallback, and it's essential for production systems that need high availability.

With an LLM API aggregator that provides a single key for multiple models, fallback becomes straightforward: you don't need to manage separate API keys or endpoints for each provider. You simply switch the model parameter and retry the same request.

Understanding the Error Types

Before implementing fallback, it's important to know which errors you should handle:

• 429 Too Many Requests: You've hit the rate limit for a specific model. Retrying the same model immediately will likely fail again, so switching to another model is a good strategy.
• 5xx Server Errors (500, 502, 503, 504): The provider is having issues. These are often transient, but if they persist, fallback to another model keeps your service running.
• Other errors (400, 401, 403, etc.): These usually indicate a problem with your request or authentication, and retrying with a different model won't help. You should handle them separately.

Designing a Fallback Chain

A fallback chain is an ordered list of models to try. When the primary model fails with a retryable error, you move to the next model in the chain. Here are some considerations:

• Similar capabilities: Choose models that can handle the same type of task. For example, if your primary is a general-purpose chat model, your fallback could be another general-purpose model from a different provider.
• Different providers: To avoid correlated failures, pick models from different providers (e.g., Claude, Qwen, GLM). This way, if one provider is down, another is likely still available.
• Cost and performance: Fallback models might have different pricing or latency. Be aware of the trade-offs, but prioritize keeping your application responsive.

Implementing Fallback with an OpenAI-Compatible Endpoint

Most LLM aggregators offer an OpenAI-compatible endpoint. This means you can use the official OpenAI SDK or any HTTP client. The key is to catch specific exceptions and retry with a different model.

Here's a Python example using the openai library:

from openai import OpenAI, APIError, RateLimitError

client = OpenAI(
    api_key="YOUR_AGGREGATOR_KEY",
    base_url="https://api.aggregator.com/v1"  # replace with actual base URL
)

# Define your fallback chain
MODELS = ["claude-3-sonnet", "qwen-max", "glm-4"]

def chat_with_fallback(messages, max_retries_per_model=1):
    last_exception = None
    for model in MODELS:
        for attempt in range(max_retries_per_model):
            try:
                response = client.chat.completions.create(
                    model=model,
                    messages=messages,
                    timeout=30  # set an appropriate timeout
                )
                return response.choices[0].message.content
            except (RateLimitError, APIError) as e:
                # Check if the error is retryable (429 or 5xx)
                if isinstance(e, RateLimitError) or (isinstance(e, APIError) and 500 <= e.status_code < 600):
                    last_exception = e
                    # Optionally add a small delay before retrying
                    continue
                else:
                    # Non-retryable error, raise immediately
                    raise
    # If all models fail, raise the last exception
    raise last_exception

# Usage
messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
try:
    answer = chat_with_fallback(messages)
    print(answer)
except Exception as e:
    print(f"All models failed: {e}")

Handling Retries and Backoff

Simply retrying immediately might not be enough if the rate limit is still active. Consider adding a short delay before retrying the same model, or skip retrying the same model and go straight to the next one. For 5xx errors, a brief exponential backoff (e.g., wait 1 second, then 2 seconds) can help if the issue is transient.

However, since you have multiple models, you can often just move to the next model without waiting. This reduces latency for your users.

Using the Aggregator's Single Key

The beauty of using an aggregator is that you use one API key for all models. The base URL and authentication remain the same; only the model parameter changes. This simplifies your code and configuration. You don't need to store multiple API keys or handle different authentication schemes.

Monitoring and Logging

To understand how often fallbacks occur, log when a fallback is triggered and which model ultimately succeeded. This can help you adjust your primary model choice or negotiate better rate limits if needed. Also, monitor the error rates to detect systemic issues.

Conclusion

Automatic fallback across multiple models is a robust pattern for building reliable LLM applications. By leveraging a single API key from an aggregator, you can easily switch between models like Claude, Qwen, and GLM when errors occur. The code example above provides a starting point; adapt it to your language and framework. Remember to handle non-retryable errors separately and to test your fallback logic thoroughly.