Swapping a Custom In-House LLM Backend for the Aggregator's OpenAI-Compatible Route

Why Swap the Backend?

If you've built a custom in-house LLM backend, you already have a working client integration. The aggregator's OpenAI-compatible route lets you keep that client but redirect calls to a broader set of models with a single API key. This guide covers how to repoint your client and keep the old backend as a fallback.

Understand the OpenAI-Compatible Route

The aggregator exposes an endpoint that mirrors the OpenAI Chat Completions API. That means:

• The request and response schemas match OpenAI's format.
• Existing OpenAI SDKs or HTTP clients work with minimal changes.
• You can switch between models (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) by changing the model name.

To use it, you need:

• An API key from the aggregator (obtained after topping up with USDC on Base).
• The base URL of the aggregator's API.

Repoint Your Client

Step 1: Update the Base URL and API Key

In your client configuration, change the base URL to the aggregator's endpoint and set the API key. For example, in Python with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.aggregator.com/v1",  # replace with actual base URL
    api_key="YOUR_AGGREGATOR_KEY"
)

If you use raw HTTP, update the request URL and headers accordingly.

Step 2: Adjust the Model Name

Replace your self-hosted model identifier with a model name supported by the aggregator. Use the exact model string from the aggregator's documentation. You can typically list available models via an endpoint like /models.

Step 3: Verify Request and Response Compatibility

The OpenAI-compatible route should accept the same fields you already use: messages, temperature, max_tokens, etc. Check for any differences:

• Some aggregator-specific parameters may not be supported.
• Response objects include standard fields like choices, usage, etc.

Test with a simple call and inspect the response.

Keep the Old Backend as Fallback

Why Keep a Fallback?

• Resilience: If the aggregator is temporarily unavailable, your service can degrade gracefully.
• Cost control: The aggregator charges official price × 1.3. For high-volume, non-critical workloads, your self-hosted model may be cheaper.
• Compliance: Some data may need to stay on-premises.

Implement a Fallback Strategy

• Health checks: Periodically ping the aggregator's endpoint. If it fails, switch to the old backend.
• Error handling: On specific errors (timeouts, 5xx), retry with the fallback.
• Routing logic: Decide at runtime which backend to use based on model, data sensitivity, or load.

Example in Python:

def call_llm(messages, model, use_fallback=False):
    if not use_fallback:
        try:
            return aggregator_client.chat.completions.create(model=model, messages=messages)
        except Exception as e:
            # Log error
            pass
    return old_backend_client.chat.completions.create(model=model, messages=messages)

Consider a Gateway

Use a library like LiteLLM or a simple proxy that abstracts both backends. This lets you switch with configuration rather than code changes.

Handle Authentication and Billing

The aggregator uses a single API key and charges per token at official price × 1.3. Ensure your client sends the key correctly and monitor usage. If you contribute keys, you earn credits at official price × 1.1 (premium × 1.2) in USDC, which can offset costs.

Migrate Gradually

• Start by routing a small percentage of traffic to the aggregator.
• Compare outputs and latency.
• Gradually increase traffic as confidence grows.
• Keep the old backend as a fallback until you're ready to decommission it.

Test Thoroughly

• Unit test the new client configuration.
• Integration test with real API calls.
• Load test to ensure the aggregator handles your volume.
• Simulate failures to verify fallback behavior.

Conclusion

Swapping to the aggregator's OpenAI-compatible route is straightforward because your existing client already speaks the right protocol. By keeping the old backend as a fallback, you gain flexibility and resilience without sacrificing reliability.