Benchmarking Six Backends on One Key Before You Commit: A Reusable A/B Script

Why benchmark before you commit

Choosing an LLM backend often involves trade-offs in quality, latency, and cost. Rather than relying on anecdotes, run a controlled A/B test across multiple models using the same prompts and evaluation criteria. With one API key from our aggregator, you can access Claude, GPT, DeepSeek, Qwen, GLM, and Kimi without managing separate accounts or credentials.

Prerequisites

• An API key from our platform (top up with USDC on Base, no KYC).
• Python 3.8+ and the requests library.
• A set of test prompts that represent your use case.

Setting up the script

Create a Python script that sends identical requests to each model and logs the responses. The script should be reusable—parameterized by model name and prompt.

import requests
import json
import time

API_KEY = "your_api_key_here"
BASE_URL = "https://api.your-aggregator.com/v1/chat/completions"

MODELS = [
    "claude-3-opus",
    "gpt-4-turbo",
    "deepseek-chat",
    "qwen-max",
    "glm-4",
    "kimi"
]

def query_model(model, prompt, temperature=0.7):
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }
    data = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature
    }
    start = time.time()
    response = requests.post(BASE_URL, headers=headers, json=data)
    latency = time.time() - start
    response.raise_for_status()
    return response.json(), latency

# Example usage
prompt = "Explain the concept of recursion in simple terms."
for model in MODELS:
    result, latency = query_model(model, prompt)
    content = result["choices"][0]["message"]["content"]
    print(f"Model: {model}, Latency: {latency:.2f}s")
    print(content[:200] + "...\n")

Designing your benchmark

• Prompts: Use a diverse set (10–20) covering different tasks: summarization, code generation, creative writing, factual Q&A.
• Metrics: Record latency, token usage (if returned), and subjective quality scores. Consider automated metrics like BLEU or ROUGE for specific tasks, but human evaluation is often more reliable.
• Repetitions: Run each prompt multiple times to average out variability, especially if temperature > 0.

Running the benchmark

1. Save the script as benchmark.py.
2. Replace API_KEY with your key.
3. Adjust the MODELS list to match the exact model identifiers provided by our API.
4. Run: python benchmark.py.

The script will output responses and latencies for each model. For a more thorough analysis, extend it to save results to a CSV file.

Interpreting results

• Quality: Manually review responses or use a rubric. Look for accuracy, coherence, and relevance.
• Latency: Compare average response times. Note that network variability can affect results; run multiple times.
• Cost: Our pricing is official price × 1.3. Use the token counts from responses to estimate costs. Remember that key contributors earn back official price × 1.1 (or × 1.2 for premium models), which can offset costs if you share your key.

Making the decision

Choose the model that best balances your priorities. If cost is critical, consider open-weight models like DeepSeek or Qwen. If quality is paramount, Claude or GPT may be worth the premium. The benchmark script gives you concrete data to justify your choice.

Next steps

• Expand the script to include multi-turn conversations.
• Add error handling and retry logic.
• Integrate with your CI/CD pipeline for continuous evaluation.

By following this approach, you can confidently select the right backend for your application without overspending or compromising on performance.