Writing a Prompt Contract Test That Runs Against All Six Backends on One Key

Why Test Across All Backends?

When you use an LLM API aggregator, you can call multiple models with one key. But different models may interpret the same prompt differently. A prompt that works well on one backend might produce poor results on another. A prompt contract test verifies that your prompt yields acceptable outputs across all supported backends. This is especially useful if you switch models for cost, latency, or capability reasons.

What Is a Prompt Contract?

A prompt contract is a set of assertions about a model's response to a given prompt. Instead of checking exact text, you define properties the response must satisfy: it must be valid JSON, contain certain keywords, fall within a length range, or match a regex. The contract is model-agnostic, so the same assertions apply to every backend.

Setting Up the Test

You'll need:

• An API key from your aggregator (top up with USDC on Base, no KYC).
• A list of model identifiers for the six backends: Claude, GPT, DeepSeek, Qwen, GLM, and Kimi. Refer to the aggregator's documentation for the exact model names.
• A test runner (e.g., pytest, Jest, or a simple script).

Example structure in Python:

import requests

API_KEY = "your-key"
BASE_URL = "https://api.aggregator.example/v1"  # replace with actual endpoint
MODELS = ["claude-model", "gpt-model", "deepseek-model", "qwen-model", "glm-model", "kimi-model"]

def call_model(model, prompt):
    response = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": model, "messages": [{"role": "user", "content": prompt}]}
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

Defining the Contract

Write assertions that any reasonable response should pass. For example, if your prompt asks for a JSON object with specific fields:

import json

def assert_contract(response_text):
    # Must be valid JSON
    data = json.loads(response_text)
    # Must contain required keys
    assert "answer" in data
    assert "confidence" in data
    # Confidence must be a number between 0 and 1
    assert isinstance(data["confidence"], (int, float))
    assert 0 <= data["confidence"] <= 1
    # Answer must be non-empty string
    assert isinstance(data["answer"], str) and data["answer"].strip()

Keep contracts focused on what you truly need. Avoid over-specifying, as models may phrase things differently.

Running the Test Against All Backends

Iterate over the model list and apply the contract to each response. Collect results rather than stopping at the first failure, so you can see which backends fail.

def test_prompt_contract():
    prompt = "Return a JSON object with keys 'answer' (string) and 'confidence' (number between 0 and 1). The answer should explain what a prompt contract is in one sentence."
    failures = []
    for model in MODELS:
        try:
            output = call_model(model, prompt)
            assert_contract(output)
        except Exception as e:
            failures.append((model, str(e)))
    if failures:
        raise AssertionError(f"Contract failures: {failures}")

Handling Model-Specific Quirks

Some models may need slight prompt adjustments, such as a system message or a different temperature. The contract test should use the same prompt and parameters across all backends to ensure fair comparison. If a backend consistently fails, you can decide whether to adjust the prompt (making it more explicit) or exclude that model from your production rotation.

Cost Considerations

Each test run calls all six backends. With the aggregator, you pay the official price × 1.3 for each call. If you are a key contributor, you receive credits at official price × 1.1 (or × 1.2 for premium models) in USDC. To keep costs low, run contract tests only when you change prompts or model versions, not on every commit.

Integrating into CI

Add the test to your CI pipeline as a separate job that runs on demand or on a schedule. Use environment variables for the API key and model list. Since calls incur cost, you may want to run the full matrix only before releases.

Summary

A prompt contract test across all six backends ensures your prompts are robust and portable. By defining model-agnostic assertions and running them against Claude, GPT, DeepSeek, Qwen, GLM, and Kimi with a single key, you catch inconsistencies early. Keep contracts simple, run tests judiciously, and leverage the aggregator's unified billing to manage costs.