Same Prompt, Five Samples: Sweeping temperature and Seed Options Across Backends on One Key

Why Compare Sampling Behavior Across Backends?

Different LLM backends expose different sampling controls. Some accept a seed parameter for deterministic sampling, others ignore it. Temperature ranges can vary—some backends clamp values to [0, 1], others to [0, 2]. When you use an aggregator that routes to multiple backends with a single API key, you can systematically compare how these differences affect output stability.

This guide shows you how to build a repeatable sampling harness to sweep temperature and seed options across backends, so you can understand each backend's behavior and make informed choices for your application.

Key Concepts: Seed and Temperature

• Seed: An integer that, when supported, initializes the random number generator for sampling. Identical prompts with the same seed and temperature should produce identical outputs (assuming the backend honors the seed).
• Temperature: Controls randomness. Lower values (near 0) make output more deterministic; higher values increase diversity. Backends may clamp temperature to different ranges.

Not all backends support seed. Some may accept the parameter but ignore it. Always verify by running the same request multiple times.

Step 1: Set Up Your Harness

Your harness should:

• Accept a list of backends to test.
• For each backend, run a fixed prompt with multiple temperature values and seed values.
• Record the exact request parameters and the full response.
• Repeat each condition multiple times to detect non-determinism.

Use a simple script (Python example below) that calls the aggregator's API with your single key. Store results in a structured format (e.g., JSON) for analysis.

import requests
import json
import time

API_KEY = "your-api-key"
BASE_URL = "https://api.example.com/v1/chat/completions"  # replace with actual endpoint

prompt = "Explain the concept of entropy in one sentence."
backends = ["backend-a", "backend-b", "backend-c"]  # replace with actual model identifiers
temperatures = [0.0, 0.5, 1.0, 1.5]
seeds = [42, 123, None]  # None means omit seed
repeats = 3

results = []

for backend in backends:
    for temp in temperatures:
        for seed in seeds:
            for i in range(repeats):
                payload = {
                    "model": backend,
                    "messages": [{"role": "user", "content": prompt}],
                    "temperature": temp,
                }
                if seed is not None:
                    payload["seed"] = seed
                headers = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}
                try:
                    resp = requests.post(BASE_URL, json=payload, headers=headers, timeout=30)
                    resp.raise_for_status()
                    data = resp.json()
                    output = data["choices"][0]["message"]["content"]
                except Exception as e:
                    output = f"ERROR: {e}"
                results.append({
                    "backend": backend,
                    "temperature": temp,
                    "seed": seed,
                    "repeat": i,
                    "output": output,
                })
                time.sleep(0.5)  # avoid rate limits

with open("sampling_results.json", "w") as f:
    json.dump(results, f, indent=2)

Step 2: Handle Backend Differences

When you run the harness, expect these variations:

• Seed support: Some backends may return an error if seed is included, or they may silently ignore it. Capture the raw response to detect errors.
• Temperature clamping: If you send a temperature outside the backend's allowed range, it may clamp to the min/max or return an error. Record the actual temperature used if the API returns it.
• Default behavior: Omitting temperature may default to a backend-specific value. Test with and without explicit temperature.

Adjust your harness to log the full request and response JSON for debugging.

Step 3: Analyze Output Stability

After collecting results, compute stability metrics per backend and condition:

• Exact match rate: For a given backend, temperature, and seed, how often do repeated runs produce identical output?
• Semantic similarity: Use embeddings or simple text distance (e.g., Jaccard) to measure variation when exact match fails.
• Seed effect: Compare runs with the same seed vs. different seeds. A backend that honors seeds will show high exact match rate for the same seed.

Group results by backend and plot stability vs. temperature. You'll likely see that lower temperatures increase stability, but seed support is the bigger factor for determinism.

Step 4: Interpret and Document

Use your findings to:

• Choose backends that support seed for reproducible experiments.
• Understand each backend's effective temperature range to avoid clamping.
• Set realistic expectations for output stability in production.

Document the version of each backend you tested, as behavior may change over time.

Practical Tips

• Start with a small set of backends and conditions to keep runtime manageable.
• Use a fixed prompt that has a clear, factual answer to make differences obvious.
• Include a control condition with temperature=0 and no seed to see baseline determinism.
• Respect rate limits; add delays between requests.
• Store raw responses for auditability.

Conclusion

Building a sampling harness with one API key lets you systematically compare how backends handle seed and temperature. By sweeping parameters and measuring stability, you can make data-driven decisions about which backends to use for tasks that require reproducibility. Remember that user pricing is official price × 1.3, and contributors are credited official price × 1.1 (premium × 1.2) in USDC—so efficient experimentation also helps manage costs.