Same Prompt, Different Answer Every Run on the Aggregator: Isolating Non-Determinism

When you call a model through an aggregator, the same prompt can produce different answers on each run. This is expected—language models are stochastic by design. But before you jump to seeds or model pinning, it helps to isolate where the variance is coming from. This guide outlines a method to separate three common sources: sampling settings (like temperature), backend routing, and model version differences.

Why variance happens

Run-to-run variance in LLM outputs typically stems from:

• Sampling randomness: Most models use probabilistic sampling (e.g., temperature, top-p). Higher values increase diversity.
• Backend routing: Aggregators may route your request to different infrastructure (e.g., different regions, providers, or hardware) that host the same model. This can introduce subtle differences in output.
• Model version drift: The underlying model behind a name (e.g., claude-3-5-sonnet) may be updated or served by different snapshots over time.

Understanding which factor dominates for your use case helps you decide whether to adjust sampling parameters, pin a specific version, or accept the variance.

Step 1: Control sampling parameters

Start by fixing all sampling-related parameters to their most deterministic settings:

• Set temperature to 0 (if supported). Note that even at temperature 0, some models may not be fully deterministic due to floating-point non-associativity or batching.
• Set top_p to 1 (or disable nucleus sampling if the API allows).
• Set top_k to 1 if available.
• Avoid seed for now—we want to observe natural variance without it.

Run the same prompt multiple times with these settings. If outputs are identical, sampling was the primary source. If not, proceed to the next step.

Step 2: Check backend routing effects

Aggregators may not guarantee that all requests for a given model go to the same backend instance. To test routing effects:

• Run the same prompt with fixed sampling settings (as in Step 1) but at different times or from different geographic locations (if possible).
• Observe whether outputs cluster into distinct groups. If you see a few distinct answer patterns, routing to different backends might be the cause.
• Some aggregators expose headers or metadata about which backend served the request. Check your aggregator's documentation for such details.

If routing seems to be the culprit, you might consider whether the aggregator offers a way to pin to a specific backend (though this is often not exposed).

Step 3: Rule out model version differences

Model versions can change silently. To test this:

• Use the aggregator's model listing endpoint (if available) to see if multiple versions of the same model are listed (e.g., claude-3-5-sonnet-20240620 vs claude-3-5-sonnet-20241022).
• If the aggregator supports explicit version pinning, call the same prompt against a specific version multiple times. If outputs are stable, version drift was the issue.
• If no version pinning is available, compare outputs over time. A sudden shift in output style or content for the same prompt may indicate a model update.

Step 4: Introduce seeds only after isolation

Once you've determined that sampling is the main source of variance (and routing/version are stable), you can try using a seed parameter if the model supports it. Seeds make sampling reproducible given the same model and backend, but they do not guarantee identical outputs across different backends or versions.

Step 5: Decide on mitigation

Based on your findings:

• If sampling is the main source: lower temperature, use seeds, or accept variance.
• If routing is the main source: check if your aggregator offers sticky sessions or backend pinning (unlikely), or consider using a single provider directly.
• If model version is the main source: pin to a specific version if the aggregator allows it.

Practical tips

• Keep a log of prompts, parameters, and outputs to track changes over time.
• Use the aggregator's model metadata (if available) to detect version changes.
• Remember that even with temperature 0, some models may produce slight variations due to hardware or batching differences.

By systematically isolating these factors, you can make informed decisions about how to achieve consistent outputs when needed, without unnecessary complexity.

How this site can help

Our aggregator provides a unified API for many models. You can top up with USDC on Base (no KYC) and use a single API key. Pricing is official price × 1.3, and key contributors are credited official price × 1.1 (premium × 1.2) in USDC. This makes it easy to experiment with different models and settings while keeping costs transparent.