Same Prompt, Same seed, Six Backends on One Key: Measuring How Much Actually Stays Fixed

Why seed-based determinism is a spectrum

When you send the same prompt and the same seed to different model backends, you might expect identical outputs. In practice, determinism varies by provider, model family, and even model version. Some backends treat the seed as a hard constraint, others as a soft hint, and some ignore it entirely. This article shows how to measure what actually stays fixed across six backends (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) using a single API key from the aggregator, and how to build pipelines that remain replay-safe when determinism is only partial.

What seed support means in practice

A backend that honors the seed will produce the same output for the same input, seed, and model version (assuming no other stochastic settings change). But even then, determinism is not guaranteed across:

• Model version updates (e.g., a new snapshot)
• Changes in temperature, top_p, or other sampling parameters
• Backend-side nondeterminism (e.g., different hardware or batching)

The aggregator passes your seed to the backend; it does not enforce determinism. So your first step is to empirically test which backends actually respect the seed.

A minimal test harness

Design a simple experiment that you can run against each backend. The goal is to quantify how much output varies when you change only the seed, and when you repeat the same seed.

1. Choose a fixed prompt that is relatively short but non-trivial (e.g., "List three uses for a paperclip.").
2. Set other parameters to fixed values: temperature = 0, top_p = 1 (or as close as the backend allows). Note that some backends ignore temperature when a seed is provided, but fix it anyway.
3. For each backend, run the same prompt multiple times with the same seed (e.g., seed=42). Record the outputs.
4. Then run the prompt with different seeds (e.g., 1, 2, 3) and record outputs.
5. Compare within-seed runs to see if the backend is deterministic for a given seed. Compare across seeds to see if the seed changes the output at all.

Use the aggregator's single API key to call each backend by specifying the model name. The exact model identifiers (e.g., claude-3-opus, gpt-4, deepseek-chat) are listed in the aggregator's documentation.

Interpreting the results

After running the tests, you will likely see three patterns:

• Seed-honoring and deterministic: Same seed → identical output every time; different seed → different output. This is the ideal case.
• Seed-ignoring but deterministic: Same seed → identical output every time (because temperature=0 and no other randomness), but different seeds also produce the same output. In this case, the seed has no effect, but the output is stable.
• Seed-ignoring and nondeterministic: Outputs vary even with the same seed, often due to backend nondeterminism.

Document which backends fall into which category for your chosen models and parameters. Keep in mind that results may change with model updates.

Designing replay-safe pipelines

Even if some backends honor the seed, you cannot rely on perfect determinism across all backends and over time. Build your pipeline with these principles:

• Log the full request and response: Store the model name, all parameters (including seed), the exact prompt, and the raw response. This lets you replay later, even if the backend changes.
• Version your prompts and model identifiers: When a model version changes, outputs may differ even with the same seed. Pin model versions where possible, and treat prompt changes as new experiments.
• Use the seed for reproducibility, not as a guarantee: The seed is a tool to increase stability, but expect some variation. Design downstream logic to tolerate small differences (e.g., by using fuzzy matching or structural checks).
• Fallback to deterministic post-processing: If you need exact reproducibility (e.g., for testing), consider storing outputs and using them as fixtures rather than re-calling the API.
• Test across backends: If your pipeline can route to multiple backends, run your seed tests per backend and document their behavior. This helps you choose the right backend for tasks that require higher determinism.

Cost considerations

Every API call, including test runs, incurs cost. The aggregator charges official price × 1.3 for usage, and contributors are credited official price × 1.1 (or × 1.2 for premium models) in USDC. Keep your test suite small and focused. Use short prompts and low max_tokens to minimize expense. Remember that the seed parameter does not affect pricing.

Conclusion

Seed-based determinism is not uniform across backends. By running controlled experiments through the aggregator, you can measure which backends honor the seed and how stable their outputs are. Use that knowledge to design pipelines that log everything, version dependencies, and tolerate partial determinism. This approach keeps your system replay-safe even when the underlying models behave differently.