Same temperature=0, Six Backends, One Key: Why 'Deterministic' Settings Still Drift Between Vendors

The myth of the universal temperature=0

Setting temperature: 0 feels like a promise: same prompt, same answer. In practice, the same request routed to six different backends can return six different styles of reply — even when the text is semantically equivalent.

Nothing is broken. Temperature is one knob in a much larger decoding pipeline, and each vendor implements that pipeline differently. When you call many models through a single key, it helps to understand what actually varies.

What temperature actually controls

Temperature scales the logits before sampling. Near zero, it sharpens the distribution so the highest-probability token dominates. That makes output less random, but it does not make it canonical.

Several things still vary per vendor:

• Which token is top-ranked. Different model weights produce different probability distributions, so the argmax token can differ even at the same step.
• Tie-breaking. When two tokens have near-equal scores, some stacks pick the lower token ID, others use different stable orders, and some apply tiny numerical offsets.
• Batching and kernels. GPU kernels, attention implementations, and batch composition can introduce small floating-point differences that occasionally flip a near-tie.
• Default system prompts. Some endpoints inject hidden instructions or formatting conventions that shift tone before your prompt is ever read.

Sampling defaults are not neutral

Even if you pass temperature=0, other sampling parameters may have defaults you never set:

• top_p defaults vary; some vendors effectively disable it at 1.0, others apply nucleus filtering by default.
• top_k may be implicit on some backends and absent on others.
• Presence and frequency penalties often default to zero but not always, and their interaction with temperature differs.
• Stop sequences and max tokens differ in default handling; truncation can look like a tone change.

A model swap that silently flips top_p from 1.0 to a vendor default around 0.9 can noticeably shift determinism and terseness.

Decoding strategies differ beneath the API

Two vendors can both document "temperature sampling" while running different internal strategies:

• Greedy decoding at temperature 0 versus near-zero sampling with small epsilon.
• Different behavior when multiple tokens share the maximum score (some fall back to random sampling, which reintroduces drift).
• Varying handling of constrained or structured output modes.

This is why the same prompt can yield a clean bulleted answer on one backend and a chatty paragraph on another — both at temperature 0.

What to pin instead of trusting defaults

If you want a model swap to change capability without silently changing tone, set the full decoding surface explicitly:

• temperature — use your intended value, and understand that 0 means "as deterministic as this vendor allows," not "universal."
• top_p — set it to 1.0 if you want temperature to be the only sampler, or a fixed value if you rely on nucleus filtering. Do not leave it implicit.
• top_k — set explicitly if the vendor supports it; otherwise document that it is unavailable.
• max_tokens — pin it so truncation never masquerades as a style shift.
• stop — declare stop sequences rather than relying on backend defaults.
• Seed — use it where supported, and treat it as vendor-scoped, not portable across vendors.
• System prompt — keep your own system message and do not assume vendor defaults match across providers.

Build a swap harness, not a vibe check

A small evaluation harness catches drift faster than manual reading:

• Run a fixed prompt set through each backend with your pinned parameters.
• Compare outputs for length, formatting, refusal behavior, and tone markers, not just correctness.
• Store per-vendor configs in one place so a model change is a config diff, not a code rewrite.
• Re-run the harness whenever you add a backend or change a pinned parameter.

Because one key reaches multiple vendors, the harness can cover Claude, GPT, DeepSeek, Qwen, GLM, and Kimi side by side without separate accounts or billing setups.

Practical checklist

• Treat temperature as vendor-local, not global.
• Always send explicit topp, maxtokens, and stop sequences.
• Use seeds where available, but never expect cross-vendor reproducibility.
• Keep your own system prompt and formatting instructions.
• Test semantic equivalence, not string equality.

Cost and access notes

The site uses USDC on Base with no KYC, and one key reaches many models. Usage is billed at the official price × 1.3. Key contributors are credited at official price × 1.1 (premium × 1.2) in USDC. Those multipliers affect what you pay, not how a vendor decodes tokens — pinning parameters remains the way to keep tone stable across swaps.