Keeping Prompt Versions Aligned When Six Backends Behave Differently: A Diff-and-Lock Workflow

Why one prompt is not one artifact

When a single application calls Claude, GPT, DeepSeek, Qwen, GLM, and Kimi through one API key, the prompt text is shared but the behaviour is not. Tokenizers differ, instruction-following strength differs, and formats like JSON output or tool-call schemas are honoured with different strictness.

The usual failure is silent drift: someone tunes the prompt against one backend, ships it, and the other five quietly degrade. The diff-and-lock workflow treats a prompt as a versioned artifact with per-backend expectations, rather than a string that happens to work somewhere.

Step 1: Freeze a baseline per backend

Before editing anything, capture what the current prompt actually produces on every backend you route to.

• Fixed probe set. Build a small set of representative inputs covering the main paths: typical, edge cases, and malformed user input. Keep the list stable so results are comparable across versions.
• Deterministic decoding. Use the lowest practical temperature and a fixed seed where supported. You are testing prompt behaviour, not sampling luck.
• Store raw outputs. Save the full response, not just parsed fields. Formatting differences and truncation are exactly the kinds of drift you need to see.
• Record metadata. Backend name, model identifier, parameter set, and date. A baseline without its parameters is not reproducible.

Step 2: Diff in two layers

Most teams diff only the prompt text. That catches edits but not behavioural change. Diff both layers.

Text diff

• Keep prompts in version control as plain files, one canonical source per logical prompt.
• Use a standard line diff to review edits before they ship.
• Isolate backend-specific fragments (formatting instructions, role phrasing) into clearly marked sections so a text diff shows whether a change is global or targeted at one backend.

Behavioural diff

• Run the probe set against each backend and compare against the stored baseline.
• Classify differences rather than merely listing them: identical, cosmetically different (whitespace, wrapper text), structurally different (parsed shape changed), or semantically different (content or decision changed).
• Treat structural differences as hard failures when your code parses the output. A backend that adds prose around a JSON object is a breaking change downstream.
• Treat semantic differences as review items, not automatic failures — a different but valid answer is a judgement call, not a bug to be patched blindly.

Step 3: Lock variants deliberately

A lock is a declared pairing: this prompt version, this backend, these parameters, this probe set, this expected behaviour. Locks prevent accidental change and make intentional change auditable.

• One lock per backend. Do not let a prompt edit be implicitly live on all backends. Promote changes lock by lock.
• Pin parameters with the lock. Temperature, max tokens, and response-format settings belong to the lock, not to global defaults that may drift.
• Record the expectation, not just the output. Write down what the correct behaviour is for each probe, so future diffs can be judged against a stated criterion.
• Allow per-backend prompt overlays. If one backend needs a different formatting nudge, express it as an overlay on the shared core. Diffing then shows exactly which backend got special handling.

Step 4: Gate promotion on the diff

A change moves through stages: edit, run probes, review diffs, promote lock, archive old lock.

• Block promotion on structural regressions. Any parse failure in any locked backend stops the release.
• Require human review for semantic changes. Route the behavioural diff to the prompt owner; the diff, not the chat transcript, is the review artifact.
• Promote backend by backend. Rolling out to the backend you changed first, then the rest, isolates the cause if something breaks.
• Archive, do not delete. Old locks are your rollback path and your record of what was verified when.

Step 5: Automate the loop, keep the judgement

The mechanical parts — running probes, storing outputs, producing text and behavioural diffs, enforcing structural checks — should run on every prompt change, ideally in CI. The judgement parts — whether a semantic difference is acceptable, whether an overlay is justified — stay with a human reviewer.

Practical notes for a multi-backend aggregator

• One key, many backends. Because all six models are called through the same aggregator, the probe runner needs only a single credential and one request shape; backend selection is just a parameter.
• Cost-aware probe runs. You top up with USDC on Base and are billed at the official price times 1.3 per call. Probe runs multiply calls across backends, so keep the probe set small and stable rather than large and regenerated. Contributors whose keys are used are credited the official price times 1.1 (premium times 1.2) in USDC — relevant if your probe infrastructure shares keys across a team, since usage accounting follows the key.
• No KYC overhead. Top-ups via USDC on Base mean the workflow can be run in CI without procurement friction, but budget for probe runs as a recurring line item, not a one-off.
• Version everything together. Prompt version, probe set version, parameter set, and lock identifier should travel together in your config so a diff is always attributable to a single change.

The short version

Capture a baseline per backend, diff both text and behaviour, lock each backend's prompt-parameter-probe combination, gate promotion on structural diffs and human review of semantic ones, and archive old locks for rollback. The prompt stays one source of truth; the locks keep six backends honest without pretending they behave the same.