Carrying One System Prompt Across Six Backends on One Key: Where It Gets Weakened or Ignored

Why one system prompt behaves differently on every backend

A system prompt is not a protocol-level guarantee. It is just another block of text with a role label. Each backend tokenizes it, weights it, and prioritizes it against the user turn according to its own training, context handling, and chat template. One API key can route to Claude, GPT, DeepSeek, Qwen, GLM, and Kimi, but the key does not normalize how those models treat instructions.

When you call many models through a single aggregator, the request shape is usually shared, but the backend is not. Two things follow:

• The same text can be obeyed strictly by one model and treated as loose background by another.
• A backend can appear to work because it returns fluent output, while quietly ignoring the constraints you set.

This guide is about finding the second case before your users do.

The three failure modes to look for

When a system prompt is "weakened," it is rarely a hard failure. It usually shows up as one of these:

• Softening: the model follows the spirit but drops hard constraints (format rules, refusal boundaries, length limits, persona bans).
• Drift: the instruction holds for the first turn or two, then decays as the conversation grows.
• Silent substitution: the model applies its own default behavior (e.g., adds caveats, uses bullet lists, switches tone) even though your prompt forbids it.

Each of these is detectable with the same technique: give the model a prompt where compliance is binary and observable, then compare across backends.

Design a probe prompt with binary compliance

Opinion questions and "write something helpful" prompts are useless for testing adherence because almost any output looks acceptable. Build probes where a single rule is either met or broken, and where the violation is machine-checkable.

Useful probe categories:

• Format constraints: "Reply with exactly one JSON object and nothing else" or "Use lowercase only." A violation is trivially parseable.
• Negative constraints: "Never mention pricing" or "Do not use the words 'certainly' or 'great question'." Check with a substring search.
• Boundary instructions: "If the request is outside topic X, reply with the single word REFUSE." Check for the exact token.
• Role bans: "Do not adopt a persona; do not use first-person opinion." Check with simple patterns.
• Length rules: "Answer in at most N words." Count words.

Keep each probe focused on one rule. A prompt with ten constraints cannot tell you which one a backend dropped.

Run the same probe across every backend

For each model reachable through your key:

1. Send the identical system prompt.
2. Send the identical user turn.
3. Record the raw output, not a cleaned version.
4. Repeat several times, because sampling and temperature make single runs noisy.

Do not tune the prompt per model during this phase. The point is to measure the prompt as written against each backend, not to find a version that flatters one model.

A few practical notes:

• Set temperature low for the first pass so you are measuring adherence, not creativity.
• Keep the user turn short and unambiguous so the probe isolates the system prompt.
• Log the model identifier returned by the API, not the one you requested, where available. Routing and aliases can surprise you.

Score compliance, don't eyeball it

Write a tiny checker per probe. It should return pass/fail plus the reason.

• JSON probes: attempt to parse; fail on extra prose.
• Forbidden-word probes: case-insensitive substring scan.
• Refusal probes: exact match on the sentinel token.
• Length probes: word count against the threshold.

Aggregate by backend over multiple runs. What you want is a per-backend adherence rate for each rule, so you can see which model is weakest on which constraint class. A model that is perfect on format but loose on refusals is a different problem from one that ignores format entirely.

Test drift, not just the first turn

Many prompts survive turn one and fail by turn five. Extend each probe into a short multi-turn script:

• Turn 1: a benign question.
• Turn 2: a question that tempts the forbidden behavior.
• Turn 3: a request that pushes against the boundary.

Re-score each turn against the same rule. If adherence drops as the conversation grows, the backend is deprioritizing the system prompt under context pressure. That is a real production risk even when the first response looked clean.

Distinguish weakening from ignoring

Not every miss means the same thing, and the fix differs:

• Ignored: the rule is violated immediately and consistently. The model likely does not weight system-role instructions strongly, or your rule conflicts with a strong default.
• Weakened: the rule holds in easy cases and breaks under pressure. The model is treating the instruction as a preference, not a constraint.
• Misread: the output is compliant but not in the way you intended, often a formatting or tokenization quirk. Check the raw string before blaming the model.

Recording the raw output is what lets you tell these apart later.

Make the prompt harder to drop

Once you know which backends are weak on which rules, strengthen the prompt in ways that travel well across models:

• Put hard constraints in short, imperative sentences near the end of the system prompt.
• Repeat the single most important constraint in the user turn as a reminder, without duplicating the whole prompt.
• Prefer explicit, checkable rules over adjectives. "Reply in one sentence" beats "be concise."
• Avoid stacking many soft preferences; they compete and get diluted.
• Use a sentinel token for refusal so compliance is unambiguous.

These changes do not guarantee parity across backends, but they reduce the gap and make failures easier to detect.

Build a regression check into your workflow

Treat the probe suite as a regression test for prompt changes and for backend changes.

• Re-run the suite whenever you edit the system prompt.
• Re-run it when you add a model to your routing.
• Keep the results per backend so you can spot a model that suddenly starts drifting.

Because you are calling multiple models behind one key, you can run the whole matrix cheaply during development. Keep probe outputs short so the evaluation itself stays inexpensive.

Cost and access notes

When you route probes through an aggregator, billing is straightforward: you pay the official model price multiplied by 1.3, settled in USDC on Base with no KYC step. If you contribute keys, credit is issued at official price times 1.1 (times 1.2 for premium access) in USDC.

That structure matters for evaluation because running the same suite across six backends is a real but bounded cost. Keep probes short, cache the results, and re-run only what changed. The goal is not exhaustive benchmarking; it is catching the backend that quietly drops your instructions before it reaches production.

A minimal checklist

• One rule per probe, with a machine-checkable pass condition.
• Identical prompt and user turn across every backend.
• Multiple runs at low temperature.
• Raw output logged, plus the model identifier the API reports.
• A multi-turn variant to catch drift.
• Per-backend adherence scores, reviewed on every prompt or routing change.