Probing /chat/completions Schema Conformance: A Field-by-Field Response Diff Across Six Backends

Why a response diff matters

A single API key that reaches multiple model families is convenient, but each backend serializes its response independently. Two responses can both be valid OpenAI-style JSON and still disagree on which fields appear, what type they hold, or whether they exist at all. If your parser assumes system_fingerprint is always present, it will eventually break on the backend that omits it.

The cheap way to find these edges is a controlled diff: send the same minimal request to each backend, capture the raw JSON, and compare field by field.

Build the baseline request

Keep the probe as small as possible so the diff reflects schema, not content.

• One system message and one short user message.
• A small max_tokens so responses are fast and cheap.
• temperature at 0 for stable output.
• stream: false — streaming arrives as SSE chunks with a different shape and is a separate exercise.
• No tools, no response format, no stop sequences. Add those later as separate probes, because they can change which fields the backend returns.

Record the exact request body. Every comparison is only meaningful relative to that input.

Capture raw responses, not parsed objects

Save the HTTP status plus the raw response text before any deserialization. Parsing first hides the very differences you are hunting: a client library may drop unknown fields, coerce types, or fill defaults.

Repeat each call a few times. Some fields are stable across calls, some vary by request, and some appear only under certain conditions. Noting which is which belongs in the diff, not in your head.

Diff top-level fields

Walk the top-level object once per backend and note presence and type.

• id — almost always present. Compare prefixes and shape rather than value; treat it as an opaque string and never parse meaning out of it.
• object — expected to read chat.completion, but confirm it rather than assume.
• created — a Unix timestamp integer. Check it is a number and not a string.
• model — check whether the backend echoes the model name you sent or returns a canonical or versioned name. If your logs join on this field, that matters.
• system_fingerprint — frequently present, but not guaranteed. Record which backends include it, whether it is a string, and whether it stays constant across identical requests. Design your parser to treat it as optional.
• usage — present on non-streamed completions in the surveyed backends, but verify.

Diff choices and finish_reason

choices is an array even for a single completion. Check:

• index — starts at 0 and is an integer.
• message.role — should be assistant.
• message.content — a string in the normal case. Note whether any backend returns extra sibling fields on the message object, such as reasoning-related keys. Do not depend on them unless you have confirmed them per backend.
• finish_reason — compare the actual values returned. stop is the common case for a short answer. Others such as length-based or content-filter values appear only under specific conditions, so a single probe will not enumerate the full set. Capture what you see and leave the enum open in your code.

Treat finish_reason as an open set. A parser that switches only on stop will misclassify anything else.

Diff the usage sub-object

This is where cross-backend variance is most common and most consequential for billing dashboards.

• prompttokens and completiontokens — usually integers; confirm they are numbers, not strings.
• total_tokens — check whether it is present and whether it equals the sum of the other two on every backend.
• Additional sub-fields — some backends include extra keys such as cached-token or reasoning-token breakdowns. Record their presence and type, and make them optional in your model.

If you compute cost from usage, define a fallback: derive totals from the two primary counters when the total is missing, and never assume extra breakdown fields exist.

Turn the diff into parser rules

Summarize the probe into three lists per field: always present, sometimes present, never present. Then encode them:

• Required fields (for example id, choices, message.content) may be accessed directly.
• Optional fields (system_fingerprint, extra usage keys) should be nullable and guarded.
• Enums like finish_reason should be handled with a default branch.
• Numbers should be validated as numbers; do not rely on implicit coercion.

Re-run the probe when backends change

Backends evolve. Re-run the same minimal request on a schedule or after any model or routing change, and keep the raw captures as fixtures for regression tests. Treat the diff as a living artifact, not a one-time investigation.

Practical notes

• Log id and model per call so you can trace which backend served a request.
• Keep probes cheap: one short message at temperature 0.
• Test streaming separately; chunk shapes differ from the non-streamed object.
• Billing on this platform is official price × 1.3 for users, with key contributors credited official price × 1.1 (premium × 1.2) in USDC on Base — so accurate usage parsing is worth the effort.