Same JSON Schema, Six Backends: Tracking Where Field Types and Nullability Drift

Why Schema Drift Matters Across Models

When you use the same JSON Schema to constrain outputs from different LLM backends, you might assume that a compliant model will always produce a predictable structure. In practice, models vary in how they interpret schema constraints, especially around field types and nullability. This drift can break downstream processing if not tracked.

Common causes include:

• Differences in how models handle optional fields (whether they omit the field, set it to null, or provide a default).
• Variations in type coercion (e.g., a number might be returned as a string when the schema allows both).
• Inconsistent enforcement of nullable or required semantics.

A Practical Approach to Tracking Drift

To manage drift, you need a repeatable process for generating outputs from each backend and comparing them against the schema. Here’s a conceptual workflow:

1. Define a representative schema – Include fields with mixed types (string, number, boolean, object, array), optional properties, and explicit nullability where relevant.
2. Prepare a set of test prompts – Use prompts that exercise the schema’s edge cases: fields that could be omitted, fields that might be null, and fields with ambiguous type constraints.
3. Call each backend with the same input – Send identical prompts and the schema to each model you want to compare.
4. Parse and validate outputs – For each response, check JSON validity and validate against the schema using a standard validator.
5. Record drift indicators – For each field, note:

• Presence/absence (was the field included?)
• Actual type (does it match the declared type?)
• Null values (was null returned when not nullable?)
• Type coercion (e.g., string vs. number)

1. Aggregate results – Build a matrix showing which fields drift per backend, and whether drift is consistent or sporadic.

Using a Unified API to Simplify Testing

Manually calling each backend with separate credentials and endpoints adds friction. An API aggregator that provides a single key for multiple models lets you focus on the comparison rather than integration details.

With such a service, you can:

• Use one API key to access Claude, GPT, DeepSeek, Qwen, GLM, and Kimi.
• Send the same request payload (including the JSON Schema) to each model by changing only the model identifier.
• Collect responses in a consistent format for analysis.

The service charges official price × 1.3 for usage, and contributors of keys earn credits at official price × 1.1 (or × 1.2 for premium keys) in USDC. Top-up is via USDC on Base without KYC.

Handling Nullability and Optional Fields

Nullability is a frequent source of drift. Some models may treat a field as nullable if it’s not in the required array, even if nullable: false is set. Others may strictly enforce nullable.

To minimize issues:

• Be explicit: use required to list mandatory fields, and set nullable explicitly for fields that can be null.
• Avoid ambiguous schemas where a field could be either omitted or null; choose one convention.
• In your tracking, distinguish between “field missing” and “field null” as separate drift types.

Automating Drift Detection

You can script the comparison process:

• Loop over model identifiers supported by your aggregator.
• For each, send the same prompt and schema.
• Validate the response against the schema and log any validation errors along with the raw output.
• Store results in a structured format (e.g., JSON lines) for later analysis.

Over time, you’ll build a profile of how each backend behaves with your schemas, allowing you to choose models that best match your reliability requirements or to adjust schemas to be more robust.

Best Practices for Cross-Model Schema Design

• Keep schemas simple – Complex schemas with deep nesting increase the chance of misinterpretation.
• Use clear type definitions – Avoid oneOf or anyOf when a single type suffices.
• Document expected behavior – Note which fields are optional and which can be null.
• Test with real prompts – Use prompts that resemble actual use cases, not just synthetic edge cases.
• Monitor regularly – Model updates can change behavior; periodic re-testing helps catch regressions.

By systematically tracking field types and nullability across backends, you can build more resilient applications that leverage multiple models without sacrificing data integrity.