Forcing JSON Output on Six Backends Behind One Key: Who Honors response_format and Who Silently Ignores It

Why JSON mode matters

When building applications that consume LLM outputs programmatically, strict JSON is often required. Many APIs expose a responseformat parameter (e.g., {"type": "jsonobject"}) to encourage structured output. However, support varies across backends. This guide explains how to probe each backend behind a single API key, interpret results, and design fallbacks.

The probing approach

Use the same prompt with a simple schema across all backends. For example, ask for a JSON object with fields name (string) and age (number). Include the schema in the prompt and set responseformat to {"type": "jsonobject"} if the backend accepts it.

• Send identical requests to each model.
• Record whether the response is valid JSON, contains extra prose, or fails to parse.
• Repeat a few times to account for variability.

Expected behaviors

Backends generally fall into three categories:

• Strict JSON: Returns only the JSON object, parseable with JSON.parse.
• Wrapped JSON: Returns JSON embedded in Markdown code blocks or surrounded by explanatory text.
• Inconsistent: Sometimes returns valid JSON, sometimes not, even with response_format.

Handling wrapped or inconsistent output

If a backend wraps JSON in prose, you can extract it using regex or a JSON parser that tolerates leading/trailing text. For example, find the first { and last } and parse the substring.

When response_format is ignored, fall back to prompt-level coercion:

• Explicitly instruct: "Respond ONLY with a valid JSON object. Do not include any other text."
• Provide a schema and an example.
• Use few-shot examples if needed.
• Consider adding a system message that reinforces the JSON-only requirement.

Implementing a robust client

Design your client to handle all cases:

1. Attempt to parse the raw response as JSON.
2. If that fails, try extracting JSON from Markdown code blocks.
3. If that fails, try locating the first { and last } and parsing the substring.
4. If still failing, consider a repair step (e.g., ask the model to fix its output) or log the failure for manual review.

Always validate the parsed JSON against your schema using a library like Ajv or Zod.

Conclusion

Not all backends honor response_format equally. By probing each model and implementing layered fallbacks, you can build resilient applications that work across multiple providers with a single API key. Remember that prompt-level coercion and client-side parsing are essential when strict JSON mode is unavailable.