Bug, Docs Gap, or Wrong Expectation? Classifying a Strange Aggregator Response Before You Report It

Why classification matters

When a response looks wrong, the fastest fix depends entirely on what kind of problem it is. A malformed request sent as a bug report wastes a round trip; a real endpoint defect filed as a support question may sit unresolved. A short triage pass saves time for you and for whoever maintains the service.

This walkthrough assumes one pattern: an OpenAI-compatible aggregator where you hold a single API key, pay the official price multiplied by 1.3 in USDC on Base, and contributors are credited official price × 1.1 (or × 1.2 on premium models). None of that billing detail changes the triage steps, but it helps you know what evidence the maintainers will want.

Step 1: Reproduce it deterministically

Before classifying anything, make the failure repeatable.

• Send the same request body at least twice. Note whether the strange output is identical, similar, or different each time.
• Set a low temperature (or the equivalent parameter for the model) and a fixed seed if the provider supports it. If the oddity disappears at low temperature, you are probably looking at sampling behavior, not an endpoint defect.
• Capture the raw HTTP status, headers, and response body, not a reformatted client object. Many client libraries swallow or reshape error fields.
• Record the exact model identifier you sent, the endpoint path, and the timestamp.

If you cannot reproduce it with a fixed request, treat it as model behavior until proven otherwise. Generative models vary between calls; a single strange answer is not a defect.

Step 2: Isolate the request format

A large share of "wrong responses" are prompt-format mistakes. Check these before anything else:

• Message roles and ordering. A system prompt placed after user turns, or a stray assistant message, can change output in ways that look like malfunction.
• Parameter names. Aggregators pass many parameters through to the upstream provider. A misspelled or non-standard field may be silently ignored rather than rejected, so the model runs with defaults you did not intend.
• Type mismatches. A number sent as a string, or a boolean sent as text, may be coerced or dropped depending on the gateway's validation strictness.
• Token and length settings. If the response is truncated or oddly cut off, inspect the finish reason and your max-token setting before assuming a truncation bug.
• Tool and structured-output schemas. Invalid JSON schemas often produce empty or partial structured output, which reads like a broken endpoint but is a schema problem.

If correcting the request makes the oddity disappear, you had a prompt-format mistake. No report is needed, but it may still be worth noting in the docs (see Step 5).

Step 3: Separate model behavior from routing behavior

Aggregators sit in front of multiple providers. It helps to know whether the oddity follows the model or the route.

• Send the same prompt to a different model through the same key. If only one model misbehaves, it is likely model-specific behavior or an upstream provider quirk.
• Send the same prompt to the same model twice. If one call succeeds and the next fails with a transport error, that points to routing, retries, or upstream availability.
• Check whether the oddity appears in streaming mode only, or in non-streaming mode only. Streaming differences (chunk boundaries, partial deltas, final usage fields) are a common source of confusion.
• Look at the response metadata. A provider or model field that differs from what you requested can explain an unexpected style or capability.

A model that answers in an unexpected tone or format, while returning a normal status and well-formed body, is behavior. A response that is malformed, missing required fields, or inconsistent with the documented schema is a candidate defect.

Step 4: Compare against the documented contract

The decisive test is the API contract, not your intuition about what the model "should" say.

• Does the response conform to the documented response schema (required fields present, correct types)?
• Does the status code match the documented meaning for that condition?
• Are documented parameters accepted and reflected where the docs say they will be?
• Does an error case return the documented error shape, or something ad hoc?

If the behavior contradicts a clear, specific statement in the docs, you likely have a defect or a docs error. If the docs are silent, vague, or ambiguous about the case you hit, you have a docs gap. That distinction determines where the report goes.

Step 5: Route the report

Match your classification to the right channel and include the right evidence.

• Endpoint defect (contract violation). Include the minimal request body, the raw response, the status code, the model identifier, timestamps, and the specific documented statement it violates. Note whether it reproduces. This goes to the issue tracker or support channel for bugs.
• Prompt-format mistake. No bug report. If the mistake is easy to make, suggest a docs clarification instead.
• Docs gap or ambiguity. File it as a documentation request. Point to the exact section, state what you expected to find, and describe the request that left you unsure. Concrete examples beat abstract complaints.
• Model behavior difference. Usually not actionable as a defect. If the docs promise a behavior the model does not deliver, that is a docs gap, not an endpoint bug.
• Upstream provider issue. If the failure tracks a single upstream provider and the aggregator is passing through its error faithfully, report it, but frame it as an upstream problem so it is triaged correctly.

Step 6: Write a report that classifies itself

A good report states its own category up front. Include:

• One line naming the category: defect, docs gap, or expectation question.
• The minimal reproducible request and the observed versus expected response.
• The exact model identifier, endpoint, and time.
• What you already ruled out: low temperature tried, alternate model tried, streaming versus non-streaming tested, request validated against the schema.

This lets maintainers skip the first round of questions and go straight to the fix.

A note on billing-adjacent confusion

Sometimes an odd response is accompanied by a question about charges. Keep the two separate. Usage and credit accounting (users pay official price × 1.3; contributors are credited × 1.1, or × 1.2 for premium models) is a billing question. A malformed response body is an API question. Mixing them in one report makes both harder to resolve, so file them separately with their own evidence.

Quick checklist

• Reproduced with a fixed request? If not, suspect model variance.
• Request validated against the schema and parameter names? If not, fix first.
• Oddity follows the model or the route? Test both ways.
• Response violates a specific documented statement? Defect.
• Docs silent or ambiguous? Docs gap.
• Report includes category, minimal reproduction, and what you ruled out? Ready to send.