Using Logprobs From One Aggregator Key to Compare Backend Confidence

Why logprobs help with model selection

Benchmarks and vibes tell you whether a model can answer. Logprobs tell you how sure it was. For classification-style work — routing a support ticket, tagging a document, picking a label from a fixed set — the shape of the model's probability distribution over candidate labels is often more informative than the text it emits.

When you call several backends through one aggregator key, you get a consistent billing and auth surface, but the backends still differ in tokenizer, prompt conventions, and how they expose logprobs. This article covers how to collect comparable signals and reason about them.

What logprobs actually are

• A logprob is the natural logarithm of the probability the model assigned to a token at generation time. A logprob of 0 means probability 1; more negative values mean lower confidence.
• Most APIs return per-token logprobs for the generated tokens, and some return the top-k alternatives at each position.
• Confidence for a whole answer is not a single number. It is a curve — one value per generated token — plus, ideally, the alternatives the model considered.

Keep in mind: logprobs are model-internal and not calibrated probabilities over real-world events. A model can be confidently wrong. Treat them as a comparative signal, not ground truth.

Step 1: Normalize the request

To compare backends fairly, hold everything except the model constant:

• Use the same prompt text, the same label set, and the same expected output format (for example, a single label word).
• Set temperature to a deterministic value if the API allows it. Sampling temperature changes the distribution you observe.
• Request logprobs and a small top-k of alternatives where supported.
• Keep the completion short. One token per label makes alignment far easier.

Step 2: Handle tokenizer differences

This is the main trap. The same word may be one token on one backend and several on another, and leading spaces or punctuation can change tokenization.

• Compare at the label level, not the token level. Map tokens back to the label string and aggregate.
• If a label spans multiple tokens, multiply probabilities (add logprobs) across the tokens that make up the label.
• Where a backend returns only the chosen token's logprob, you still have a usable per-answer confidence, just not a full label distribution.
• Record the raw tokens alongside your derived label score so you can audit oddities later.

Step 3: Build the comparison harness

A small script is enough:

• Loop over the models you want to compare, calling the same endpoint with the same payload via your aggregator key.
• For each response, capture: the label the model emitted, the logprob(s) for that label, and any top-k alternatives that map onto your label set.
• Convert to a comparable score — for example, the logprob of the chosen label, or a normalized probability across your fixed label set when alternatives are available.
• Store one row per (example, model) so the analysis is a simple groupby.

Because billing runs through a single key, you can attribute spend per model in the same harness. Note that usage is charged at official price × 1.3, and that contributing keys are credited back at official price × 1.1 (premium models × 1.2) in USDC — useful if you are running large comparison batches.

Step 4: Read the confidence curves

Plot or summarize the per-token logprobs and look at these patterns:

• Flat and high: consistent confidence across the answer. Often a sign the task is easy for the model.
• Sharp drop at one token: the model hesitated at a specific decision point — frequently the label token itself. Worth inspecting.
• Early collapse then recovery: the model committed to a label and then rationalized. Confidence in the label token matters more than confidence in filler.
• Consistently low across the label: the model is unsure between labels. This is exactly where your routing logic should be careful.

For classification, the single most useful number is usually the logprob of the label token (or aggregated label span), not the average across the whole completion.

Step 5: Turn curves into a decision

• Rank models by agreement with your labeled examples, then look at confidence on the disagreements. A model that is confidently wrong is a worse fit than one that is unsure and deferrable.
• Use a confidence threshold to escalate low-confidence calls to a stronger model or to a human. Compare how often each backend falls below the threshold.
• Check calibration informally: bucket examples by confidence and see whether higher-confidence buckets are actually more accurate. Backends differ here.
• Prefer the cheapest backend that clears your accuracy and calibration bar, not the one with the highest average confidence.

Caveats

• Not every backend exposes logprobs, and support for top-k alternatives varies. Design your harness to degrade gracefully.
• Logprobs can be affected by system prompts, formatting instructions, and output constraints. Change the prompt and the curve changes.
• Do not interpret logprobs as guarantees. They describe the model's internal distribution, not the correctness of the answer.
• Re-run comparisons when you change models or prompts; confidence profiles drift.

Putting it together

One aggregator key, one harness, many backends: request logprobs, align on labels rather than tokens, and compare confidence curves on the same examples. The output is a defensible, evidence-based choice of which model to trust for your classification task — and a confidence threshold that lets you route the uncertain cases somewhere safer.