Dashboard Says One Thing, Your Script Says Another: Reconciling Token Counts Across Views

Why Token Counts Diverge

You check your aggregator's usage page and see a token count that doesn't match the count your script logged. This is a common experience, and understanding why it happens is the first step to reconciling the numbers.

Differences arise from several factors:

• Tokenization differences: Different models tokenize text in different ways. A local tokenizer may not match the exact tokenizer used by the model you're calling.
• What gets counted: Some APIs count only input and output tokens, while others include additional tokens for system prompts, function calls, or formatting.
• Timing and aggregation: The dashboard may aggregate usage over a different time window or with a delay compared to your local logging.
• Retries and errors: If your script retries failed requests, the API may count tokens from attempts that your local tally ignores.
• Streaming vs. non-streaming: Counting tokens from a streamed response can be trickier and may lead to undercounting if not done carefully.

The Aggregator's Perspective

The aggregator's usage page is the authoritative record for billing. It tracks every request that reaches the API, including those that may not have completed successfully from your script's point of view.

Key points:

• The dashboard reflects the provider's token counting, which is based on the model's tokenizer.
• It includes all billable tokens, such as those from system prompts and function definitions.
• It may apply rounding or aggregation that your local count does not.

Building a Weekly Reconciliation Checklist

To catch discrepancies early, set aside time each week to compare your local logs with the aggregator's dashboard. Follow these steps:

1. Export your local usage data: Gather the token counts your script recorded for the week. Include timestamps, model names, and input/output token counts.
2. Export the aggregator's usage data: Download or copy the usage report from the dashboard for the same period.
3. Align time windows: Ensure both datasets cover the same start and end times. Adjust for any timezone differences.
4. Compare totals per model: Break down the comparison by model, as tokenization and pricing may vary.
5. Investigate large gaps: If a model shows a significant difference, check for retries, errors, or streaming issues in your logs.
6. Check for untracked requests: Look for any API calls that your script may not have logged, such as those from other team members or automated jobs.
7. Document findings: Note the discrepancies and their likely causes. This helps spot trends over time.
8. Adjust your logging if needed: If you find consistent undercounting, refine your local token counting logic.

Tools and Techniques

• Use the model's official tokenizer library for local counting to minimize tokenization differences.
• Log the raw usage data returned by the API, if available, as it often includes exact token counts.
• Implement retry logic that logs each attempt separately to avoid undercounting.
• For streaming responses, consider using a tokenizer on the full response after it's assembled.

When to Contact Support

If, after thorough reconciliation, you find unexplained discrepancies that affect your billing, reach out to the aggregator's support. Provide your local logs and the specific time ranges in question. Most issues can be resolved by identifying differences in tokenization or unlogged requests.

Conclusion

Regular reconciliation between your local token counts and the aggregator's dashboard is a good practice. It ensures you understand your usage and billing, and it helps you catch issues early. By following a weekly checklist, you can keep your token accounting accurate and avoid surprises.