Reading Rate Limit Headers on an OpenAI-Compatible Aggregator

Why Rate-Limit Headers Matter

When you use an OpenAI-compatible aggregator, your requests pass through a proxy layer before reaching the actual model provider. This means rate-limit headers may come from two sources: the aggregator itself and the upstream vendor. Understanding which is which helps you diagnose throttling, adjust your client behavior, and plan capacity.

Common Rate-Limit Headers

OpenAI-compatible APIs typically return headers like:

• x-ratelimit-limit-requests: Maximum requests allowed in the current window.
• x-ratelimit-remaining-requests: Requests left in the current window.
• x-ratelimit-reset-requests: Time until the request limit resets.
• x-ratelimit-limit-tokens: Maximum tokens allowed in the current window.
• x-ratelimit-remaining-tokens: Tokens left in the current window.
• x-ratelimit-reset-tokens: Time until the token limit resets.

Some aggregators also include custom headers like retry-after for backoff hints.

Aggregator vs. Upstream Vendor Limits

The aggregator may enforce its own rate limits to manage traffic and protect upstream vendors. These limits are often based on your account tier or usage patterns. Upstream vendor limits are imposed by the model provider (e.g., OpenAI, Anthropic) and may be stricter or more generous.

How to tell them apart:

• Header naming: Aggregator-specific headers might have a prefix or differ slightly.
• Documentation: Check the aggregator's docs for their limit policies.
• Behavior: If you hit a limit on one model but not another, it's likely an upstream vendor limit.
• Response codes: 429 status codes indicate rate limiting; the error message may specify the source.

Logging Rate-Limit Headers

Logging these headers helps you monitor usage and debug issues. Here's a practical approach:

• Capture all x-ratelimit-* headers from each response.
• Include the model name, timestamp, and request ID for correlation.
• Store logs in a structured format (e.g., JSON) for easy analysis.
• Set up alerts when remaining requests or tokens drop below a threshold.

Example pseudo-code:

import requests

response = requests.post(url, headers=headers, json=payload)
rate_limit_info = {
    'model': model,
    'timestamp': time.time(),
    'limit_requests': response.headers.get('x-ratelimit-limit-requests'),
    'remaining_requests': response.headers.get('x-ratelimit-remaining-requests'),
    'reset_requests': response.headers.get('x-ratelimit-reset-requests'),
    'limit_tokens': response.headers.get('x-ratelimit-limit-tokens'),
    'remaining_tokens': response.headers.get('x-ratelimit-remaining-tokens'),
    'reset_tokens': response.headers.get('x-ratelimit-reset-tokens'),
}
log(rate_limit_info)

Best Practices for Handling Rate Limits

• Respect the retry-after header if present.
• Implement exponential backoff with jitter.
• Cache responses where possible to reduce API calls.
• Distribute requests across multiple models if allowed.
• Monitor your usage patterns to predict when you'll hit limits.

Conclusion

Rate-limit headers are a window into how the aggregator and upstream vendors manage traffic. By logging and interpreting them correctly, you can build resilient applications that handle throttling gracefully.