Pointing an OpenAI SDK at the Aggregator: Which Client Defaults Silently Break First

Why Client Defaults Matter

The official OpenAI SDK comes with sensible defaults for OpenAI's own API. When you point it at a multi-model aggregator, those defaults may not align with how the aggregator routes requests to different providers. The result: silent failures, unexpected behavior, or degraded performance.

This guide covers the client-side defaults that are most likely to cause issues and how to adjust them.

max_tokens: The Silent Truncator

The OpenAI SDK often defaults maxtokens to a value like 16 or leaves it unset. For some models, an unset maxtokens means "use the model's maximum," but for others it may default to a small number.

• Problem: If you rely on the SDK default, some models may truncate responses early. This is especially common when switching between models with different default behaviors.
• Solution: Always set max_tokens explicitly to a value that makes sense for your use case. The aggregator passes this value through to the underlying provider, so being explicit ensures consistent behavior.

Retries: When Automatic Retries Hurt

The OpenAI SDK automatically retries failed requests (e.g., on 429 or 5xx errors) with exponential backoff. While useful for OpenAI, this can be problematic with an aggregator that may route to providers with different rate limits or error semantics.

• Problem: Aggregator-level rate limits or provider-specific errors might trigger retries that compound load or cause unexpected delays.
• Solution: Consider reducing the number of retries or disabling them for non-idempotent operations. Also, ensure your error handling accounts for aggregator-specific error codes.

response_format: Not Universally Supported

The responseformat parameter (e.g., { "type": "jsonobject" }) is a popular OpenAI feature. However, not all models support it. The aggregator may pass it through to providers that ignore it or return an error.

• Problem: If you rely on response_format to enforce JSON output, some models may return plain text, breaking your parsing logic.
• Solution: Either avoid response_format for models that don't support it, or implement fallback parsing. Always validate the response structure.

Timeout: Defaults May Be Too Short

The OpenAI SDK sets a default timeout (often 600 seconds for large models, but some SDK versions use shorter defaults). Aggregators add a hop between you and the provider, and some models (especially larger ones) may take longer to respond.

• Problem: A short timeout can cause premature request cancellation, especially for streaming or long-running completions.
• Solution: Increase the timeout to accommodate the slowest model you plan to use. Monitor response times and adjust accordingly.

Stream Chunk Parsing: Assumptions About Format

When streaming, the OpenAI SDK expects chunks in a specific format (e.g., data: {...} with a choices array). The aggregator normalizes responses to match OpenAI's format, but subtle differences can arise.

• Problem: If the aggregator returns chunks with slight variations (e.g., missing fields or different ordering), the SDK's parser may throw errors or misinterpret data.
• Solution: Test streaming with each model you use. If issues occur, consider using raw HTTP streaming and parsing manually, or check if the aggregator provides SDK-specific guidance.

Other Defaults to Watch

• base_url: Ensure you've set the base URL to the aggregator's endpoint, not OpenAI's.
• api_key: Use your aggregator key, not an OpenAI key.
• model names: Use the exact model identifiers supported by the aggregator.

Best Practices

• Explicit configuration: Always set maxtokens, timeout, and responseformat explicitly rather than relying on defaults.
• Error handling: Implement robust error handling that accounts for aggregator-level errors and provider-specific quirks.
• Testing: Test each model you plan to use with a small request to verify behavior.
• Documentation: Refer to the aggregator's documentation for model-specific notes and limitations.

By understanding and adjusting these client defaults, you can avoid silent failures and ensure reliable operation across multiple models.