Wiring the Aggregator Into LangChain's ChatOpenAI Wrapper: Base URL, Model Aliases and Fallbacks

Why ChatOpenAI still works

LangChain's ChatOpenAI class speaks the OpenAI-compatible chat completions protocol. The aggregator exposes that same surface, so an app already built on ChatOpenAI usually only needs a new base URL, a new key, and adjusted model strings. You are not rewriting the chain — you are changing the transport underneath it.

This is the lowest-friction migration path: no new LangChain integration package, no new message format, no change to how you build prompts or parse responses.

Repointing the client

Three values matter: the API key, the base URL, and the model name.

• API key — issue one from the aggregator dashboard. It is a single key that fronts every backend you have access to, so you can keep one secret per environment instead of one per vendor.
• Base URL — point baseurl (or openaiapi_base on older versions) at the aggregator's OpenAI-compatible endpoint. This is the field that actually redirects traffic; everything else stays the same.
• Model name — replace the vendor model string with the aggregator's alias for that model.

Keep these in environment variables rather than literals so you can flip between a direct vendor endpoint and the aggregator per environment.

What stays identical

• messages construction and role semantics
• temperature, max_tokens, and other sampling parameters
• streaming via stream=True and astream
• tool / function calling shape
• response objects, including usage metadata

The main behavioural difference is routing: your request lands on whichever backend the alias resolves to.

Model aliases explained

An alias is a stable string in your code that maps to a concrete backend model. The aggregator fronts Claude, GPT, DeepSeek, Qwen, GLM and Kimi, so a single naming scheme lets you switch families without touching the rest of the chain.

Practical implications:

• Treat aliases as configuration. Put them in a settings module or env var, not scattered through prompt files.
• Aliases are not portable across providers. A string that resolves on the aggregator will not resolve on a raw vendor endpoint. If you support both, keep the two mappings separate.
• Do not infer capability from the alias. Context window, tool support and vision support vary by underlying model. Check the model's documentation rather than assuming parity with whatever you used before.
• Pin what you depend on. If a chain depends on a specific model's behaviour, keep the alias pinned rather than tracking a moving "latest" label.

A small mapping table in your codebase — logical role to alias — keeps the rest of the app readable:

• fast → a small, cheap model for classification and routing
• default → the workhorse for most chain steps
• reasoning → a larger model for planning or hard extraction

Fallbacks across backends

LangChain's with_fallbacks wraps a runnable so that a failure on the primary triggers the next candidate. Applied to chat models, it gives you resilience without custom retry code.

With the aggregator, fallbacks get more useful because the fallback can be a different model family, not just a different deployment of the same one. If one backend is degraded or returns an error, the chain continues on another.

Design notes:

• Order by cost and capability. Put your preferred default first, then a cheaper or differently-hosted model. Do not silently fall back from a strong model to a weak one for tasks where quality matters.
• Decide what counts as a failure. with_fallbacks reacts to exceptions. Rate-limit and server errors surface as exceptions; a low-quality answer does not. If you need quality gating, add your own validation step.
• Watch for behavioural drift. A fallback model may format tool calls or JSON slightly differently. Validate structured output downstream rather than trusting shape.
• Keep the pool small. Two or three candidates is usually enough; long chains of fallbacks make failures slow and debugging hard.

Because all candidates share one key and one base URL, fallback wiring is just a list of ChatOpenAI instances with different model strings.

Streaming and callbacks

Streaming works the same way as with a direct vendor endpoint, but the callback surface is where multi-backend routing becomes visible.

• Token callbacks fire per chunk regardless of backend, so a UI built on onllmnew_token needs no change.
• Retriever and chain callbacks are unaffected — routing happens below the chain layer.
• Usage metadata is reported on the final response. It reflects the model that actually served the request, which matters when a fallback fires. Log the resolved model name alongside token counts so cost attribution stays honest.
• Streaming plus fallbacks: a failure mid-stream can surface after some tokens are already rendered. Design your UI to handle a truncated stream rather than assuming a clean end.

If you log per-call metadata, capture the model alias you requested and whatever model identifier comes back, so you can tell a primary hit from a fallback hit after the fact.

Things to verify before shipping

• Tool calling: confirm the alias you picked supports it, and test your JSON schema against it.
• Long prompts: check the context window of the underlying model, not the alias.
• Structured output: run your parser against a sample from each fallback candidate.
• Cost visibility: your spend is metered at the aggregator level; keep an eye on which aliases dominate your traffic.

Cost model, briefly

Calls bill at the official model price times 1.3. If you contribute keys to the pool, you are credited the official price times 1.1 (1.2 for premium models) in USDC on Base. Topping up is done in USDC on Base with no KYC step. Budget against the multiplier rather than assuming parity with direct vendor pricing.

Migration checklist

1. Issue an aggregator key and set the base URL via env var.
2. Replace vendor model strings with aliases, centralised in one mapping.
3. Add with_fallbacks with two or three candidates across different families.
4. Re-run your streaming UI and confirm token callbacks still fire.
5. Test tool calling and structured output against every candidate in the fallback chain.
6. Log resolved model identifiers for cost attribution.