Storing Multi-Turn Conversation State Outside the API on One Key: Trimming and Rehydrating History Per Backend

Why keep history outside the API

Many chat APIs offer server-side threads or conversation objects that hold state for you. They are convenient, but they bind your application to one vendor's data model and lifecycle. If you call several models through one aggregator key, that lock-in becomes friction: each backend has its own thread semantics, retention rules, and limits.

A portable alternative is to own the state:

• Store every message in your database in a neutral format.
• Assemble the messages array yourself before each request.
• Send the full (or trimmed) array to whichever backend you choose.

With one API key that reaches multiple model families, this means switching from one model to another is a config change, not a migration.

Design a vendor-neutral message schema

Store messages in the common shape most chat APIs accept, with fields your app controls:

• id — your own stable identifier (not vendor-assigned).
• conversation_id — groups messages into a thread.
• role — system, user, assistant, or tool when you support tool calls.
• content — text, or a structured payload for multimodal or tool data.
• created_at — ordering and auditing.
• model / provider — which backend produced an assistant message; useful for debugging and for per-backend trimming rules.
• token_estimate — optional cached estimate so trimming does not require re-tokenizing the whole history.

Keep the system prompt as a first-class record (either a dedicated conversation field or a system message with a fixed position) so trimming never removes it by accident.

Normalize at the adapter boundary

Each backend may differ in how it expects content blocks, tool calls, or system prompts (some take a top-level system field, others accept a system role in the array). Handle that in an adapter layer:

• The database always stores the neutral schema.
• A per-backend adapter maps neutral messages to that provider's request format.
• The response is mapped back to the neutral schema before it is persisted.

This keeps your storage stable while letting you change providers freely. It also means a conversation started on one model can continue on another without rewriting stored data.

Trim before you send, not after you fail

Context windows differ across models, so build the request to fit the target backend rather than waiting for an error:

• Compute an input budget from the model's context size minus room for the response.
• Include the system prompt first; it is usually fixed and must survive every trim.
• Keep the most recent turns, since they carry the current intent.
• Preserve user/assistant turn pairs where the API requires alternating roles — do not leave a dangling assistant message at the start of the window.
• Drop the oldest exchanges first; drop whole turns rather than truncating mid-sentence when possible.
• If a single recent turn is too large, summarize or truncate it explicitly and record that you did so.

Token estimates stored per message make this cheap: sum backwards from the newest message until the budget is reached, then stop at a clean turn boundary.

Rehydrate on every call

Treat the stored history as the source of truth and rebuild the request each time:

1. Load messages for the conversation in order.
2. Prepend or confirm the system prompt.
3. Apply the target backend's trim policy.
4. Map through the adapter into the provider's request shape.
5. Send the request with the same one API key you use for other models.
6. Persist the assistant reply in the neutral schema, tagged with the model that produced it.

Because the array is assembled fresh, you can change the trim policy, the system prompt, or the backend between any two turns.

Keep prefixes stable for caching

Some providers cache repeated input prefixes. Unstable history hurts that:

• Append new messages rather than editing old ones.
• Keep the system prompt byte-identical across calls in a conversation.
• When trimming, cut from the front so the remaining prefix still matches previous requests.
• Avoid reordering messages or injecting timestamps into content if you want prefix reuse.

Practical safeguards

• Store the raw response alongside the normalized one when you need to debug provider-specific behavior.
• Version your schema and adapters so old conversations still rehydrate correctly.
• Log the estimated input size and which messages were trimmed; this makes token-limit issues diagnosable.
• Test the same conversation against two different backends to confirm the neutral schema round-trips cleanly.

Summary

Owning conversation state means your message history is portable: trimmed to fit each backend, rehydrated before every call, and sent through one API key. Vendors change; your stored format and adapter layer stay the same.