Long Chat, Shrinking Budget on One Key: Trimming Multi-Turn Context Without Losing the Thread

Why context grows and why it costs

In a multi-turn chat, every request usually resends the whole conversation so far. Turn 20 carries the tokens of turns 1–19 with it. That is convenient for the model, but it means cost and latency climb roughly with the square of the conversation length: each new turn pays for all prior turns again.

When you call models through a single metered API key, this is a direct budget line. You pay the official per-token price × 1.3 on each call, so wasted context is wasted spend. Trimming old turns is one of the few optimizations that reduces cost on every subsequent request without changing which model you use.

The two core strategies

There are two broad families of context control. Most production systems combine them.

• Sliding window: keep only the last N turns (or last N tokens) verbatim and drop everything older. Simple, predictable, cheap to implement. Risk: the model forgets facts established early in the conversation.
• Summarization / rolling memory: replace old turns with a compact summary, then keep recent turns verbatim. Preserves facts but adds a summarization call and some complexity.

A common hybrid: keep the system prompt plus a running summary plus the last few turns verbatim.

Designing a sliding window

• Bound by tokens, not turns. Turns vary wildly in length; a token budget is what actually controls cost. Count tokens with the target model's tokenizer where possible.
• Never cut mid-message. Trim at message boundaries so the transcript stays valid for every backend's chat format.
• Pin the important anchors. Always keep the system prompt, any explicit user constraints, and the most recent user message. Window the middle.
• Keep pairs together. If you drop an assistant reply, drop the user turn that prompted it, and vice versa, to avoid dangling references.

Designing rolling summaries

• Summarize in chunks, not every turn. Rewrite the summary when the dropped region grows past a threshold, so you are not paying for a summarization call on every request.
• Summarize for facts, not prose. Ask for decisions, entities, constraints, open questions, and unresolved tasks. Skip pleasantries and redundant restatements.
• Keep the summary structured. A short bulleted "state of the conversation" block is easier for downstream models to use than a paragraph, and it transfers better across different backends.
• Store the summary, not just the transcript. Persist summary plus recent turns so a new session can resume without replaying everything.

Keeping the thread coherent across backends

Because one key can reach several model families, the trimmed context has to be portable.

• Prefer plain role/content messages over provider-specific fields when storing history.
• Put durable instructions in the system prompt, not buried in early user turns that may be trimmed.
• When switching models mid-conversation, re-send the summary and recent turns rather than assuming cached state.
• Normalize tool/function-call transcripts into readable text in the summary so another backend can follow what happened.

Verifying the trimmed conversation still works

Trimming is only safe if you can show it did not break answers. Treat it like a regression test.

• Build a small eval set from real sessions. Take conversations where you know the correct next answer, then run them through the trimmed pipeline.
• Check fact retention. Ask questions whose answers were established only in the dropped region; the model should answer from the summary.
• Check instruction retention. Verify constraints from the system prompt and early user turns are still honored after trimming.
• Compare against full context. Run the same prompt with untrimmed history and diff the answers. Track where they diverge.
• Log what was dropped. Store the summary and the trim boundary with each request so failures are diagnosable.
• Watch for drift over long runs. Re-run the eval as the summary is rewritten; small omissions compound.

Tuning the trade-off

There is no universal right window size. Tune against your own eval set:

• Shrink the window until answer quality drops on your tests, then back off.
• Increase summarization frequency only when fact-retention tests fail.
• Measure cost per successful turn, not cost per call. A cheap call that produces a wrong answer is expensive.

How this fits the pricing model

Every token you avoid resending is a token you do not pay for at official price × 1.3. Trimming is therefore a straightforward lever on spend, and it composes with model choice: keep a strong model for hard turns and a cheaper one for routine turns, with the same trimmed context fed to either.

If you also contribute keys, note that contributor credit (official price × 1.1, or × 1.2 for premium) is separate from what callers pay; trimming context does not change the credit formula, it just reduces the token volume everyone is transacting.

A practical checklist

• Bound history by tokens, trim at message boundaries, pin system prompt and latest turn.
• Summarize dropped regions into a structured state block; refresh in chunks.
• Store summary + recent turns in a provider-neutral format.
• Keep a regression eval of real sessions and re-run it whenever trimming logic changes.
• Compare trimmed vs full-context answers before trusting a new window size.