Qwen for Long Multi-Turn Chats on One Key: Trimming History Without Breaking Continuity

Why long chats drift and get expensive

Multi-turn conversations with Qwen degrade for two predictable reasons:

• Context growth. Every turn appends messages. A long session eventually pushes the window, forcing either truncation by the provider or an error you must handle.
• Signal dilution. Old turns compete with current instructions. The model may follow stale constraints or repeat resolved questions.

The naive fix is to drop the oldest messages once a token threshold is hit. It works until the dropped messages contain the only record of a decision, a name, or a constraint. Then continuity breaks abruptly.

A better policy budgets by turn count, not just tokens, and replaces dropped turns with a compact summary rather than deleting them silently.

A three-part history policy

Treat each request's message list as having three layers:

1. System layer — stable instructions, persona, output format. Always present, never trimmed.
2. Summary layer — one or a few messages holding distilled facts from dropped history. Injected right after the system message.
3. Live layer — the most recent N turns, verbatim, in original order.

When the conversation exceeds your turn budget, move the oldest live turns into the summary layer and delete them from the live layer. The model never sees a gap; it sees a shorter but complete story.

Step 1: Set a turn budget

Define a maximum number of live turns, where one turn is a user message plus the assistant reply. Choose it by testing on your own workloads:

• Short Q&A bots tolerate a small budget because each question is mostly independent.
• Task-tracking chats need a larger budget because references like "the second option" depend on recent turns being verbatim.

Pick the smallest budget that passes your continuity checks. Smaller budgets mean shorter prompts and more predictable latency.

Step 2: Write a trimming trigger

Check the budget before each request, not after the reply. The flow is:

• Count live turns in stored history.
• If the count is within budget, send as-is.
• If it exceeds the budget, take the oldest turns beyond the limit, fold them into the summary layer, then send.

The number of turns folded per trim is your chunk size. Folding a few turns at a time keeps summaries incremental and avoids one giant rewrite. Do not trim mid-turn; always fold whole user/assistant pairs so the transcript stays well-formed.

Step 3: Summarize with the same model

Use Qwen itself to compress the turns you are about to drop. A reliable summary prompt asks for:

• Decisions and conclusions reached so far.
• Named entities — people, files, identifiers, values the user specified.
• Open questions and promised follow-ups.
• Constraints — format, tone, things to avoid.

Ask for terse bullet output, and cap the length. Then merge the new summary into the existing summary layer rather than appending a second block; repeated merging keeps the layer from growing every trim.

A simple merge rule: keep the old summary, add the new bullets, and instruct the model to produce a single consolidated block with duplicates removed. If you want to avoid an extra call, write deterministic merge code for bullets that are already short and structured.

Step 4: Pin facts that must survive

Summaries are lossy. For facts that must never disappear, keep a small, explicitly maintained pin list outside the model:

• User-stated preferences and names.
• Chosen identifiers, paths, or keys the user supplied.
• Active task constraints.

Inject the pin list verbatim into the summary layer at every request. Pins do not count against the turn budget. Keep the list short enough to review by hand; if it grows past a page, it is doing the summary's job and should be compressed.

Step 5: Verify continuity after trimming

Run a fixed regression script after any prompt-policy change. A useful pattern is a scripted conversation of well over the budget, where early turns establish facts that later turns rely on:

• Turn 1 names a project and a required output format.
• Middle turns discuss options.
• Final turns ask the model to apply the early format and facts.

Score the final answers on whether they recall the pinned facts, respect constraints, and answer the actual question. If a fact disappears, either add it to the pin list or raise the budget slightly. This loop, not intuition, should set your thresholds.

Keeping spend flat

Because billing is per token, spend follows prompt size. A turn-based budget with a capped summary and a bounded pin list keeps the prompt size roughly constant across a long session: the live window is fixed, the summary has a length ceiling, and the pins are small. Long chats then cost about the same per turn as short ones rather than growing linearly.

Two practical notes:

• Summarization calls themselves cost tokens. Summarize in chunks so each compression pass is small, and cache the latest summary so it is reused across turns.
• On this platform you top up with USDC on Base without KYC, call Qwen and other models with one API key, and pay the official price multiplied by 1.3. Key contributors are credited back official price × 1.1, or × 1.2 for premium models, in USDC. Treat the per-token figure as your planning baseline and keep prompts lean; trimming is the main lever you control.

Putting it together

The policy is small enough to implement in one request wrapper:

• Store turns as structured pairs with a timestamp.
• Before each call, enforce the turn budget and fold overflow into the summary layer using a fixed summarization prompt.
• Inject pins verbatim after the system message.
• Send system, summary, pins, then the live turns.
• Run the scripted continuity check whenever you change the budget, chunk size, or summary prompt.

Qwen handles long conversations well when the history it receives is curated. A turn budget plus a merged summary plus a few pins gives the model a coherent, bounded view of the session — and keeps your token spend predictable as the chat gets longer.