Stop Sequences That Cut a Chinese Sentence in Half: Tuning stop Across Backends on One Key So Multi-Byte Text Isn't Truncated Mid-Character

Why Stop Sequences Matter for Chinese Text

Stop sequences tell the model when to stop generating. But when you're working with Chinese text, a stop sequence that works fine for English can cut a Chinese sentence in half—sometimes even in the middle of a multi-byte character. This happens because different backends interpret and apply stop sequences differently: some operate byte-wise, others token-wise.

Byte-wise vs. Token-wise Stop Handling

• Byte-wise stop handling: The backend scans the raw byte stream for the stop sequence. Since Chinese characters are encoded in multiple bytes (UTF-8 uses 3 bytes per CJK character), a byte-wise match can occur inside a character if the stop sequence happens to align with a byte subsequence. This can split a multi-byte character, resulting in invalid UTF-8 or a replacement character (�).
• Token-wise stop handling: The backend checks for the stop sequence at the token level. Tokens are typically subword units that respect character boundaries, so a stop sequence is only matched when it corresponds to a complete token sequence. This avoids splitting characters, but the exact tokenization varies by model.

Backends built on different tokenizers (e.g., BPE, SentencePiece) will have different token boundaries. A stop sequence that is a single token in one model might span multiple tokens in another.

The Problem with a One-Size-Fits-All Stop Sequence

When you use a single API key to access multiple models (Claude, GPT, DeepSeek, Qwen, GLM, Kimi), you can't assume all backends handle stop sequences identically. A stop sequence like "。" (Chinese full stop) might be applied byte-wise by one backend and token-wise by another. In the byte-wise case, if the stop sequence is a substring of a multi-byte character's byte representation, truncation can occur mid-character.

Even when the stop sequence itself is multi-byte, byte-wise matching can still cause issues if the stop sequence appears as a byte subsequence across character boundaries. For example, the bytes for "。" (E3 80 82) could appear as part of other characters' byte sequences.

A Safe Pattern to Avoid Broken Characters

To prevent mid-character truncation, follow these guidelines:

• Use token-wise backends when possible: If you have control over the backend, prefer models that apply stop sequences at the token level. Most modern LLM APIs do this, but it's worth verifying.
• Choose stop sequences that are unlikely to appear as byte subsequences: Avoid single-byte or common byte patterns. Use longer, distinctive sequences that are less likely to occur inside multi-byte characters. For example, instead of a single punctuation mark, use a phrase like "\n\n" or a unique marker string.
• Post-process to validate UTF-8: After receiving the response, check that the output is valid UTF-8. If you detect replacement characters or invalid sequences, you can trim the last incomplete character or re-request with a different stop sequence.
• Test across backends: When using multiple models with one key, test your stop sequences with each model to see how they behave. Send a prompt that forces the stop sequence to appear near multi-byte characters and inspect the output.
• Consider client-side stopping: If the backend's stop handling is unreliable, you can avoid using stop sequences altogether and instead implement client-side stopping: stream the response and stop reading when you detect your desired stop pattern, ensuring you only cut at character boundaries.

Practical Example

Suppose you want to stop generation after a Chinese sentence ends with "。". Instead of using "。" as a stop sequence, you could use a longer pattern like "。\n" or a custom marker such as "<|endofsentence|>". This reduces the chance of byte-wise false positives. Alternatively, if the model supports it, use a stop token ID rather than a string.

Conclusion

Stop sequences are a powerful tool, but they require care when working with multi-byte text and multiple backends. By understanding how each backend applies stop sequences and following the safe pattern above, you can avoid truncated Chinese characters and ensure clean, valid output—all while using a single API key across models.