When the Stream Dies Mid-Sentence: Detecting a Truncated SSE Response and Resuming Without Double-Charging

Why stream truncation matters

Streaming responses via Server-Sent Events (SSE) are common for LLM APIs. When the connection drops mid-sentence, you get a partial output that looks syntactically similar to a complete one. Without proper detection, you might either treat it as final (losing the rest of the answer) or retry the whole request (wasting tokens and money).

Distinguishing a dropped connection from a legitimate stop

A legitimate stop is signaled by the API. Look for an explicit completion event such as [DONE] or a finish reason field (e.g., finish_reason: "stop"). If the stream ends without that marker, it was likely truncated. Also consider:

• Connection errors: TCP resets, timeouts, or HTTP errors (5xx, 429) indicate a problem.
• Heartbeats: Some APIs send keep-alive comments; their absence can hint at a dead connection.
• Token count: If the number of tokens received is far below the model's max output, truncation is possible.
• Content shape: A response ending mid-word or mid-sentence is a strong signal, but not definitive—always rely on the finish reason.

Capturing the partial output safely

Do not discard the partial text. Buffer all received deltas in order. When the stream ends unexpectedly, you have a prefix of the intended output. This prefix is valuable: you can resume from it instead of regenerating from scratch.

• Store the partial text along with the original prompt and any parameters.
• Note the finish reason if present (e.g., length for max tokens, which is not a connection drop but a different kind of truncation).
• Keep the last few tokens to avoid cutting mid-token when resuming.

Resuming as a second call

To continue, send a new request that includes the original prompt plus the partial output as an assistant message, and instruct the model to continue from where it left off. For example:

{
  "messages": [
    {"role": "user", "content": "<original prompt>"},
    {"role": "assistant", "content": "<partial output>"},
    {"role": "user", "content": "Continue exactly where you left off. Do not repeat any text."}
  ]
}

This approach uses the model's context to maintain coherence. The second call will generate only the remaining tokens, so you pay for the continuation, not the entire response again.

Avoiding double-charging

Double-charging happens when you retry the full request instead of resuming. To prevent it:

• Always check for a finish reason before treating a response as complete.
• On truncation, resume with the partial output rather than re-sending the original prompt alone.
• If the API supports idempotency keys, use them for retries—but note that resuming with context is not a retry of the same request; it's a new request with additional input.
• Monitor token usage: the sum of tokens from the first (partial) and second (continuation) calls should roughly equal the tokens of a single complete response.

Edge cases and best practices

• Max tokens reached: If the finish reason is length, the model stopped because it hit the token limit. You can resume with a higher limit or ask it to continue.
• Content filtering: A finish reason like content_filter means the output was intentionally stopped. Resuming may not be appropriate.
• Multiple truncations: If the connection drops repeatedly, consider fallback strategies such as non-streaming requests or a different network path.
• Idempotency: For non-streaming requests, idempotency keys help avoid duplicate charges on retries. For streaming, resuming with context is the primary defense.

Conclusion

Detecting truncation requires checking for explicit completion signals. Capturing the partial output and resuming with context lets you complete the response without paying twice. Always validate finish reasons and handle edge cases to build robust LLM integrations.