Cancelling Mid-Stream: What Happens to the Connection, the Tokens and the Bill

What Happens When You Abort a Stream Early

When you abort an SSE (Server-Sent Events) stream, you're essentially closing the HTTP connection before the server has finished sending all events. Here's what happens server-side:

• Connection termination: The client sends a TCP FIN or RST packet, signaling the connection is closing. The server detects this via the socket's read event or a write failure.
• Generation stop: Most inference servers monitor the connection and stop generating tokens once they detect the disconnect. This prevents wasted compute.
• Resource cleanup: The server releases GPU memory, KV cache, and other resources associated with that request.

Are Partial Tokens Billed?

Yes, in most cases. Billing is based on tokens generated (output) and tokens in the prompt (input). When you abort mid-stream:

• Input tokens: Always billed, as they were processed to start generation.
• Output tokens: Only the tokens actually generated before the abort are billed. Since the server stops generating, you don't pay for tokens that were never produced.

This means aborting early can save you money on output tokens, but you'll still pay for the input and any output already generated.

How to Cleanly Close the Socket

Abruptly closing the connection can leave resources hanging. Use these best practices:

• Use AbortController: In JavaScript, create an AbortController and pass its signal to fetch. Call controller.abort() to cancel the request.
• Handle errors gracefully: Catch AbortError exceptions to distinguish user-initiated cancellations from actual errors.
• Close the EventSource: If using EventSource, call eventSource.close() to terminate the connection.
• Server-side timeouts: Implement server-side timeouts to automatically clean up stale connections.

Example: Aborting a Fetch Stream

const controller = new AbortController();
const signal = controller.signal;

fetch('https://api.example.com/v1/chat/completions', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'claude', messages: [...] }),
  signal
})
.then(response => {
  const reader = response.body.getReader();
  // Read stream...
})
.catch(err => {
  if (err.name === 'AbortError') {
    console.log('Stream aborted by user');
  }
});

// Later, to abort:
controller.abort();

Implications for API Aggregators

When using an API aggregator like ours, the same principles apply. The aggregator forwards your request to the upstream provider. If you abort:

• The aggregator detects the disconnect and stops the upstream request.
• Billing is based on tokens actually generated and returned.
• Your account is charged according to the aggregator's pricing (e.g., official price × 1.3).

Best Practices

• Abort when needed: Don't let unnecessary generation run; abort if the response is no longer needed.
• Monitor costs: Understand that input tokens are always billed, so craft prompts carefully.
• Implement retries: For transient errors, not for user aborts.
• Use streaming wisely: Streaming is great for UX, but ensure you handle aborts to avoid waste.