Bridging the Aggregator's Stream to a Browser SSE Consumer on One Key: Heartbeats, Backpressure and Early Close

Understanding the Streaming Path

When you call a chat completion endpoint with streaming enabled, the aggregator returns a chunked HTTP response. Each chunk typically contains a server-sent event (SSE) with a JSON payload representing a delta—a small piece of the model's output. Your browser needs to consume these events incrementally to show text as it arrives.

The challenge is bridging two worlds: the server-to-server connection between your backend and the aggregator, and the browser-to-server connection between your frontend and your backend. Each has different constraints around latency, connection lifetime, and error recovery.

Two Browser Consumption Patterns

You can consume a stream in the browser using either the native EventSource API or the fetch API with a ReadableStream reader.

• EventSource: Simple, automatically reconnects, but only supports GET requests and cannot send custom headers (like your API key). It is best suited for server-to-browser push where your backend handles authentication.
• Fetch reader: Full control over headers and request body, works with POST, and lets you parse SSE frames manually. It is more flexible but requires you to implement reconnection logic if needed.

In both cases, your backend acts as a proxy: it holds the real API key, calls the aggregator, and forwards events to the browser in a format the browser can consume.

Heartbeat Keep-Alives

Long-lived streaming connections can be dropped by intermediaries (load balancers, proxies, CDNs) if no data flows for a period. Sending periodic heartbeat comments (lines starting with :) keeps the connection alive without adding visible content.

• Send a heartbeat comment every 15–30 seconds of inactivity.
• Ensure heartbeats are forwarded to the browser so the client also sees activity.
• Do not confuse heartbeats with actual data events; clients should ignore comment lines.

Heartbeats are especially important when the model is slow to produce the first token or when there are gaps between tokens.

Backpressure and Flow Control

Backpressure occurs when the consumer (browser) reads slower than the producer (aggregator) sends. With fetch readers, you control the read loop and can pause reading when the client is slow. With EventSource, the browser handles buffering internally.

To avoid unbounded memory growth:

• In your backend proxy, respect the writable stream's drain event before writing more data.
• If the browser tab is backgrounded, consider pausing reads to avoid unnecessary bandwidth.
• For EventSource, you cannot apply backpressure directly, but you can throttle your server's writes to match the client's consumption rate indirectly.

Handling Client Disconnects

A client disconnect happens when the browser closes the connection (tab closed, navigation, or network loss). Your backend must detect this promptly to stop reading from the aggregator and free resources.

• Listen for the close or aborted event on the response object.
• When detected, cancel the upstream request to the aggregator (if supported) or stop reading and discard remaining chunks.
• Clean up any timers (e.g., heartbeat intervals) to prevent leaks.

Prompt cleanup reduces the risk of continuing to accrue token charges for output no one will see.

What Still Gets Billed

The aggregator charges based on tokens processed by the model. When you close a stream early, the model may have already generated tokens that were sent but not consumed. Billing depends on the aggregator's policy, but generally:

• You are billed for all tokens the model generated up to the point the stream is terminated.
• If you cancel the upstream request, the model may stop generating further tokens, limiting additional charges.
• Heartbeats and SSE framing do not incur token charges.

To minimize unnecessary billing, always cancel the upstream request when the client disconnects, rather than letting the model continue generating.

Putting It Together

1. Browser initiates a request to your backend (via EventSource or fetch).
2. Backend calls the aggregator with streaming enabled, using your API key.
3. Backend forwards each SSE event to the browser, adding heartbeats as needed.
4. Backend monitors for client disconnect and cancels the upstream request.
5. Billing reflects tokens generated until cancellation.

By following these patterns, you can build a responsive streaming UI while managing costs and resources effectively.