Streaming Tool Calls on One Key: Reassembling Partial arguments JSON From Each Backend

Why streaming tool calls need careful handling

When you stream a chat completion that includes tool calls, the model doesn't send the full arguments JSON at once. Instead, it sends incremental fragments inside delta.tool_calls chunks. Each chunk may contain a piece of the function name, a partial arguments string, or just an index. Reassembling these fragments correctly is essential—especially when using an aggregator that routes to multiple backends (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) behind one API key.

Different backends may chunk the arguments JSON differently. Some send many small fragments; others send larger pieces. There is no guarantee that a fragment ends at a JSON boundary. Your code must treat the argument stream as an opaque string until the tool call is complete.

How argument fragments arrive in chunks

In a streaming response, each chunk contains a choices[0].delta object. When a tool call is being generated, delta.tool_calls is an array (often with one element). Each element has:

• index: identifies which tool call this fragment belongs to (important when multiple calls are streamed).
• id and type: usually only present in the first chunk for that index.
• function.name: may arrive in one or more chunks.
• function.arguments: a string fragment that must be concatenated with previous fragments for the same index.

Backends may also send an empty delta or a finish_reason chunk. Your reassembly logic should ignore chunks without tool call data and accumulate only the fields that appear.

Reassembly strategy: accumulate, then parse

The safest approach is to buffer all argument fragments per tool call index and parse only after the stream ends or the finish reason indicates tool calls are complete.

1. Initialize an empty object to track tool calls by index.
2. For each chunk, if delta.tool_calls exists, iterate over its entries.
3. For each entry, ensure a record exists for its index with an empty name and arguments string.
4. Append function.name if present, and append function.arguments if present.
5. After the stream ends (or when finishreason is toolcalls), parse the accumulated arguments string as JSON.

This parse-on-complete approach avoids partial JSON errors and works uniformly across backends because it never assumes fragment boundaries.

Strict-mode fallback for malformed JSON

Even with careful reassembly, some backends might occasionally emit arguments that are not valid JSON (e.g., trailing commas, unquoted keys, or truncation due to token limits). A strict-mode fallback can help you recover gracefully without breaking the aggregator flow.

• First, attempt JSON.parse on the accumulated string.
• If parsing fails, try a lenient repair: trim whitespace, remove trailing commas, and wrap unquoted keys in quotes (only if safe).
• If repair still fails, treat the tool call as invalid and either retry the request or return a structured error to the caller.
• Never silently drop the tool call—log the raw arguments for debugging.

Because the aggregator routes to different backends, your fallback should be backend-agnostic. Avoid backend-specific assumptions about JSON formatting.

Handling multiple tool calls and interleaved chunks

Some backends stream multiple tool calls in parallel, interleaving fragments by index. Always group by index and do not assume fragments for one call arrive contiguously.

• Use a map keyed by index to store accumulated name and arguments.
• Process each chunk independently and append to the correct record.
• After the stream, sort by index if order matters for your application.

This approach also handles cases where a backend sends the function name in a later chunk than the arguments.

Practical tips for aggregator users

• Always use the index field to disambiguate tool calls; never rely on array position alone.
• Buffer arguments as strings; do not attempt to parse incrementally.
• Set a reasonable maximum buffer size to avoid memory issues with very large arguments.
• Test with each backend you plan to use (Claude, GPT, DeepSeek, Qwen, GLM, Kimi) to observe chunking patterns, but keep your code generic.
• Remember that the aggregator charges official price × 1.3 for usage, and contributors earn official price × 1.1 (or × 1.2 for premium) in USDC—so efficient streaming also helps control costs.

Example pseudocode

tool_calls = {}

for chunk in stream:
    delta = chunk.choices[0].delta
    if not delta.tool_calls:
        continue
    for tc in delta.tool_calls:
        idx = tc.index
        if idx not in tool_calls:
            tool_calls[idx] = {"name": "", "arguments": ""}
        if tc.function.name:
            tool_calls[idx]["name"] += tc.function.name
        if tc.function.arguments:
            tool_calls[idx]["arguments"] += tc.function.arguments

# After stream ends
for idx, tc in tool_calls.items():
    try:
        args = json.loads(tc["arguments"])
    except json.JSONDecodeError:
        args = repair_json(tc["arguments"])  # fallback
    # use tc["name"] and args

This pattern works with any backend behind the aggregator and keeps your integration robust.