Chaining Multiple Backends in One Request Pipeline: Passing State Between Calls on a Single Key

Why Chain Backends?

Chaining backends means using the output of one model as the input to another within a single request pipeline. This pattern is useful when different models excel at different tasks—for example, using one model for drafting and another for review, or using a fast model for initial filtering and a more capable model for final generation. With a single API key that provides access to multiple backends (e.g., Claude, GPT, DeepSeek, Qwen, GLM, Kimi), you can orchestrate these calls without managing separate credentials.

Pipeline Architecture

A sequential pipeline executes calls in order, passing state from one step to the next. The key is to treat each model call as a discrete step that produces an output which becomes part of the input to the next step.

• Step 1: Send an initial prompt to backend A.
• Step 2: Take the output from step 1, optionally transform it, and send it to backend B.
• Step 3: Continue as needed.

All calls use the same API key and are billed according to the pricing of each backend (official price × 1.3 for users).

Passing State Between Calls

State can be passed explicitly by including the previous output in the next prompt, or implicitly by maintaining a conversation history. The simplest approach is explicit inclusion:

response_a = call_model("model-a", prompt="Generate a draft.")
draft = response_a["output"]
response_b = call_model("model-b", prompt=f"Review this draft: {draft}")

For more complex pipelines, you might accumulate context in a dictionary and format it appropriately for each backend.

Checkpointing Intermediate Results

A critical concern in multi-step pipelines is failure handling. If step 3 fails, you don't want to re-execute steps 1 and 2, as that would incur additional costs. Checkpointing means saving the output of each successful step to persistent storage (e.g., a database, file, or in-memory cache) so that on retry, you can resume from the last successful step.

Implementation strategies:

• Persist after each step: Write the output to a durable store immediately after receiving it.
• Use an idempotency key: Tag each pipeline run with a unique ID to avoid duplicate execution.
• Implement a resume mechanism: On failure, load the last checkpoint and continue from the next step.

Example:

import json

def run_pipeline(pipeline_id, steps):
    state = load_checkpoint(pipeline_id) or {}
    for i, step in enumerate(steps):
        if f"step_{i}" in state:
            continue  # skip already completed
        result = call_model(step["model"], prompt=step["prompt"].format(**state))
        state[f"step_{i}"] = result["output"]
        save_checkpoint(pipeline_id, state)
    return state

This ensures that if the pipeline fails at step i, re-running will skip steps 0..i-1 and only re-execute step i and beyond.

Cost Considerations

Each model call is billed independently. By checkpointing, you avoid paying twice for the same computation. Remember that users pay the official price × 1.3 for each call, and contributors earn credits at official price × 1.1 (or × 1.2 for premium models). This means efficient pipeline design directly impacts your costs.

Error Handling and Retries

Transient errors (e.g., network issues) should be retried with exponential backoff. However, avoid retrying the entire pipeline; instead, retry only the failed step. Use the checkpoint to resume. Also, consider timeouts and fallback models: if a backend is unavailable, you might route to an alternative model, but be aware that this changes the output and may affect downstream steps.

Best Practices

• Keep steps modular: Each step should have a clear input and output.
• Validate outputs: Before passing to the next step, ensure the output meets expectations (e.g., format, length).
• Log everything: Record prompts, outputs, and timings for debugging and auditing.
• Use a single key: Leverage the convenience of one API key for all backends, simplifying authentication.
• Monitor costs: Track usage per pipeline run to understand spending.

Conclusion

Chaining multiple backends in a single request pipeline is a powerful technique for building complex AI workflows. By passing state between calls and checkpointing intermediate results, you can create robust, cost-efficient pipelines that handle failures gracefully. With a unified API key, you can seamlessly integrate various models without the overhead of managing multiple credentials.