Splitting One Nightly Batch Across Cheap and Capable Models: A Routing Rule You Can Write in Ten Lines

Why split a batch?

Nightly batches often contain mixed workloads: some tasks require deep reasoning, while others just need quick processing. Using a single high-end model for everything wastes money. A routing rule can send simple tasks to cheaper models and complex tasks to capable ones, keeping quality high while cutting costs.

The ten-line routing rule

The core idea: for each item, decide which model to use based on a simple heuristic. Here's a rule you can implement in about ten lines of Python:

def route(item):
    if item.complexity > THRESHOLD:
        return "claude-3-opus"
    if item.complexity > LOWER_THRESHOLD:
        return "gpt-4-turbo"
    return "deepseek-chat"

This rule uses a complexity score (e.g., token count, keyword presence, or a precomputed difficulty) to route to one of three models. Adjust thresholds based on your quality needs.

How to determine complexity

• Token count: Longer inputs often need more capable models.
• Keyword flags: If the item contains certain keywords (e.g., "analyze", "summarize"), route to a stronger model.
• Pre-labeled difficulty: If you have historical data, train a tiny classifier to predict difficulty.
• Random sampling: For a quick start, randomly assign a small fraction to the capable model to measure quality difference.

Handling failures and fallbacks

Even cheap models can fail. Wrap each call with a fallback: if the cheap model returns low confidence or an error, retry with the next tier up. This keeps the batch robust.

def safe_call(item, model_list):
    for model in model_list:
        try:
            return call_model(model, item)
        except:
            continue

Cost and quality monitoring

Track per-model success rates and costs. If the cheap model performs well, you can lower the threshold to save more. If quality drops, raise it. Always validate on a held-out set before deploying a new rule.

Integrating with your API aggregator

With a unified API key, you can call multiple models through one endpoint. Our service charges official price × 1.3, and key contributors earn back official price × 1.1 (or × 1.2 for premium models) in USDC. This means you can route across models without managing separate billing. For contributors, the effective cost per call is reduced—but always remember the base multiplier.

Example: batch processing steps

1. Compute a complexity score for each item.
2. Route to a model using the rule.
3. Call the model via the aggregator.
4. On failure, fallback to a stronger model.
5. Log model used, cost, and output quality.
6. Periodically review and adjust thresholds.

Conclusion

A simple routing rule can significantly reduce costs for nightly batches without complex infrastructure. Start with a few thresholds, measure quality, and iterate. The ten-line rule is a practical starting point that you can adapt to your specific workload.