Routing by Prompt Length on One Key: Sending Short and Long Requests to Different Backends

Why Prompt Length Matters

Sending a long prompt to a model with a small context window can cause silent truncation, where the model ignores part of your input. This leads to incomplete or incorrect responses. Conversely, using a large-context model for short prompts may waste resources. Routing by prompt length helps you balance cost and reliability.

Measuring Tokens Before Sending

Before you choose a backend, estimate the number of tokens in your prompt. Token count varies by model, but you can use a tokenizer library or a simple heuristic: roughly one token per word for English text. For accuracy, use the tokenizer specific to the model family you plan to call. Many API aggregators expose a token counting endpoint or include token usage in responses, but you should measure before sending to make routing decisions.

Setting Up Routing Rules

Define thresholds based on the context limits of your available models. For example, if you have a cheap model with a small context window and a more expensive model with a large context window, set a token threshold that safely fits within the small model's limit. Prompts below the threshold go to the cheap model; prompts above go to the large-context model. Always leave a buffer for the response tokens, as the context window includes both input and output.

• Short prompts: Route to models that are cost-effective and fast, even if their context window is smaller.
• Long prompts: Route to models with large context windows to avoid truncation, even if they cost more per token.

Implementing the Router

With a single API key from an aggregator, you can call different models by specifying the model name in the request. Your router logic should:

1. Estimate the token count of the prompt.
2. Compare it to predefined thresholds.
3. Select the appropriate model name.
4. Send the request with that model name.

This can be implemented as a function in your application code or as a lightweight proxy service.

Avoiding Silent Truncation

Silent truncation occurs when the input exceeds the model's context window and the excess is dropped without warning. To prevent this:

• Always check that the estimated token count plus the expected response length is less than the model's context window.
• If the prompt is too long for any available model, consider chunking or summarizing the input before sending.
• Monitor responses for signs of truncation, such as incomplete answers or the model ignoring parts of the prompt.

Cost Considerations

Using an aggregator that charges a markup over official prices (e.g., 1.3×) and credits contributors (e.g., 1.1× or 1.2×) means your routing decisions directly affect your costs. By sending short prompts to cheaper models, you minimize expenses while still handling long prompts with appropriate models.

Example Workflow

Suppose you have two models: Model A (cheap, 4K context) and Model B (expensive, 32K context). Set a threshold at 3K tokens (leaving 1K for response). For prompts under 3K tokens, use Model A; for prompts between 3K and 31K tokens, use Model B. For prompts over 31K tokens, consider splitting the input.

Conclusion

Routing by prompt length is a simple yet effective way to optimize cost and avoid truncation. By measuring tokens before sending and using a single API key to access multiple models, you can build a robust system that automatically selects the right backend for each request.