Reordering Few-Shot Examples on One Key: Where the Same Exemplars Land Differently Across Backends

Why Example Order Matters

Few-shot prompting relies on in-context learning: the model infers the task from the examples you provide. The order of those examples can influence the model's interpretation, especially when examples are ambiguous or when the task has multiple valid patterns. Different backends process sequences differently due to variations in architecture, training data, and tokenization, so the same set of examples in the same order may yield different results.

Backend Variability

When you call different models through a single API key (e.g., Claude, GPT, DeepSeek, Qwen, GLM, Kimi), you are interacting with distinct model families. Each has its own biases:

• Recency bias: Some models weight later examples more heavily.
• Primacy bias: Others may focus on the first example.
• Pattern sensitivity: Certain models are more sensitive to order when examples have similar surface forms.
• Tokenization effects: The same text can tokenize differently, altering the effective context.

These differences mean that an ordering tuned for one backend may not generalize.

Designing an Order-Robust Test

To determine whether example order matters across backends, run a controlled experiment:

1. Fix the example set: Choose a small, representative set of few-shot examples (e.g., 4–6 examples) that cover the task's edge cases.
2. Generate permutations: Create several orderings—random shuffles, reverse order, and a few hand-picked sequences.
3. Evaluate across backends: For each ordering, query multiple backends with the same prompt template and a fixed set of test inputs.
4. Measure consistency: Compare outputs for correctness and stability. Use metrics like exact match, F1, or human judgment if needed.
5. Analyze variance: Look at how performance varies across orderings for each backend. High variance indicates order sensitivity.

Choosing a Backend-Agnostic Ordering

The goal is to find an ordering that performs reasonably well across all backends, not just the one you tuned on. Follow these heuristics:

• Diversity first: Start with examples that are diverse in input and output, then gradually narrow to more similar cases.
• Easy to hard: Place simpler examples earlier to establish the task, and more complex or ambiguous ones later.
• Avoid extremes: If a backend shows strong recency or primacy bias, an ordering that balances both ends can mitigate the effect.
• Use a validation set: Evaluate candidate orderings on a held-out set of inputs across backends, and pick the one with the best average performance.
• Consider ensemble ordering: If no single ordering works well everywhere, you can dynamically reorder based on the backend, but that requires backend detection—often impractical with a unified API key.

Practical Implementation

When using an API aggregator that routes to multiple backends, you can implement order testing as part of your prompt development pipeline:

• Store your examples in a structured format (e.g., JSON) so you can easily permute them.
• Write a script that iterates over orderings and backends, calls the API, and logs results.
• Use a simple scoring function to rank orderings by cross-backend performance.
• Once you find a robust ordering, hard-code it in your prompt template.

Remember that model updates can change behavior, so periodically re-run your order tests.

Cost Considerations

Each API call consumes tokens, and testing multiple orderings across backends increases usage. With the pay-as-you-go model (official price × 1.3), plan your experiments to be efficient: use a small but representative test set, and limit permutations to a manageable number (e.g., 5–10 orderings). If you contribute examples or improvements back to the community, you may earn credits (official price × 1.1, or × 1.2 for premium models) in USDC, which can offset your testing costs.

Conclusion

Example order can have a measurable impact on few-shot performance, and this impact varies by backend. By systematically testing orderings across the backends you use, you can select an ordering that is robust to backend swaps, ensuring consistent results with a single API key. Focus on diversity, gradual difficulty, and empirical validation rather than assuming one ordering fits all.