Splitting Traffic Between Two Prompt Versions on One Key: A Lightweight A/B Setup Without a Feature-Flag Service

Why a Lightweight A/B Setup for Prompts

Prompt changes can shift model behavior in subtle ways. Instead of shipping a new prompt blindly, you can split traffic between two versions and compare outcomes. A feature-flag service is overkill for many small teams; a deterministic hash on a stable request ID gives you a clean 50/50 split with minimal infrastructure.

The Core Idea: Hash a Stable Request ID

Assign each request to a variant by hashing a stable identifier (e.g., user ID, session ID, or a request-scoped UUID) and taking the result modulo 2. This yields a deterministic, roughly 50/50 split without storing assignments.

• Use a stable ID that is consistent for the same user or session.
• Hash with a simple function (e.g., SHA-256) and convert to an integer.
• variant = hash(id) % 2 gives 0 or 1 for two variants.

Because the assignment is deterministic, the same ID always maps to the same variant, avoiding inconsistent experiences.

Implementing the Split

In your application code:

1. Extract or generate a stable request ID (e.g., from an authenticated user ID or a cookie).
2. Compute the hash and modulo to pick variant A or B.
3. Select the corresponding prompt version.
4. Send the request to your aggregator API using your single key.

No external service is needed; the logic lives in your codebase.

Logging Which Variant Hit Which Backend

For each request, log at least:

• Request ID (the stable ID used for hashing)
• Variant (A or B)
• Model/backend used (e.g., Claude, GPT, DeepSeek)
• Timestamp
• Outcome metrics (e.g., latency, token counts, user feedback if available)

This log can go to your existing logging system (stdout, a database, etc.). The key is to capture the variant and backend so you can later join with outcome data.

Comparing Outcomes

After collecting enough data, compare metrics between variants. Since assignment is random (via hash), differences can be attributed to the prompt version. Use simple statistical tests (e.g., chi-square for categorical outcomes, t-test for continuous metrics) to check significance.

• Ensure you have a sufficient sample size before drawing conclusions.
• Segment by model if you are splitting across multiple backends.

Using a Single Aggregator Key

With an aggregator, you use one API key to call multiple models. Your A/B test does not require separate keys or accounts; you simply route requests to different prompts and possibly different models under the same key. Billing is consolidated, and you can track costs per variant by logging token usage alongside variant assignment.

Avoiding Common Pitfalls

• Unstable IDs: If the ID changes between requests, users may flip variants. Use a consistent ID.
• Hash bias: Use a good hash function; modulo bias is negligible for large hash spaces.
• Logging gaps: Ensure every request logs the variant; otherwise, comparisons are biased.
• Early stopping: Don’t peek at results too often; decide on a sample size beforehand.

This approach gives you a pragmatic A/B test without extra services, leveraging your aggregator key and existing logging.