Benchmarking Latency Per Model on One API Key: TTFT, Total Time and Variance

Latency is a critical factor in production LLM applications. Vendor-published numbers often reflect ideal conditions, not your workload. With a single API key from an aggregator, you can benchmark multiple models under identical conditions. This guide shows you how to measure TTFT, total time, and p95 jitter across six popular models.

Why Benchmark Latency Yourself?

Vendor benchmarks typically use optimized prompts and may not reflect real-world variability. By running your own tests, you capture:

• The impact of your network and location
• Model behavior under your specific prompt patterns
• Variance that affects user experience
• True comparative performance across providers

Models to Test

Using one API key, you can access models like Claude, GPT, DeepSeek, Qwen, GLM, and Kimi. The exact model names depend on your aggregator's catalog, so adjust the script accordingly.

What to Measure

• Time to First Token (TTFT): How long until the first token arrives. Critical for perceived responsiveness.
• Total Duration: Time from request to final token. Affects overall task completion.
• Jitter (p95): The 95th percentile of latency. Shows worst-case performance users might experience.

The Benchmark Script

Below is a Python script that sends the same prompt to each model, repeats it multiple times, and logs TTFT, total time, and calculates p95 jitter.

import openai
import time
import statistics

# Configure your aggregator's base URL and API key
client = openai.OpenAI(
    base_url="https://api.your-aggregator.com/v1",
    api_key="YOUR_API_KEY"
)

MODELS = [
    "claude-3-sonnet",  # replace with actual model IDs
    "gpt-4",
    "deepseek-chat",
    "qwen-max",
    "glm-4",
    "kimi-latest"
]

PROMPT = "Explain the concept of latency in distributed systems in three sentences."
RUNS = 5  # number of times to test each model

def measure_ttft(model):
    start = time.time()
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": PROMPT}],
        stream=True
    )
    first_token_time = None
    for chunk in stream:
        if first_token_time is None:
            first_token_time = time.time()
        # continue consuming stream to complete
    end = time.time()
    ttft = first_token_time - start
    total = end - start
    return ttft, total

results = {}
for model in MODELS:
    ttfts = []
    totals = []
    for _ in range(RUNS):
        ttft, total = measure_ttft(model)
        ttfts.append(ttft)
        totals.append(total)
        time.sleep(1)  # avoid rate limiting
    results[model] = {
        "ttft_mean": statistics.mean(ttfts),
        "ttft_p95": sorted(ttfts)[int(0.95 * len(ttfts))],
        "total_mean": statistics.mean(totals),
        "total_p95": sorted(totals)[int(0.95 * len(totals))],
    }

for model, data in results.items():
    print(f"{model}: TTFT mean={data['ttft_mean']:.2f}s p95={data['ttft_p95']:.2f}s | Total mean={data['total_mean']:.2f}s p95={data['total_p95']:.2f}s")

Interpreting Results

• TTFT matters most for chat: Lower TTFT feels snappier.
• Total time matters for batch: If you process many prompts, total throughput is key.
• Jitter indicates consistency: High p95 means some users get slow responses.

Compare models under your own network and prompt style. A model that excels in one region may lag in another.

Accounting for Cost

When benchmarking, remember that users pay official price × 1.3, while contributors get official price × 1.1 (or × 1.2 for premium models) back in USDC. This means your effective cost varies, but latency itself is independent of pricing.

Next Steps

• Expand the script to log token counts and calculate tokens per second.
• Test with longer prompts to see how context length affects latency.
• Run the benchmark periodically to detect performance regressions.
• Share your findings with the community to build a collaborative latency table.