First Token vs Full Completion: Measuring Per-Backend Latency Through One Aggregator Key

Why measure latency per backend?

When you access multiple model backends through one aggregator key, latency can vary significantly between providers. Measuring both time-to-first-token (TTFT) and full completion time helps you understand the user experience and make informed decisions about model selection. This guide shows you how to measure these metrics through a single API key, without needing separate accounts or KYC.

Key concepts: TTFT vs full completion time

• Time-to-first-token (TTFT): The time from sending a request until the first token of the response is received. This affects perceived responsiveness, especially for streaming responses.
• Full completion time: The total time from request to the last token of the response. This depends on the number of output tokens and the model's throughput.

Both metrics matter: TTFT for interactivity, full completion for overall wait time.

Prerequisites

• An API key from the aggregator (top up with USDC on Base, no KYC).
• Access to multiple models (e.g., Claude, GPT, DeepSeek, Qwen, GLM, Kimi).
• A programming environment (Python example below).
• Basic familiarity with HTTP requests and streaming.

Step-by-step measurement

1. Set up your client

Use any HTTP client. Here's a Python example using requests:

import requests
import time

API_KEY = "your_aggregator_key"
BASE_URL = "https://api.aggregator.com/v1"  # replace with actual endpoint

2. Craft a standardized prompt

Use the same prompt for all backends to ensure fair comparison. For example:

prompt = "Explain the concept of latency in one paragraph."

3. Measure TTFT and full completion time

Send a streaming request and record timestamps:

def measure_latency(model, prompt):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True
    }
    start = time.time()
    first_token_time = None
    with requests.post(f"{BASE_URL}/chat/completions", json=payload, headers=headers, stream=True) as resp:
        resp.raise_for_status()
        for line in resp.iter_lines():
            if line:
                if first_token_time is None:
                    first_token_time = time.time()
                # process token if needed
    end = time.time()
    ttft = first_token_time - start if first_token_time else None
    total = end - start
    return ttft, total

4. Run across multiple models

Loop through the models you want to compare:

models = ["claude-3-opus", "gpt-4", "deepseek-chat", "qwen-max", "glm-4", "kimi"]
for model in models:
    ttft, total = measure_latency(model, prompt)
    print(f"{model}: TTFT={ttft:.2f}s, Total={total:.2f}s")

5. Repeat and average

Latency varies due to network conditions and load. Run each measurement several times and average the results for more reliable comparisons.

Best practices

• Use streaming: TTFT is only meaningful with streaming responses.
• Control output length: Longer outputs increase full completion time; keep max_tokens consistent.
• Warm-up requests: Discard the first request to avoid cold-start effects.
• Geographic consistency: Run tests from the same location to reduce network variability.
• Account for rate limits: If you hit rate limits, results may be skewed. Adjust concurrency accordingly.

Interpreting results

• High TTFT and low total time: model starts slowly but generates quickly.
• Low TTFT and high total time: model starts quickly but generates slowly.
• Both high: possible network issues or backend congestion.

Use these insights to choose models for interactive applications (prioritize low TTFT) or batch processing (prioritize low total time).

Cost considerations

Through the aggregator, you pay the official price × 1.3 for each request. Measuring latency does not incur extra cost beyond the tokens used. If you contribute model capacity, you are credited official price × 1.1 (or × 1.2 for premium models) in USDC.

Conclusion

Measuring TTFT and full completion time per backend through one aggregator key is straightforward. By following this guide, you can make data-driven decisions about which models to use for different workloads. Remember to run multiple trials and keep conditions consistent for accurate comparisons.