Measuring Time-to-First-Token vs. Total Latency on One Aggregator Key: Why the Backend That Feels Fastest Isn't Always the Fastest

Why latency comparisons go wrong

When you call an LLM API, you can measure two distinct numbers:

• Time-to-first-token (TTFT) – how long until the first streamed token arrives.
• Total latency – how long until the full response is complete.

These numbers answer different questions. TTFT reflects how quickly the backend starts responding, which dominates perceived speed in interactive apps. Total latency reflects how long the full completion takes, which matters for batch jobs or when you must wait for the entire answer.

Comparing backends by a single latency number is misleading. A backend with low TTFT but slow token generation may finish later than a backend with higher TTFT but faster streaming. Without controlling for these factors, you cannot say one backend is “faster” than another.

The role of streaming

When you set stream: true, the API sends tokens as they are generated. Your client can render the first token immediately, even if the full response takes much longer. This decouples perceived responsiveness from total completion time.

With stream: false, you receive the entire response at once. TTFT and total latency become effectively the same measurement, and you cannot start rendering early.

This is why latency comparisons are apples-to-oranges unless you fix the streaming mode. A backend tested with streaming will appear to have lower TTFT than the same backend tested without streaming, even though the underlying model speed is unchanged.

How to measure fairly on one aggregator key

Using a single API key across multiple backends lets you compare them under identical conditions. To get meaningful numbers:

• Use the same prompt across all backends. Vary prompt length separately if you want to study its effect.
• Fix max_tokens to a constant value. Total latency grows with output length, so different limits make comparisons invalid.
• Test both stream: true and stream: false and report TTFT and total time separately for each mode.
• Run multiple trials and report a central tendency (e.g., median) plus variability. Single measurements are noisy.
• Warm up if your client or network has cold-start effects, and note whether the backend caches responses.
• Measure on the client side from request send to first byte received (for TTFT) and to last byte received (for total time). Server-side metrics may not reflect your actual experience.

When streaming, record the timestamp of the first chunk and the timestamp of the final chunk. When not streaming, TTFT is effectively the time to the full response.

Interpreting the results

A backend that streams quickly but generates tokens slowly may have low TTFT and high total latency. That can feel fast in a chat UI because the user sees output immediately, but it may be poor for a workflow that waits for the complete answer.

Conversely, a backend with higher TTFT but faster token generation might finish sooner overall, yet feel sluggish because the user waits longer for the first token.

Neither is universally better. The right choice depends on your use case:

• Interactive chat or UI – prioritize low TTFT so the user sees progress quickly.
• Batch processing or full-response workflows – prioritize low total latency.

Practical checklist

Before comparing backends on your aggregator key:

• [ ] Same prompt, same max_tokens, same stream setting.
• [ ] Multiple trials per backend.
• [ ] Separate TTFT and total latency in your logs.
• [ ] Client-side timestamps, not just server-reported.
• [ ] Note any caching or warm-up effects.
• [ ] Report median and spread, not just a single number.

What this means for perceived speed

Streaming changes the user experience without changing the backend’s raw speed. If you test only with stream: false, you will miss how the app actually feels. If you test only with stream: true, you may overlook that total completion time still matters for some tasks.

Measure both. Report both. And when you compare backends, state the streaming mode and output length, because otherwise the numbers are not comparable.