An LLM Latency Benchmark Prompt That Plans the Test for You

A single latency number lies. "This model responds in 900 milliseconds" hides whether that's the wait before the first token appears or the time to stream a full answer, and those two feel completely different to a user. Any llm latency benchmark worth running splits latency into its parts and measures them on the workload you actually serve.

This prompt does the planning. Give it your workload shape, prompt sizes, output lengths, and concurrency, and it designs a benchmark: which conditions to test, how many trials each, and what to record for time to first token and tokens per second. Run the plan, paste the results back, and it reads them into a routing decision.

The point is that you benchmark your traffic, not someone else's. A vendor's chat-sized test predicts nothing about your 8K-token summarization calls.

Why latency reports don't help you choose

The top results either rank providers for you or assume heavy infrastructure. AIMultiple's LLM latency benchmark is a careful provider comparison, but it tests its own prompt set and gives you no methodology to reuse on your workload. Anyscale's latency metrics docs define TTFT and TPOT well but assume a serving stack. And cloud guides like Databricks' throughput benchmark are bound to one platform.

So you get someone else's numbers, not a way to produce your own. That's the gap. A prompt that designs a benchmark for your prompt-size and output-length mix beats a leaderboard built on a workload that isn't yours.

Here's the stance: median latency is the wrong metric to optimize. Users don't remember the median; they remember the slow request that made them think the thing froze. Benchmark the p95 and p99, because tail latency is what churns people, and a model with a great median and an ugly tail will feel worse than its average suggests.

What you can do with this prompt

• Design a latency benchmark matched to your real prompt sizes and outputs.
• Separate TTFT from tokens-per-second so the trade-off is visible.
• Decide how many trials per condition to see the spread, not just the median.
• Plan for tail latency (p95, p99), not just averages.
• Interpret a results table into a concrete model-routing decision.
• Re-run the same plan after a model update to catch regressions.

Anatomy of the latency-benchmark prompt

The prompt has two modes: design a plan from your workload, then interpret the results you collect.

Variables:
  {{workload_profile}}  – prompt sizes, output lengths, concurrency
  {{models}}            – the models/endpoints to compare
  {{slo}}               – your latency targets (e.g. TTFT < 1s)
  {{results_table}}     – measured numbers, for the interpret pass

Prompt (plan mode):
  Role: performance engineer designing a latency benchmark.
  Output a test matrix: conditions x trials, what to measure
  (TTFT, TPOT, tokens/sec, p50/p95/p99), and how to isolate
  network from model time.

Output contract (restate on the final line):
  - test matrix (conditions, trial count)
  - metrics to record per trial
  - the routing decision rule, once results exist

The {{workload_profile}} variable is the difference between a benchmark that predicts your production behavior and one that measures a generic chat turn. Prefill is compute-bound and decode is memory-bound, so input length and output length pull latency in different directions.

Step-by-step usage

1. Profile your real workload

Fill {{workload_profile}} with the truth: maybe 6K-token inputs and 300-token outputs for summarization, or 200-token inputs and 1K outputs for drafting. These shapes drive everything. Guess them and the benchmark measures a workload you don't run.

2. Set latency targets you'll act on

{{slo}} is your bar: "TTFT under one second for chat, tokens-per-second above 40 for streaming." Targets turn raw numbers into pass/fail, which is what a routing decision needs.

3. Generate the test matrix and run it

The plan tells you which conditions to test and how many trials each. Run them against each model, recording TTFT and per-token time per trial. Capture network time separately so you don't blame the model for your round-trip.

4. Paste the results back for interpretation

Drop the measured numbers into {{results_table}} and run the interpret pass. It reads TTFT, tokens-per-second, and the tail percentiles against your SLO and recommends which model serves which workload.

5. Route, then re-benchmark on updates

Route long-output streaming to the high-tokens-per-second model and latency-sensitive chat to the low-TTFT one. Then re-run the same plan after any model update, because providers change inference behavior without changing the model name.

Prompt-craft patterns for latency benchmarking

Measure the tail, not the average. A model averaging 800ms with a p99 of four seconds will feel broken to one user in a hundred, and that user complains loudest. Tell the prompt to plan enough trials per condition to estimate p95 and p99, not just enough for a median.

Per condition, run enough trials to estimate p95/p99,
not just p50. Report the spread. A tight median with a
fat tail is worse than a slightly slower, stable model.

Separate prefill from decode. A long input punishes TTFT (prefill, compute-bound), while a long output punishes total time through decode (memory-bound). A benchmark that only varies one hides half the picture. Vary input length and output length independently so you can see which one your workload is sensitive to.

Account for provider variance over time. The same model id can get faster or slower as a provider re-balances capacity, and managed endpoints add queuing delay under load that a single-request test never sees. So benchmark under realistic concurrency, pin what you can, and re-run on a schedule. A number you measured last quarter isn't a number you have today.

Variables you'll set

Getting started

1. Profile your real prompt sizes, output lengths, and concurrency.
2. Set TTFT and tokens-per-second targets you'll route on.
3. Generate the test matrix and run it against each model.
4. Record TTFT and per-token time per trial, with network time split out.
5. Paste results back and read them against your SLO.
6. Route each workload to the model that wins its metric.
7. Re-benchmark on every model update. The Performance Regression Harness pack turns this into a repeatable check that flags latency regressions automatically.

Latency is one axis of model choice; quality is the other. Pairing a latency benchmark with a quality eval keeps you from routing to a fast model that answers worse, which is where an LLM Eval System Design playbook fits.

Latency benchmarking sits next to two cost-and-quality jobs: reducing AI agent token cost, since the cheapest model and the fastest model are rarely the same one, and context window budgeting for AI agents, because input length drives both your TTFT and your bill.