Build an LLM Eval Dataset From Production Logs With One Prompt

Most eval-dataset guides hand you principles and a governance checklist, then leave the actual curation as homework. Read them and you still face a folder of raw logs and no fast way to turn it into something a scorer can run. To build an llm eval dataset that catches real regressions, the bottleneck isn't theory. It's the grind of sampling, deduping, and labeling thousands of messy transcripts by hand.

A prompt collapses that grind. The idea is simple: feed a batch of raw production interactions to a model, and have it return a clean eval set, one row per case, with the input, the expected behavior, a difficulty tag, and the source line. You read the rows, drop the noise, and keep what mirrors real usage.

That's the whole move. Curate from logs with a prompt, review with your eyes, ship a set you can defend.

Why most teams never build an LLM eval dataset that matters

The ranking pages make this look like a framework problem. Maxim's golden-dataset guide is thorough on governance and NIST alignment, but it ships no copyable prompt and routes you toward its platform. The popular seven-ways listicle on Dev.to names the methods without an artifact to run. And MLflow's dataset docs assume you're already inside MLflow Tracing.

None of them sits down with your logs and produces the rows. That's the gap. You can read about coverage and decontamination all week and still have nothing to feed a scorer on Friday.

Here's the stance worth defending: a curated set of 150 cases drawn from your own logs beats a 5,000-row public benchmark every time. The public set tests something adjacent to your product. Your logs test the thing you actually ship, including the inputs you'd never think to invent.

What you can do with this prompt

• Sample representative cases from a batch of raw production transcripts.
• Dedupe near-identical inputs so coverage isn't faked by repetition.
• Label each case with an expected behavior, not exact expected text.
• Tag every case by difficulty and topic so blind spots become visible.
• Flag inputs that look adversarial or out-of-distribution for separate handling.
• Mark which cases are real-from-logs versus synthetic fill-ins.

Anatomy of the eval-dataset prompt

The prompt takes a batch of logs plus your definition of the job, and emits a structured eval set you can hand to any scorer.

Variables:
  {{raw_logs}}         – a batch of real input/output transcripts
  {{job_description}}  – what correct behavior looks like
  {{difficulty_tags}}  – e.g. easy / typical / edge / adversarial
  {{target_count}}     – how many cases to return after deduping

Prompt:
  Role: eval-dataset curator building a golden set from real traffic.
  Task: sample, dedupe, and label. Do NOT invent inputs that
        aren't represented in {{raw_logs}} unless gap-filling,
        and mark those as synthetic.

Output contract (restate on the final line):
  For each eval case, one row:
    - id
    - input (verbatim or lightly redacted)
    - expected behavior (described, not exact text)
    - difficulty tag
    - topic
    - source (log line ref, or "synthetic")

The {{difficulty_tags}} variable is what turns a flat list into a coverage map. Without it you get 150 typical cases and zero edges, which is exactly where models break.

Step-by-step usage

1. Pull a real batch of logs

Drop fifty to a few hundred raw transcripts into {{raw_logs}}. Don't pre-clean them. The messy ones, the truncated inputs, the user who pasted a whole file, are the cases worth keeping. Redact secrets before you paste; the prompt shouldn't see live credentials.

2. Define the job in behavior terms

{{job_description}} says what correct looks like: "answer only from the retrieved context, and say 'not in the docs' when the answer isn't there." Behavior-based labels survive model updates. Exact-text labels don't.

3. Set difficulty tags you'll actually use

Four buckets is plenty. Easy, typical, edge, adversarial. The point is to read the output and notice you have ninety typical cases and three edges, then go find more edges.

4. Read every row for representativeness

The output is a draft set, not a finished one. For each row, ask: would a user really send this? Cut the rows that read invented. Keep the ones that make you wince because you've seen that exact bad input in support tickets.

5. Freeze it and version it

Once you're happy, freeze the set and give it a version. The eval dataset is an asset that outlives any scorer or model. Treat changes to it like schema changes, with a note on what moved and why.

Prompt-craft patterns for dataset curation

Label behavior, never exact strings. A case whose expected field reads "returns valid JSON with all three keys" stays scorable across model versions. One that reads "returns exactly {"status":"ok"}" breaks the day the model adds a trailing field. This is the single most common reason eval sets rot.

For the expected-behavior field, describe what a correct
answer must DO, not the literal text it must MATCH.
"Cites at least one source from the context" survives.
"Says the following sentence" does not.

Dedupe on intent, not characters. Two inputs that differ only in a name are the same test. Tell the prompt to collapse them, or your coverage numbers lie. Real distribution matters more than row count.

Watch the model's labeling bias. Claude tends to be conservative when tagging difficulty and will mark borderline cases "edge"; GPT-4o leans toward "typical" and under-counts the hard ones. When the tags look too clean, that's usually the model smoothing, not your traffic being easy. Spot-check the adversarial bucket by hand.

Variables you'll set

Getting started

1. Export fifty to a few hundred real transcripts and redact secrets.
2. Write {{job_description}} as one concrete behavior sentence.
3. Pick four difficulty tags you'll genuinely sort by.
4. Run the prompt and read every row for whether a user would send it.
5. Cut invented-looking rows; hunt for more edge cases by hand.
6. Freeze the set and give it a version number.
7. Hand it to a scorer. The free Eval Question Generator seeds the gap-filling cases when your logs are thin on edges and adversarial inputs.

Curation rarely stops at the set. Once you have cases, you need a way to score open-ended outputs, which a LLM Eval System Design playbook structures end-to-end, and a way to keep the set honest as prompts change.

A dataset is only half the loop. The other half is the harness that runs it, covered in how to build an LLM eval harness with a prompt, and the discipline of catching silent drift, which is what prompt regression testing is for.