A Prompt to A/B Test Prompts and Pick the Real Winner

Two prompts, and a gut feeling that the second one's better. That's how most prompt changes ship: someone tweaks the wording, eyeballs a couple of outputs, and calls it an improvement. Then quality quietly drops on the cases nobody checked. To a/b test prompts properly, you run both against the same cases and let the scores decide.

This prompt is the judge. It takes variant A, variant B, and a fixed set of test cases, runs the comparison, and scores each output against your metric. The result is a win, lose, or tie verdict with the per-case counts, so a two-point lead on a noisy sample gets called what it is: a tie.

The discipline that makes it work is isolating one change. If B differs from A in three ways, a better score can't tell you which one mattered.

Why most A/B testing guides sell a dashboard

The pages that rank walk you toward a platform. Maxim's A/B testing guide covers the principles, then positions its experimentation product as the answer and ships no copyable comparison prompt. PromptLayer's post on A/B testing prompts is a tool pitch. And Braintrust's practical guide couples the workflow to its own SDK.

So the principles are clear and the prompt is missing. That's the gap. A self-contained judge that scores A against B on your cases, in any model, beats a dashboard you have to instrument your whole stack to use.

Here's the take: most prompt "improvements" are ties, and admitting that is the skill. Models are stochastic, so two prompts will produce different outputs even when neither is better. If your test can't distinguish a real two-point gap from run-to-run noise, you're not measuring quality, you're measuring luck. The honest verdict is "tie" far more often than people want to hear.

What you can do with this prompt

• Score two prompt variants on the same fixed set of test cases.
• Enforce a single changed variable so the result is attributable.
• Get a per-case win, lose, or tie rather than one blended average.
• See the score spread, so noise doesn't masquerade as a winner.
• Surface the cases where the loser actually won, which reveal trade-offs.
• Re-run the comparison on a new model to check the winner still holds.

Anatomy of the A/B comparison prompt

The prompt takes both variants and the shared cases, then emits a scored comparison with an honest verdict.

Variables:
  {{variant_a}}    – prompt version A
  {{variant_b}}    – prompt version B (one change from A)
  {{test_cases}}   – the same inputs both run against
  {{metric}}       – what "better" means, defined

Prompt:
  Role: experiment judge comparing two prompt versions.
  For each test case, score A's output and B's output on
  {{metric}}, then declare per-case winner or tie. Note the
  single change between variants; refuse to compare if A and
  B differ in more than one dimension.

Output contract (restate on the final line):
  Per case: score_a, score_b, winner (a | b | tie)
  Overall: tally, verdict, and a noise caveat if the
           margin is within run-to-run variance.

The refusal clause matters. If you paste two variants that differ in five ways, the prompt should call that out rather than hand you a meaningless winner.

Step-by-step usage

1. Change exactly one thing

Make {{variant_b}} differ from {{variant_a}} in one dimension: the role line, or the output format, or one example. Not all three. This is the rule everyone breaks and the reason most A/B results are uninterpretable.

2. Fix the test cases

{{test_cases}} is the same set both variants run against, drawn from real usage. Reusing your eval dataset here is the obvious win. Different cases for each variant isn't an A/B test, it's two unrelated runs.

3. Define the metric concretely

{{metric}} says what better means: "follows the JSON contract and includes all required fields" beats "higher quality." A vague metric lets the judge invent its own standard, and then you can't reproduce the result.

4. Read the per-case verdicts, not just the tally

The overall winner hides the trade-off. Variant B might win eight cases and lose two badly, and those two losses could be your most important inputs. Scan where the loser won; that's where the real decision lives.

5. Respect the noise caveat

If the prompt flags the margin as within run-to-run variance, believe it. Re-run with more cases or accept the tie. Shipping a "winner" that's actually noise is how prompt quality drifts sideways for months.

Prompt-craft patterns for prompt comparison

One variable, always. The whole logic of an A/B test is attribution, and attribution dies the moment you change two things. If you genuinely want to test a new role and a new format, run two sequential tests, not one combined one. The combined test feels efficient and teaches you nothing.

Before scoring, diff {{variant_a}} and {{variant_b}}.
If they differ in more than one dimension, STOP and report
the dimensions. A multi-change test has no attributable result.

Report spread, not just mean. A mean hides whether B won by a mile on every case or squeaked past on average while losing badly on a few. Have the judge report the distribution so a fragile winner can't hide behind a tidy average.

Watch for judge bias by position and model. An LLM judge tends to favor the first output it sees, so randomize which variant is labeled A per case. Models also differ: Claude is comparatively willing to call a genuine tie, while GPT-4o more often manufactures a winner when asked to pick, so force the tie option explicitly and restate it on the final line. Pin the judge model, because switching it mid-experiment changes the scale under you.

Variables you'll set

Getting started

1. Make variant B differ from A in exactly one dimension.
2. Fix a shared {{test_cases}} set from real usage.
3. Define {{metric}} as a checkable behavior, not "quality."
4. Run the comparison and read per-case verdicts, not just the tally.
5. Believe the noise caveat; re-run or accept the tie.
6. Ship the winner only if the margin clears variance.
7. Make it repeatable. The LLM Eval System Design playbook structures the judge prompt, the metric, and the tie logic so comparisons stay consistent across experiments.

An A/B test needs a set of cases to run on, and a harness to run them. The Agent Eval Harness Builder playbook designs that set and the runner so your comparisons aren't improvised each time.

Prompt A/B testing leans on two neighbors: building an eval dataset from logs for the shared cases both variants run against, and prompt regression testing to make sure today's winner doesn't silently regress on the next edit.