A Mutation Testing Prompt That Writes Tests to Kill Survivors

A suite at 90% line coverage feels safe. Then you flip a > to a >= in the code, run the tests, and they all still pass. That mutant survived. Your coverage number measured lines executed, not behavior checked, and the gap between the two is where the real bugs live.

A mutation testing prompt closes that gap without a heavyweight tool. It asks the model to introduce small faults, check whether your current tests would catch each one, and write tests for the ones that slip through. The output isn't a coverage percentage. It's a list of survivors and the tests that kill them.

This is the prompt, the kill-report contract, and where it beats both the research papers and the vendor tooling.

Why coverage lies and mutation testing doesn't

Mutation testing introduces deliberate, tiny bugs, a flipped operator, a removed line, a swapped boundary, then runs your suite. A mutant that makes a test fail is "killed." One that doesn't is a "survivor," and every survivor is a behavior your tests don't actually pin down.

The serious work here lives in two places, and neither hands you a copyable prompt. Meta's Automated Compliance Hardening and Atlassian's mutation-coverage assistant both run inside proprietary systems. Atlassian's is wired into their Rovo Dev CLI: it reads mutation reports, recommends a class to target, generates tests, and re-runs the mutants to confirm the kill. Real workflow, real results (56–84% baseline coverage climbing to 80–96%), but you can't paste it anywhere. The academic side, like the LLM mutation testing study, gives prompt structure with placeholders like MUT_NUMBER and CODE_ELEMENT, useful as theory, not as a thing a developer runs Monday morning.

The prompt below borrows the loop from both and makes it portable.

What you can do with this prompt

• Generate a set of representative mutants for one module, grouped by mutation type.
• Find which mutants your existing tests already catch, so you don't waste effort.
• Get a killing test written for each surviving mutant, in your framework.
• Re-check after adding the tests to confirm the survivors are dead.
• Harden a single high-risk file before a release without standing up a mutation framework.
• Teach a junior engineer what their tests don't check, concretely.

Anatomy of the prompt

The loop is generate, run, kill, re-check. The prompt encodes all four and ends with a strict output contract so the kill report comes back the same every time.

Role:    You are a test engineer running mutation testing on {{module}}.
Context: {{source_code}} and {{existing_tests}}.
Task:    1. Generate {{mutant_count}} mutants across these types: boundary
            (< to <=), arithmetic (+ to -), logical (&& to ||),
            removed statement, return-value swap.
         2. For each mutant, decide: would the existing tests catch it?
         3. List survivors. For each survivor, write a test that kills it.
Output:  A table — Mutant ID | Type | Code change | Caught? | Killing test.
         Then a fenced block with every killing test, ready to paste.

Step-by-step usage

1. Gather inputs

Pick one module and its existing tests. Mutation testing is expensive per line of reasoning, so scope it tight. The riskiest 200 lines beat the whole package.

2. Fill variables

Set {{module}}, {{source_code}}, {{existing_tests}}, and {{mutant_count}} (start around 8 to 12). More mutants means more reasoning and more drift.

3. Run the prompt

Read the "Caught?" column first. If everything's caught, raise the mutant difficulty: ask for boundary-off-by-one and inverted-conditional mutants explicitly.

4. Post-process

Add the killing tests. Then run them against the real code, not the mutant, to confirm they pass on correct behavior. A "killing test" that fails on correct code is just a broken test.

5. Iterate

Re-paste the updated suite and ask for a fresh mutant batch. Repeat until survivors drop to a level you're comfortable with. Rarely zero.

Prompt-craft patterns

Name the mutation operators. Don't say "introduce bugs." List the operator types (boundary, arithmetic, logical, statement deletion, return swap). Vague mutation instructions produce cosmetic mutants the model knows its tests will catch.

Generate mutants only of these types, two each:
- boundary: change a comparison operator (< -> <=, > -> >=)
- logical: flip a boolean operator (&& -> ||)
- statement: delete one non-trivial line

Separate "would it catch" from "write the kill." Make the model reason about detection before writing tests. Collapse the two and it writes a test for every mutant, including ones already caught, doubling your suite for nothing.

Validate kills against real code. Bake into the contract: "each killing test must pass on the unmutated source." Otherwise you get tests asserting the bug.

Model behavior is worth a line here too. Claude is steadier at the "would the existing tests catch this mutant?" reasoning step and tends to admit when it isn't sure; GPT-4o is more likely to declare a mutant caught without actually tracing the test, which inflates the kill rate. If a run reports an unusually clean result, ask the model to show its reasoning per mutant, the trace from input to assertion. The honest answer often flips a few "caught" cells back to "survivor."

The opinionated take: don't bother running mutation testing across an entire repo with an LLM. Per-mutant reasoning gets expensive and noisy at scale, and that's exactly what dedicated frameworks (Stryker, PIT, mutmut) exist for. The prompt's edge is targeted hardening, one risky module before a release, where standing up a framework isn't worth it and you want killing tests in ten minutes, not a CI job in an hour.

Variables you'll set

Getting started

1. Choose one module that would hurt if it broke quietly.
2. Paste the source and its existing tests.
3. Set {{mutant_count}} to 10 and name the operator types.
4. Run, then read the survivor list before anything else.
5. Add the killing tests and run them on the real code.
6. Re-run with a fresh mutant batch to confirm.
7. For repeatable runs across services, the Test Coverage Gap Harness reads coverage reports and points you at the modules worth mutating first.

Mutation testing tells you which tests are weak. Two follow-ups keep the rest of the suite honest: generating unit tests that assert real behavior so you start from a stronger base, and catching flaky tests so a killed mutant stays killed. If you want the broader scaffolding, the Polyglot Test Harness Scaffolder builds the suite these mutants test against.