A Flaky Test Detection Prompt for Your CI Pipeline

A test goes red, you re-run it, it goes green, and you merge. That reflex is how flaky tests survive for months while quietly training the team to ignore the pipeline. A flaky test detection prompt breaks the reflex: feed it the failure history, and it tells you whether this is noise or a real regression hiding behind intermittent behavior.

The signal isn't in a single failed run. It's in the pattern. A test that fails one run in five with no code change is flaky. A test that started failing after a specific commit is a regression wearing a flake costume. The prompt reads history, classifies, and hands you a quarantine decision with a structured contract you can route to a tracker.

The write-ups that rank here are in-house tooling stories from large engineering orgs. Impressive, and useless if you don't have their infrastructure. None hand you a copyable prompt. That's the gap.

What you can hand the detector

A flaky test detection prompt is a reusable instruction that takes failure history plus a failing log and returns a classification with a quarantine decision. History is the input that makes it accurate. The contract makes the output actionable.

Useful inputs:

• A pass/fail history table for one test across recent CI runs
• The failing log alongside a known-good run of the same test
• A suite where three tests fail intermittently and you can't tell which are real
• A test that only fails when run in parallel with others
• A failure that correlates with a specific runner or time of day

The job each shares: separate nondeterministic noise from a genuine break, then decide what to do about it. That decision is worth packaging.

Anatomy: history in, quarantine call out

The prompt frames the model as a test-reliability engineer, takes the history and log in variables, and locks the output to a classification plus a decision.

Variables
  {{failure_history}}  — recent pass/fail runs for this test
  {{failing_log}}      — the failure output
  {{recent_changes}}   — commits touching this test or its deps

Prompt
  Role: You are triaging test reliability for the CI pipeline.
  Task: Use {{failure_history}} to decide if this is flaky or
        a real regression. Rank likely root causes. Decide
        quarantine vs fix.

Output contract
  classification:  flaky | real_regression | inconclusive
  root_causes:     ranked hypotheses (timing race, shared state,
                   order dependency, resource limit)
  decision:        quarantine | fix_now | gather_more_data
  rationale:       tied to the history pattern

The inconclusive and gather_more_data options matter. A prompt forced to pick flaky-or-real on thin evidence guesses. Giving it an honest "not enough signal yet" exit produces better calls than false confidence.

Step-by-step: from red test to decision

1. Pull the failure history

Most CI systems expose per-test history or you can grep recent runs. A simple run/result table is enough.

2. Add the failing log and recent changes

{{failing_log}} gives the model the error shape; {{recent_changes}} lets it rule out a regression.

3. Run and read the classification

flaky, real_regression, or inconclusive. Trust inconclusive — it's the model declining to guess.

4. Act on the decision

quarantine means skip-and-track so it stops blocking. fix_now means the flake hides a real bug. gather_more_data means run it 20 more times first.

5. Track the quarantine

A quarantined test that's never tracked becomes a permanently dead test. Open an issue when you quarantine.

Patterns that keep classification honest

Demand the history pattern in the rationale. Force the model to tie its call to the actual pass/fail sequence. A rationale that references "3 failures in 20 runs, no code change" is checkable. One that says "this looks flaky" isn't.

Rank root causes, don't pick one. Flakiness usually has competing causes — a timing race and shared state can both be live. Ranked hypotheses point your investigation; a single guess sends you down one hole.

Quarantine is a decision, not a default. Make the model justify quarantine versus fix. Quarantining a test that hides a real concurrency bug ships the bug. The decision field forces that consideration.

Variables you'll set

An opinion worth holding

Auto-retrying a failing test until it passes is the worst flaky-test "strategy" there is, and it's everywhere. It doesn't fix flakiness; it hides it and ships whatever real bug the flake was masking. Quarantine-and-track is the honest move: pull the unreliable test out of the blocking path, but open an issue so someone actually root-causes it. A retry loop with no tracking is just deciding to be wrong intermittently and never find out.

Getting started

1. Copy the anatomy into your model of choice.
2. Pull a pass/fail history for one suspected-flaky test.
3. Add the failing log and any recent relevant commits.
4. Check whether the classification matches your gut, then trust the evidence over the gut.
5. Open a tracking issue every time you quarantine.
6. Re-run the prompt after a fix to confirm the test stabilized.

To run this across the whole pipeline automatically, the Flaky Test Detection Harness Agent Pack surfaces nondeterministic failures from CI history, ranks root-cause hypotheses, and creates tracked fix issues in Jira.

When a flake turns out to be a real failure, route it through the CI Failure Diagnosis Harness Agent Pack to get the file-specific fix. For more on packaging reusable prompts, read how to choose a reusable AI prompt pack and the companion CI failure diagnosis prompt from pipeline logs.