An Alert Tuning Prompt That Kills the 3 a.m. Noise

Every on-call engineer has muted a channel at 2 a.m. and then missed the one page that mattered. That's alert fatigue, and it's not a discipline problem. It's a tuning problem. The fix is boring and repeatable: audit the rules, find the ones that never lead to action, and cut or reshape them. An alert tuning prompt makes that audit something you run in an afternoon instead of dread once a quarter.

The prompt that works takes your alert rules and their firing history, then returns a verdict for each one: keep, tune, or delete, with a reason you can argue with. Not advice. Verdicts. That structure is what the best-practice blogs never give you.

What an alert tuning prompt produces

An alert tuning prompt is a reusable AI prompt that ingests a list of alert rules plus how often each fired and how often the firing was actionable, then emits a per-rule keep/tune/delete verdict with reasoning. The locked output is a table you act on, not prose you nod along to.

The advice is everywhere; the tooling isn't. Better Stack's alert-fatigue guide lays out evaluation windows, group_by deduplication, and inhibition clearly, but it's prose you apply by hand. The Sherlocks on-call playbook gives sharp numbers, delete anything unused for 90 days, aim for 30 to 50% of alerts requiring human action, under five actionable pages per engineer weekly, but no prompt to apply them at scale. You're left auditing rules in a spreadsheet.

What you can do with this prompt

• Run a full alert-rule audit and get a keep/tune/delete verdict for every rule.
• Flag rules that haven't fired actionably in 90 days for deletion.
• Convert static-threshold alerts into SLO burn-rate alerts where it makes sense.
• Recommend evaluation windows and grouping to collapse duplicate pages.
• Produce a before/after page-volume estimate so you can justify the cuts.

Anatomy of the prompt pack

Variables
  {{alert_rules}}      – rule name, condition, current threshold
  {{fire_history}}     – per rule: times fired + times actionable (90d)
  {{slo_targets}}      – relevant SLOs, so alerts can tie to burn rate
  {{paging_policy}}    – what severity pages a human vs logs only
  {{protected_rules}}  – rules to keep regardless (safety/compliance)

Prompt
  Role: SRE auditing alert rules to cut noise without cutting signal.
  Task: classify each rule, recommend a change, estimate noise reduction.
  Output contract (locked):
    1. Per-rule verdict – KEEP / TUNE / DELETE + one-line reason
    2. Tuning detail     – for TUNE: new threshold, window, or grouping
    3. Burn-rate moves   – which static rules to convert to SLO-based
    4. Protected         – rules left untouched and why
    5. Estimated impact  – projected page-volume change

The fire_history variable is the whole game. Without "fired 40 times, actionable 2 times" per rule, the model is guessing which alerts are noise. With it, the 90-day rule and the actionability target turn into a defensible verdict.

How models behave on an alert audit

Claude is the most disciplined at producing one verdict per rule and not collapsing similar rules into a summary, which matters when you have eighty rules and need eighty decisions. GPT-4o reasons well about which static thresholds should become burn-rate alerts but tends to soften "DELETE" into "consider deleting" unless you instruct it to commit to a verdict. Gemini is quickest at spotting duplicate rules that should be grouped but needs the actionability target stated as a number, not a description.

Feed all three the same fire_history and the verdicts converge. Feed them vibes and they diverge wildly. The data is the difference.

Prompt-craft patterns

Pattern 1 — verdict, then reason. Force a decision before the explanation so you can skim the table.

For each rule output exactly one of KEEP, TUNE, or DELETE on the
first line, then one sentence of reasoning. Do not hedge the verdict.

Pattern 2 — the 90-day rule as a hard input. Make the actionability threshold explicit.

Recommend DELETE for any rule in {{fire_history}} that fired but
required human action 0 times in 90 days, unless it appears in
{{protected_rules}}.

Pattern 3 — burn-rate over static thresholds. Tie noise to the SLO, not the metric.

Where {{slo_targets}} exists, prefer converting a static-threshold
rule to a fast-burn + slow-burn alert on the error budget. Show the
two thresholds.

The stance worth holding: most "on-call is broken" complaints are alerting failures wearing a roster costume. Fix the alerts and the rotation stops feeling brutal. Adding engineers to a noisy rotation just spreads the misery; tuning the rules removes it.

Variables you'll set

Getting started

1. Export your alert rules and pull 90 days of firing + actionability data.
2. Mark any safety or compliance rules as {{protected_rules}}.
3. Run the prompt on Claude for one clean verdict per rule.
4. Apply the easy DELETEs first, then the TUNE changes, then burn-rate conversions.
5. Watch page volume for two weeks against the estimated impact.
6. Re-run quarterly; noise creeps back as services change.

The tuning thresholds you set here come straight from your SLOs, which is why this pairs with the SLO definition prompt for computing the burn rates your alerts should fire on. And the rules you keep need somewhere to point, which is the runbook authoring prompt, because an alert with no runbook is just a notification.

In the catalog, the Engineering On-Call Coordination Agent Pack covers triage and paging, and the Observability Instrumentation Harness generates the alert stubs you'll be tuning in the first place.

Cut the noise, keep the signal, and link every survivor to a runbook. Quiet pagers aren't the goal. Trustworthy ones are.