Detect PII and Secret Leaks in AI Agent Output With One Prompt

An AI agent that reads a support ticket, calls three tools, and writes a summary touches personal data at every hop. Somewhere in that chain it pastes a customer's Social Security number into a log line, or echoes an API key a tool returned, or writes an email address into a vector store as plain text. Nobody notices until an audit does. If you need to detect PII leaks in AI agent output before it reaches a user or a log, the fix isn't another dashboard. It's a scan step the agent runs on its own output.

Most of the tooling being sold for this is a library you pip install and wire into one runtime. That's fine if you run one Python service. It doesn't help the team whose agents span a Node gateway, a Claude tool call, and a Zapier step. A prompt does, because it runs wherever the model runs.

This walks through a reusable leak-scan prompt: what it checks, how the output contract locks its verdicts into a table you can act on, and where it beats a regex-only scanner.

Why agents leak data by default

An agent treats every string the same. A name, a card number, and a boilerplate greeting all flow through the same LLM call, the same trace log, and the same retrieval index. There's no built-in notion that one of those is a liability. The AIR Blackbox writeup on agent PII leaks makes the same point: agents leak by default because they don't classify data before moving it, and redaction has to be added as its own step.

The leaks cluster into a few shapes. A DEV Community post on what agents should never leak lists the usual suspects: PII in tool-call arguments, API keys sitting in agent context, financial data buried in a reasoning chain, and medical or legal text the agent wasn't supposed to repeat. Both pages then point you at their own SDK. Neither hands you a model-agnostic check you can paste into whatever stack you already run.

That's the gap. The detection logic is simple enough to express as a prompt. What's missing everywhere is the prompt.

What you can do with a leak-scan prompt

• Scan a single agent response before it renders to a user
• Gate tool-call arguments so an agent can't send an SSN to an external API
• Sweep a batch of trace logs to find historical leaks after the fact
• Flag high-entropy strings and known key prefixes as secrets, not just names
• Produce a redaction plan (mask, hash, tokenize, or drop) per finding
• Feed a pass/fail signal into a CI gate for agent changes

Anatomy of the scan prompt

The whole thing is a role, a triple-fenced input, and an output contract that forces a table instead of a paragraph. Structure matters here: a scanner that answers in prose is useless downstream because you can't diff it or route on it.

Role: You are a data-leak scanner for AI agent output.

Input:
"""
{{agent_output}}
"""
Context: {{data_policy}}   # what counts as sensitive here

Output contract (return this exact shape, no preamble):
| span | category | severity | why | redaction |
|------|----------|----------|-----|-----------|
- category ∈ {PII, SECRET, FINANCIAL, HEALTH, CLEAN}
- severity ∈ {high, medium, low}
- redaction ∈ {mask, hash, tokenize, drop, none}
Final line: VERDICT: PASS or VERDICT: FAIL

Feed it the agent's raw output in {{agent_output}} and your definition of sensitive in {{data_policy}}. It returns one row per finding, ending with a machine-readable verdict a pipeline can branch on. No "Here's what turned up" wrapper, because the last learning most teams hit is that a prose preamble breaks the parser.

How the models differ on this job

Behavior isn't uniform, so don't write the prompt once and assume it ports. Claude honors a ## Output format heading and the VERDICT: final line reliably across long inputs, which matters because agent outputs you're scanning can be long. GPT-4o drifts on long inputs unless the table schema is restated on the final line of the prompt, right before the pasted output ends, so the contract stays in its recent-token window. Gemini tends to over-flag, marking generic first names as high-severity PII; tightening {{data_policy}} with an explicit allowlist of non-sensitive tokens calms it down.

None of the vendor pages show how to detect PII leaks in AI agent output across models. Both the AIR Blackbox and DEV posts treat detection as a single library call with no model-by-model story, because their answer is their SDK, not a prompt you'd run on three different models.

Prompt-craft patterns that make it hold

Put the contract last. Models weight recent tokens. A scan prompt that opens with the table spec and then pastes 4,000 tokens of agent output will forget the spec. Paste the output first, restate the table shape immediately after, and the verdict format stays locked.

Separate detection from redaction. Ask the model to name the leak and propose an action, but don't ask it to rewrite the text in the same pass. A model that redacts inline will quietly paraphrase clean sentences too, and now you can't tell what changed. Two prompts, or two clearly-fenced sections, keep the audit clean.

Define "clean" explicitly. If CLEAN isn't in the category set, the model invents a leak to justify its existence. Give it a way to say nothing's wrong, or every scan returns a false positive.

Variables you'll set

Getting started

1. Grab the last 20 responses your agent produced and paste one into {{agent_output}}.
2. Write a two-line {{data_policy}}: what's sensitive, what's explicitly fine.
3. Run the scan on Claude and on GPT-4o; compare the tables.
4. Wire the VERDICT: FAIL line into your pre-render or pre-log step.
5. Keep a deterministic regex pass in front for card and account numbers.
6. Re-scan a batch of old logs to find leaks you've already shipped.

Once the scan runs clean on live output, the same prompt doubles as a gate on tool-call arguments, which is where the worst leaks happen. An agent that emails a summary is one bad tool call away from mailing a customer's data to an external endpoint. This is a cousin of two jobs the catalog already covers: the Agent Commit Security Harness checks what an agent tries to commit, and the scan here checks what it tries to say.

A leak that a model catches on the way out is cheap. The same leak found in a breach notice isn't. If you're hardening agents, pair this with the runtime side of the problem in prompt injection defense for AI agents, and with the source-side check in scanning commits for secrets. For the connected-tool version, the Prompt Injection Defense Harness covers the input half of the same threat.