Eval Engineer Career Toolkit

Run eval engineering for your agentic workflows as a documented, governed program — a coverage-designed golden set, audited graders, a regression gate policy with a real flake budget, and a drift runbook with re-baselining rules. (Operates the practice week over week; for one-time eval architecture of an LLM feature, see the LLM Eval System Design Playbook.)

A 4-step agentic workflow pack for coding built to run with ChatGPT, Claude, and Gemini. Open the Markdown files, fill the variables, and paste into your model. Most buyers get a reviewable result in about 15 minutes.

• Design a governed golden set for agent runs — scenario items with tool-failure injection, boundary probes, a sealed holdout, and an item lifecycle with owners and refresh rules
• Assign every quality dimension to the cheapest trustworthy grader — deterministic trajectory checks, behaviorally anchored rubrics, or bias-controlled LLM judges
• Put the graders themselves under audit: leniency drift, position bias, self-preference, verbosity bias, and rubric collapse each get a concrete control and cadence
• Ship a regression gate policy sized to change class — smoke, full, and holdout levels with trial policies for nondeterministic agents, absolute floors, relative-drop rules, and zero-tolerance items
• Enforce a flake policy with teeth: infrastructure-only retries, a quarantine list with expiry, and a degraded-gate condition that pauses upgrades
• Catch decay between releases with a four-family drift catalog (input, behavior, outcome, grader), duration-based alerts, and a triage tree ending in one of three re-baselining verdicts
• Keep the whole program alive with an operating calendar — weekly review, monthly grader audit, 6-week set refresh, quarterly holdout run