Wiring an Agent Eval Suite Into CI

Putting an agent eval suite into CI means every pull request runs a regression check before a human ever looks at the diff. Here's the part teams get wrong: they treat this like a research experiment they run when curiosity strikes, rather than a build step with a fixed budget and a fixed threshold. I've watched three or four teams stand these systems up now, and the failures cluster in the same three places every time: arbitrary gating thresholds, judge noise nobody accounted for, and suites that grow until nobody waits for them anymore. ### Why this is harder than unit testing A function returns 4 or it doesn't. Nobody argues about it, nobody reruns it hoping for a different answer. Agent behavior doesn't offer that luxury. Ask the same agent the same question twice at temperature 0.7 and you'll sometimes get two different tool-call sequences that both solve the task, and occasionally a third that invents a file path that never existed. You're gating a merge on a signal that has variance built into it on purpose, which is a strange thing to build a CI pipeline around if you're used to thinking in asserts. Anthropic and OpenAI have both put out guidance, in their respective agent-building and evals documentation, arguing that eval suites should be read as statistical samples rather than assertions. A single transcript tells you almost nothing about the underlying success rate; it's one draw from a distribution you can't see directly. Once that reframe lands, the whole design of the CI check changes. The question stops being "did it pass" and becomes "did the pass rate move," which sounds like a small linguistic shift and turns out to change almost every downstream decision. ### Picking a threshold that survives contact with reality Every team starts the same way: 95% pass rate on the golden set, block the merge below that. Every team eventually walks the number back down, usually within a month, because it was chosen before anyone had baseline data to justify it. Skip that step. Run the suite against main for a week or two before gating anything on it, log the actual pass rate distribution you get, and set the threshold from the variance you observe rather than a round number that sounded rigorous in a planning meeting. A pattern that holds up better: gate on regression from baseline, not an absolute floor. Say main sits at 91%. A PR that drops the same eval set to 84% is worth blocking regardless of what the absolute number implies about quality in some abstract sense. But if main naturally drifts between 88% and 93% week to week purely from judge noise, a fixed floor of 90% will flip red and green on PRs that touched nothing relevant, and engineers will start merging past the check out of habit within a few weeks. That habit is close to impossible to walk back once it sets in; I've seen a team spend two sprints just trying to re-earn trust in a check everyone had quietly learned to ignore. Severity tiers do real work here too, and they get skipped constantly because they're an extra decision nobody wants to make on day one. A regression involving destructive actions, an agent deleting files or sending an email it had no business sending, should hard-block every single time. No threshold math, no delta calculation, just stop the merge. A regression in tone or verbosity on a summarization task can sit in a warning tier a reviewer glances at without being forced to resolve it. Collapsing every failure into one pass/fail gate is probably the single most common mistake in early agent CI setups, and it's the fastest route to a team that stops trusting the suite altogether. ### The judge is the noisiest part of the system LLM-as-judge is the standard approach for scoring open-ended agent output, and it's also where most of the flakiness lives. Ask the same judge model to grade the same transcript twice and you won't always get the same score back. That's not a bug you patch out eventually; it's inherent to using a generative model as a grader in the first place. A few things actually help. Discrete rubric categories beat continuous scales, full stop: "did the agent complete the task, yes, partial, no" produces meaningfully more consistent scoring than asking for a number from 1 to 10, because a continuous scale invites the judge to split hairs that don't track any real difference in output quality. Anthropic's own eval documentation recommends binary or low-cardinality rubrics for this exact reason, and it's one of the few pieces of judge guidance I've seen hold up consistently across different agent domains. Majority voting helps too, but only if you're selective about when you pay for it. Running the judge three times and taking the majority score on every transcript triples your judge cost for almost no benefit, since most transcripts are cleanly good or cleanly bad on the first pass. Run once. Only trigger a second and third pass when the first score lands inside a narrow band around your pass/fail line. The judge only needs backup on the ambiguous middle, and that middle is usually a small fraction of total volume. Pin the judge model version, and mean it. Swapping GPT-4o for a newer snapshot mid-quarter, or moving between Claude versions without a re-baseline, will shift your pass rate for reasons that have nothing to do with any change in your agent's code. Treat the judge like a pinned dependency, version it in your config, and change it deliberately with a re-baselining pass rather than letting an API default silently move underneath you. Same-family judging is worth avoiding when the budget allows for it. There's a subtle self-preference effect where a model tends to rate outputs from its own family a bit more favorably, which is exactly the kind of bias you don't want sitting quietly inside a gate that's supposed to be neutral. Cross-family judging costs more, sure, but the signal comes out cleaner, and clean signal is the entire point of building the gate in the first place. ### The ten-minute ceiling Ten minutes is a real number, not a nice-to-have, and it shapes nearly every other decision downstream of it. Past that window, engineers start merging on stale runs, or they stop looking at the check entirely. So the suite has to be fast, and fast means it cannot be your full eval corpus. That's the whole constraint, stated plainly. Split into tiers. A small smoke set, something like 20 to 40 scenarios covering core agent behaviors, runs on every PR inside the ten-minute window. A larger regression set, hundreds of scenarios covering edge cases and long-tail tool interactions, runs nightly or on merge to main. Nothing novel here: this is the same shape as fast unit tests on every commit paired with slower integration suites on a schedule. Agent evals need the same tiering, just applied to a scoring mechanism noisier than an assert statement. Parallelize hard. Each scenario is independent, so 30 scenarios across 30 workers takes roughly the same wall-clock time as one scenario running alone. The bottleneck is almost never compute; it's API rate limits on whatever model powers the agent and the judge. Batch calls, respect whatever rate-limit headroom you've negotiated with your provider, and cache anything deterministic (tool outputs from mocked environments, mainly) so you're not re-fetching identical fixture data every single run. And prune. A scenario that's passed 500 times in a row across two months of PRs isn't telling you anything anymore; it's dead weight sitting inside a ten-minute budget that has no slack in it. Audit the smoke set periodically, retire or demote whatever never fails, and use the freed-up slot for something that actually correlates with a bug you shipped recently. ### What this buys, and what it doesn't An eval suite in CI catches regressions before a customer finds the agent's new failure mode in production, which is worth a great deal on its own. It does not replace human review of agent behavior, and treating a green check as full sign-off is a failure mode of its own, just a quieter one. A judge can be wrong on any single transcript even while being right on average across a hundred of them; averages hide exactly the kind of individual miss that matters most in production. Keep a person spot-checking a sample of passing runs, not only the failing ones. A suite that only surfaces its own failures will drift, slowly and without anyone noticing, toward whatever the judge happens to reward that quarter. None of this is exotic. It's the same discipline that made deterministic CI trustworthy two decades ago, bent to fit a system that refuses to be fully deterministic. The teams doing this well aren't running some especially clever framework; they treated threshold-setting, judge reliability, and runtime budget as three separate problems and solved each on its own terms, instead of hoping one tidy rubric would somehow cover all three at once.