Evaluating Agentic Review Effectiveness
by Dorothy Cheng, Software Engineer
We re-ran our AI code reviewer against 98 of our own pull requests on nine different models and had a judging panel grade every finding. On our code, the most expensive model lost. Here is what that taught us about routing models, and why our layered review setup came out of it stronger.
At Violet, we had five AI review automations commenting on every PR, each in a different role and mostly on different models. And ultimately, we had no real idea which of them were actually catching bugs.
So we went back through five weeks of output, measured it against 729 merged PRs, and checked which models found real issues in our code and whether the way we layered the reviewers held up.
The layered review model
Our pipeline is not one big reviewer. It’s several narrow ones, each with a different job and, deliberately, a different model behind it:
- A rules worker (Claude Opus 5) that routes changed files to domain-specific rule packs and posts inline findings. This is the workhorse.
- An adversarial worker (Kimi K3) on a different model family, that hunts for correctness defects that the rulebook misses.
- A synthesizer (GPT-5.6 Sol) that dedupes everything the workers found into one summary thread for the human reviewer.
- A general-purpose PR review automation (a Claude Fable 5 + GPT-5.6 Sol blend) that posts severity-labeled findings.
- A complexity bot (GPT-5.5) that flags Big-O and performance risk.
Our reasoning when we set it up: a rulebook pass is cheap and consistent but only finds what it is told to look for; an adversarial pass on a different model family catches what the rulebook shares blind spots with; a synthesizer keeps the human from reading all the bots. We built it on intuition. The question was whether the data agreed.
How we measured it
The easy metric is how many comments the bot posted. But, it’s also not very helpful. A reviewer that posts twenty findings nobody acts on is worse than one that posts two that get fixed.
So we built the harder pipeline:
- Harvest. Every comment, thread, and label from 729 merged PRs, classified by source and anchored to the code it referenced.
- Clustering. Repeated mentions of the same defect collapsed into one issue, so a finding echoed by three bots and a human counts once. That gave us 1,072 distinct issues.
- Acted-upon rate. For each finding: was the thread resolved, did a human explicitly respond, or did the anchored code actually change before merge?
- Replay matrix. We took the rules worker's exact production prompt and re-ran it against 98 historical PRs across nine models, then had a two-model judging panel grade every finding as a match to a real production finding, a valid new catch, an invalid one, or a disagreement. More than 1,500 runs in total. Costs below are provider billing credits, a relative measure rather than dollars.
- Outcome backtest. For our AI-assigned risk labels, we traced each PR forward: reverted, referenced by an explicit fix, or patched in the same lines within 30 days? Only verified linkage counted, never self-reported "fix" titles.
More expensive doesn’t always mean better
The replay matrix produced a result we didn't expect. On our repo and our prompt, Claude Fable 5, the most expensive model in the grid and our baseline, found about 0.73 valid issues per PR at 170 credits a run. Claude Opus 5, the model already running in production, found 1.16 per PR at 103 credits. The cheaper models (Claude Sonnet 5, GPT-5.3-Codex, Grok 4.6, Kimi K3) produced very few valid findings on this prompt.
The interesting result was in the middle. GPT-5.6 Sol matched Fable's valid-findings rate within noise (about 0.7 per PR) at roughly a fifth of the cost, with a higher share of its findings landing as agreed-valid rather than disputed. Per valid finding: Sol about 49 credits, Opus 89, Fable 233.
GPT-6 Astra trailed Opus by about 0.7 valid findings per PR but found 29 more issues Opus did not. That makes it a poor replacement for Opus but a useful second reader.
So we stopped ranking models on raw quality and started comparing valid findings per credit, plus what each one caught that the others missed.
The layers earned their keep
The acted-upon data backed the layered design more clearly than we expected.
The rules worker posted the most findings and acted on them 85% of the time. It's the right thing to run on every PR.
The adversarial worker fired rarely, about five findings across five weeks, but every single one was acted on. That's what we wanted from a second-opinion layer: low volume, high hit rate. It runs on a different model family on purpose, so it is a genuine second opinion rather than the same model reading twice.
The synthesizer, once deduped, had zero original findings. Every one of its inline posts repeated a worker finding. That's expected. It isn't meant to find anything. It's there so reviewers read one thread instead of five. We kept it for the workflow, not the catches.
The other two reviewers did not survive the same test. The general-purpose PR review automation landed at 74% acted-upon with real volume, so the quality was fine. The problem was overlap: on PRs where it and the rules worker both ran, they re-found roughly half of the same issues. Once our domain rule packs gave the rules worker full coverage, running both on every PR was paying twice for one review.
The complexity bot posted findings that were sometimes technically accurate and never actionable. Its five findings drew zero action; the one human reply was a reasoned "accurate mechanics, ship as is." It's been taken out of the default rotation.
Could a Low-risk label skip the human?
Every PR also gets an AI-assigned Low, Medium, or High risk label. Once you trust the reviewers, the obvious next step is letting Low-labeled PRs merge on their own. We backtested it before anyone proposed it.
The label is a decent classifier. A manual re-review of 48 Low-labeled PRs agreed with 46 of them. The label is also not a merge gate: of these Low-labeled PRs, 27 are already linked to a verified post-merge outcome within 30 days, with most follow-up windows still open, meaning a revert, an explicit fix PR, or a follow-up that patched the same lines. In practice, Low meant a PR probably wouldn't break anything major. It didn't mean the PR wouldn't need a follow-up fix.
We also sliced by codebase area, looking for any domain with at least 20 PRs and zero verified escapes. There were none. Every part of the repo with enough volume to judge had at least one. Human review stays on every PR, in every domain, and the risk label goes back to being what it’s good at: telling reviewers where to dig deeper.
What we would tell another team
Start by benchmarking on your own PRs. The ranking we got from 98 replays of our own history didn't match what the price would have predicted. Opus beat Fable on our code at a lower cost.
Measure cost per valid finding. Cost per run hides the real difference: the roughly 5x price gap between Sol and Fable works out to about 49 credits versus 233 for similar output. That's the number we'll use for routing decisions going forward.
When adding a second model, look for one that catches different things. Astra lost to Opus head to head and still found 29 issues that Opus never did.
Finally, tag your bots. If every automated comment carries a machine-readable marker (one line of HTML is enough), attribution is exact instead of a judging exercise.
Why this matters for hardware teams
Violet syncs engineering and manufacturing data between systems that were never designed to talk to each other. If our sync code is wrong, a purchase order could go out with the wrong quantity, or a part revision might not match what's actually on the floor. The review bar for that code has to be evidence-based, and "the AI approved it" only counts as evidence once you've measured what that approval is worth, and which AI gave it.
Reach out or book a demo here.
Abolish silos.
