Context
Review capacity is the bottleneck on a platform team serving 250+ engineers. Every pull request needs someone to check it does what the ticket said, and that nothing sensitive slipped into the diff. Both are mechanical checks that consume senior attention.
The problem
The obvious move, putting an LLM on pull requests, fails in a specific way: a model that always produces a confident review is worse than no review. Reviewers learn to ignore it, or worse, trust it. The hard requirement was not accuracy on average, it was knowing when the model could not verify something.
- step: Pull requestopened
- step: GitHub Actiontriggers review
- data: Jiraacceptance criteria
- step: Claude Codereviews diff vs criteria
- control: Verdictpass · fail · unknown
- result: Human reviewersees the summary
Approach
- 01
Ground the review in the ticket, not the diff alone
The reviewer pulls the linked Jira issue and checks the change against its acceptance criteria. Without that anchor the model reviews style; with it, the model reviews intent.
- 02
Make Unknown a first-class verdict
When the change cannot be verified against the criteria, the reviewer returns Unknown rather than guessing. This is the design decision the whole thing rests on. It keeps human attention on the cases that need it, and stops the tool becoming background noise.
- 03
Scan for exposed credentials on the same pass
Secrets in a diff are cheap for a model to spot and expensive for a human to miss. Folding it into the same review means no extra pipeline stage.
- 04
Ship it as a GitHub Action, not a service
Running as an Action meant adoption per repository was a file, not an onboarding. That is what took it from 30 repositories to 70+.