PR review agent: an AI code reviewer that only reports bugs it can prove
An AI code reviewer built on the Claude Agent SDK that reports a bug only after re-running the proof itself, and hands over a verified fix with it.
The public landing page Dashboard overview: a fix replayed until it is verified A finding and the trail that proves it The verified fix, changed line in green
Context
AI code reviewers are quick to flag things that sound like bugs and turn out not to be. Once a team has chased a few of those, it stops reading the comments. I wanted a reviewer my own team could trust on Theq, which meant one that stays quiet unless it can prove what it says.
My role
Designer and developer. I set the rules for what counts as proof and the security model around running untrusted code, then built the tool: the command line, the GitHub Action, the sandbox and a local dashboard.
Approach
Claude proposes; deterministic code verifies. The agent reads the code and can run a small set of tools, but every finding has to carry evidence that the tool then checks on its own: a failing test that is re-run in a sandbox, a test that passes on the base branch and fails on the pull request, a compiler or lint error that is new in the change, or a quote that must exist at the cited line. Findings backed by a test that runs are reported as Verified. Findings with only static evidence are kept only at high confidence, and everything else is dropped and listed in the run notes.
Fixes meet the same bar. A proposed patch must make the failing test pass three runs in a row, leave the existing tests and static checks clean, and it may never touch tests or build configuration, because changing those could make a check pass without fixing anything.
Because it runs other people's code, the safety model came first. Untrusted code only runs in a sandbox with no network. The agent has no shell, no write access and no GitHub token, it works on fresh checkouts so git-ignored secrets never exist where it looks, and the repository under review cannot reconfigure it.
For whole-repository scans it ranks code by risk (recent churn, and paths such as authentication, uploads and SQL) and reviews the riskiest parts first within a spending limit.
Deliverables
- A
pr-reviewcommand line: review a GitHub pull request or a local branch, scan a whole repository, and set a repository up with one command - A GitHub Action that reviews every non-draft pull request
- A local dashboard to start scans, read each bug's verification trail, compare the code before and after the fix, apply or undo a fix, and triage what is left
- A public landing page
Outcome
I pointed it at Theq, the marketplace app my team ships. Across eight scans it found 14 distinct bugs in the API and the AI assistant, three of them high severity. Every one was proven by a failing test, and 13 came with a fix that passed its checks.
On its seeded test repositories it found both planted bugs, proved each with a failing test and verified a fix for each, with no false positives. On a harmless refactor it reported nothing.