AI Code Review in CI: Quality Gates That Cut Noise, Not Corners

What you need to know

• Order matters more than model choice. Deterministic gates (linters, type checkers, secret scanners) run first and block; the LLM reviewer runs second and mostly advises; humans review last, on load-bearing paths — a sequence argued forcefully in Addy Osmani's mid-2026 essay on agentic code review.
• Severity tiers keep the gate credible. Block on a short list of near-deterministic security findings, warn on performance, suggest on style, open a discussion on architecture. A reviewer that blocks merges on style opinions gets switched off within a month.
• False positives are a config problem, not fate. Learnings files, path filters and confidence thresholds are the three levers. ByteDance's BitsAI-CR reached 75.0% precision in production only after adding a dedicated filtering stage — generation alone was not enough.
• AI-authored PRs need the gate most. A 2026 study of the AIDev dataset found most AI-generated pull requests receive no review at all, and when reviewed, the activity is dominated by other AI agents rather than humans.
• Measure resolution, not volume. The comment-resolution rate — did the flagged line actually change before merge? — is the one metric that separates a useful reviewer from an expensive lorem-ipsum generator.

The methodology below is deliberately tool-agnostic. Named products appear in the landscape section, dated as of July 2026, but every pattern here — the pipeline order, the tier matrix, the learnings file, the metrics — lives in your repository and survives any vendor swap.

Why AI code review became a default quality gate — and why teams turn it off

Two curves crossed somewhere in 2025. The volume of code arriving in pull requests went vertical as coding agents matured, while the supply of human review attention stayed exactly where it has always been. Addy Osmani's essay on agentic code review (June 2026) frames the resulting inversion bluntly: review, not writing, is now the bottleneck skill in software engineering, and teams that generate code four times faster than they can read it accumulate risk at the difference between those two rates. The industry telemetry he cites points the same direction — sharply rising churn and defect rates, review durations stretching, and a growing share of PRs merged with no review at all.

The obvious response — point a model at the diff — is now genuinely cheap. A mid-sized diff with surrounding context runs to a few tens of thousands of tokens; at July 2026 frontier-model prices that is an order-of-magnitude cost of tens of pence, or a few tens of rupees, per review pass. For a team in Bengaluru or London shipping thirty PRs a week, metered LLM review costs less than one developer-hour a month. Seat-priced products cluster around $24–40 per developer per month (roughly £19–32, or ₹2,100–3,500). The economics are not the obstacle.

The obstacle is noise. Ask engineers who have lived with a badly configured AI reviewer and you hear the same story, in the same order: week one, delight at a real bug caught; week three, irritation at nitpicks on test files; week six, the team scrolls past every comment; week eight, someone opens the PR that removes the integration. A Hacker News thread on AI review tooling from early 2026 captures the practitioner split precisely: out-of-the-box reviewers get described as "pure noise" — flagging non-existent problems with confidently misleading scores — while the same class of tool, given extensive codebase-specific context and prompting, moves teams to a culture where the first question on any PR is "has the AI looked at it?". The difference between those two outcomes is not the model. It is configuration discipline, and it is the subject of the rest of this guide.

One more framing rule before the mechanics, borrowed directly from Osmani: tier by risk, not by author, and treat the AI reviewer as a sensor, not a verdict. A sensor's job is to surface information reliably enough that a decision-maker trusts it. The moment your sensor starts making merge decisions it is not qualified to make, people stop trusting the readings — and then you have neither a sensor nor a gate.

The pipeline: static analysis first, LLM review second, humans last

The single highest-leverage design decision is sequencing. A well-ordered review pipeline has three stages, each catching what the previous one cannot, each cheaper per finding than the one after it.

Stage one: deterministic gates. Linters, type checkers, formatters, secret scanners, dependency audits and your test suite run first, and they alone hold unconditional blocking power. They are fast, they are free of false positives in the probabilistic sense — a type error is a type error — and every finding they catch is a finding the LLM never wastes tokens or reviewer attention on. If you have not yet built this layer for AI-assisted work, start with our guide to the QA discipline that separates demos from products: that article is about making the code trustworthy; this one is about making the review of it scale. There is no point running an LLM reviewer on a diff that fails tsc.

Stage two: LLM review. The model reads the diff plus context and looks for what deterministic tools structurally cannot see: logic that contradicts the PR description, a subtly inverted conditional, a missing permission check on a new endpoint, an N+1 query introduced on a hot path, error handling that swallows the case the ticket was about. This stage runs only after stage one passes, posts findings as structured comments with severities, and blocks the merge only in the narrow circumstances described in the next section.

Stage three: humans. Human attention goes where automation is weakest: does this change do what the requirement actually meant, is this the right boundary to draw, what breaks two services away if this assumption is wrong? Osmani's essay recommends heterogeneity within stage two for exactly this hand-off reason — in one practitioner experiment he recounts, four AI reviewers run in parallel across 146 PRs produced findings of which 93.4% were caught by exactly one tool, and none by all four. Different reviewers have largely disjoint blind spots; a second cheap reviewer widens coverage far more than a better single one. But the deeper point is that no number of parallel sensors substitutes for the human owning the merge decision on anything load-bearing.

Tiered gates: block, warn, suggest, discuss

The fastest way to lose the room is to let a probabilistic reviewer block merges on matters of taste. The fix is a severity contract, agreed by the team and enforced in CI, that maps every class of finding to exactly one pipeline behaviour. Four tiers cover practically everything:

Two properties make this matrix work. First, the block tier is short and near-deterministic: every item on it is something the team would block on anyway, phrased concretely enough that a model rarely misfires. When a block-tier finding does fire wrongly, treat it as a sev-2 against the review config itself — the credibility of the whole gate rests on that tier's precision. Second, the tiers give the model somewhere to put its lower-confidence observations. A reviewer forced to choose between "blocking issue" and "silence" will over-block; give it a suggest tier and the marginal observations land where they cost nothing.

ByteDance's production system offers supporting evidence for taxonomy-first design. The BitsAI-CR paper describes review capability built on a comprehensive taxonomy of review rules, with a two-stage architecture — a RuleChecker that generates candidate findings against the taxonomy, then a ReviewFilter that discards low-quality ones. Structure before generation, filtering after: the same shape as the tier matrix above.

Taming false positives: learnings files, path filters, confidence thresholds

Every team that keeps an AI reviewer past the honeymoon converges on the same three mechanisms. They are boring, they are config files, and they are the entire difference between a gate and a nuisance.

Learnings files

A learnings file is a version-controlled list of your codebase's intentional oddities — the things a reasonable outside reviewer would flag and be wrong about — appended to every review prompt. Commercial tools grew this feature because users demanded it (CodeRabbit's "learnings" mechanism, which remembers dismissed patterns, gets singled out approvingly in the Hacker News thread above); if you run your own review step, it is ten minutes of work:

# .github/review/learnings.md
# Appended to every LLM review prompt. One intentional pattern per line.
# Prune quarterly: every entry here is a false positive we paid for once.

- API payloads use snake_case to mirror the Postgres schema. Do not
  suggest camelCase.
- `dangerouslySetInnerHTML` in src/cms/RichText.tsx receives output of
  rehype-sanitize only. Do not flag it as XSS.
- We return Result<T, E> from service functions instead of throwing.
  Do not suggest converting to exceptions.
- Raw SQL in supabase/migrations/ is intentional and reviewed by a
  human DBA. Comment only on destructive operations without a guard.
- retry(fn, { attempts: 1 }) in tests is deliberate — it exercises the
  wrapper. Do not flag as pointless.

The pruning comment matters. A learnings file is a list of lessons paid for in false positives, and like any allowlist it rots; review it quarterly, and delete entries whose underlying code is gone. This is the same principle as the steering files you may already keep for coding agents — our guide to AGENTS.md and CLAUDE.md files that actually steer agents covers the generation side of exactly this discipline.

Path filters

The cheapest false positive is the one never generated. Lockfiles, snapshots, generated clients, vendored code and migration dumps attract LLM commentary and never benefit from it. Exclude them at diff-collection time, before the model sees a token — this also cuts cost, since lockfile churn regularly dwarfs the hand-written part of a diff:

# .github/review/rules.yml — read by the review step in CI
paths:
  include:
    - "src/**"
    - "services/**"
    - "supabase/migrations/**"
  exclude:
    - "**/*.lock"
    - "package-lock.json"
    - "**/__snapshots__/**"
    - "**/*.generated.*"
    - "vendor/**"
    - "docs/**"

tiers:
  block:      # short, near-deterministic; a wrong block is a config sev-2
    - committed secret or credential
    - SQL or shell command built by concatenating user input
    - authorization check removed or weakened on an existing endpoint
  warn:
    - N+1 query on a request path
    - unbounded list endpoint (no limit or pagination)
    - external call without a timeout
  suggest:
    - naming, duplication, clarity
  discuss:
    - new external dependency
    - cross-service schema change

confidence:
  minimum: 0.7        # findings below this are dropped silently
  block_minimum: 0.9  # blocking requires near-certainty

Confidence thresholds

Ask the model to self-rate each finding and drop everything under a floor — with a materially higher floor for the block tier. Self-rated confidence is imperfect and the Hacker News thread contains fair complaints about confidently wrong scores, which is why the threshold is a filter and not a promise: it removes the model's own marginal guesses, and the block tier's higher floor means the only merge-stopping findings are ones the model is least likely to be wrong about. The empirical case for filtering as such is strong: the BitsAI-CR authors report 75.0% precision in review-comment generation in production — a figure achieved with the ReviewFilter stage discarding weak candidates, at a scale of more than 12,000 weekly active users inside ByteDance. Generation plus filtering is the pattern that survived contact with production; generation alone did not.

Wiring it up: GitHub Actions and GitLab CI patterns

The wiring is deliberately thin: collect a filtered diff, hand it to a review script with your rules and learnings, and let exit codes express the tier contract. Everything opinionated lives in the two config files above, so swapping the underlying model or tool later is a one-line change.

GitHub Actions

# .github/workflows/pr-quality-gates.yml
name: pr-quality-gates
on:
  pull_request:
    types: [opened, synchronize, reopened]

concurrency:
  group: review-${{ github.event.pull_request.number }}
  cancel-in-progress: true          # a new push obsoletes the running review

permissions:
  contents: read
  pull-requests: write              # the review step posts comments

jobs:
  static:                           # stage one: deterministic, blocking
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: "npm" }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test
      - uses: gitleaks/gitleaks-action@v2      # secret scan
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

  llm-review:                       # stage two: runs only if stage one passed
    needs: static
    if: "!contains(github.event.pull_request.labels.*.name, 'skip-ai-review')"
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }    # full history so the base branch ref exists
      - name: Collect filtered diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD \
            -- 'src/**' 'services/**' 'supabase/migrations/**' \
            ':(exclude)**/*.lock' ':(exclude)**/__snapshots__/**' \
            > pr.diff
      - name: LLM review — blocks on security tier only
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          node tools/review/llm-review.mjs \
            --diff pr.diff \
            --rules .github/review/rules.yml \
            --learnings .github/review/learnings.md \
            --fail-on block          # exit 1 only for block-tier findings

Trace the behaviour: the review job cannot start until lint, types, tests and the secret scan are green, so the model never reviews broken code. fetch-depth: 0 makes origin/<base> resolvable for the three-dot diff, which compares against the merge base — the PR's actual changes, not drift on the base branch. The path filters run at diff time, so excluded files cost no tokens. The script posts warn/suggest/discuss findings as PR comments through the granted pull-requests: write permission and exits non-zero only for block-tier findings above the 0.9 confidence floor; make llm-review a required status check and the tier contract is enforced by the platform. The escape-hatch label keeps humans sovereign — applying skip-ai-review is visible in the PR history, which is precisely the auditable override the block tier needs.

GitLab CI

# .gitlab-ci.yml (review stages only)
stages: [static, review]

lint-and-types:
  stage: static
  image: node:22
  script:
    - npm ci
    - npm run lint
    - npm run typecheck
    - npm test

llm-review:
  stage: review
  image: node:22
  needs: [lint-and-types]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  variables:
    # Self-hosted option: point at an internal OpenAI-compatible gateway
    # (vLLM, or a proxy in front of a private deployment) and no diff
    # ever leaves your network.
    LLM_BASE_URL: "https://llm-gw.internal.example.com/v1"
  script:
    - git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
    - git diff origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME...HEAD > mr.diff
    - node tools/review/llm-review.mjs
        --diff mr.diff
        --rules .gitlab/review/rules.yml
        --learnings .gitlab/review/learnings.md
        --fail-on block

The self-hosted variable is the part regulated teams care about. Banks in Mumbai and health-tech firms in Leeds have the same constraint: pull-request diffs are sensitive IP and sometimes regulated data, and shipping them to a third-party SaaS needs sign-off that may never come. An internal gateway serving an open-weight model turns the entire pattern into an on-network service — the CI config does not change, only the URL. If you go this route, the reviewer is itself an agent handling untrusted input (the diff), so run it with the least privilege it needs; our guide to sandboxing AI agents with microVMs and allowlists covers that hardening in depth.

Reviewing AI-generated PRs: pre-screening before a human ever looks

Here is the uncomfortable finding of 2026. A study of the AIDev dataset — "These Aren't the Reviews You're Looking For: How Humans Review AI-Generated Pull Requests", from a team at Nicolaus Copernicus University — examined what actually happens to agent-authored PRs in the wild. Most receive no review at all. When review activity does occur, it is largely dominated by AI agents rather than humans, and much of it is better described as agent steering — nudging the authoring agent with follow-up instructions — than as standalone evaluation. The authors note this undermines the comforting assumption, common in mining studies, that review metrics indicate human oversight. On current evidence, a large share of AI-written code is merging with no human having meaningfully read it.

You cannot fix that by exhorting people to review harder; the volume asymmetry that created the problem is permanent. What works is pre-screening: agent-authored PRs go through a strictly gated path so that by the time a human looks — and for anything load-bearing, a human must still look — the cheap questions are already answered.

A pre-screening path that works in practice: agent PRs open as drafts and carry an ai-authored label applied automatically by the authoring pipeline. Stage one runs as normal, with no exemptions. Stage two runs with the same tier config but a stricter posture — lower confidence floor for warnings, and the PR description checked against the diff, because an agent's summary of its own work is a claim, not a fact. Only when both stages are green does the PR leave draft and enter a human queue, annotated with the LLM reviewer's findings as a map of where to spend attention. The human reads requirements-fit, interface choices and blast radius, and owns the merge. Review depth follows the risk of the change, not the species of its author — agent PRs are not second-class, they simply never skip the queue.

One rule deserves stating on its own: the reviewing model should not be the authoring model with the same context, or you have recreated the self-marked exam. Different model, or at minimum a fresh context with an adversarial brief. The heterogeneity finding from Osmani's essay applies with extra force here — disjoint blind spots are precisely what you are paying a second model for.

Measuring signal: comment-resolution rate, precision and cost per PR

An unmeasured AI reviewer drifts towards noise, because adding rules is always easier than deleting them. Four numbers, reviewed monthly, keep it honest:

Resolution rate is the anchor, and it has real-world precedent: the BitsAI-CR team's headline production metric is an "outdated rate" — whether flagged code was actually modified in subsequent commits — for which they report 26.7% on Go code at ByteDance's scale. The exact healthy number for your team depends on tier mix and codebase age, so treat your first quarter as baseline-setting rather than target-chasing. What matters is the trend and the response to it: when resolution falls, the correct move is almost always to delete rules and raise thresholds, not to add more detection.

Precision sampling is deliberately manual — twenty findings a month is an hour of a senior engineer's time and produces the ground truth that no dashboard can. If you already run evaluation suites for your prompts and agents, fold review-quality sampling into the same cadence; the machinery in our guide to putting evals in CI applies unchanged, with review findings as the eval set. And treat any change to the review prompt, rules file or model version as a change requiring a regression pass against a saved set of past diffs with known-good findings — review configs regress exactly the way prompts do.

Tool landscape snapshot — and how to stay tool-agnostic

As of July 2026 the field is crowded and moving fast, so hold this section lightly and the methodology above tightly. The main options a team evaluating this quarter will meet:

Independent, current comparison data is thin — most published comparisons are vendor-authored. DeepSource's own tool comparison (updated March 2026) is worth reading with that bias declared: its benchmark against 200-plus real CVEs found accuracy spreads from single digits to over 80% across tools, and its central argument — that only deterministic analysis should hold hard blocking power, with LLMs layered above — happens to match the pipeline order this guide arrived at from practitioner evidence. Osmani's essay collates third-party precision figures for several commercial reviewers that differ by multiples between tools; the honest conclusion from both sources is that tool performance varies enormously by codebase and configuration, and your own sampled precision on your own diffs beats any published number.

Staying tool-agnostic is therefore not indecision but design. The assets that matter — rules.yml, the learnings file, the tier contract, the metrics dashboard, the saved regression diffs — all live in your repository in the pattern above, and every commercial tool can be wired in behind the same thin CI step it would replace. Choose a tool for a year, not a decade; re-run the evaluation annually against your own resolution-rate data. If you are also choosing the authoring side of the stack, our comparison of Claude Code, Cursor and Codex pairs naturally with this exercise — several vendors now sell both halves, and the bundling discounts are real but should not decide your review architecture.

The closing thought is the same one the opening curves imply. Code generation will keep getting cheaper; trustworthy judgement about code will not. Teams in Bengaluru and Birmingham alike are discovering that the engineers who thrive in this regime are the ones who can design and run the judgement pipeline — the gates, the thresholds, the metrics — rather than merely feed it. That is a skill worth showing in public. If you have built a review pipeline that cut noise without cutting corners, the write-up, the config and the before/after resolution numbers belong on your Builder profile, where the people hiring for exactly this can find them.