How to Make AI Find Real Bugs in Pull Requests Instead of Generating Trivial Comments

Most neural network bots for code review work the same way. They receive the diff from a commit, glance at the modified lines, and generate dozens of superficial comments. As a result, developers get formatting tips, missing docstrings, or obvious hallucinations when the AI lacks context from neighboring files. The Agent-Field project team decided to approach the problem differently and created the PR-AF tool.

What is PR-AF and Who Is It For

PR-AF stands for Pull Request AgentField. This is an open-source tool for deep code auditing at the CI/CD stage.

The project doesn't try to compete with fast interactive utilities like Claude Code, which return an answer in a couple of seconds right in the terminal. PR-AF is designed for different tasks: it runs for 35 to 50 minutes, but it builds a dynamic agent graph, extracts AST trees from the repository, and verifies each finding for falsifiability.

The tool is useful for teams that need strict automated quality control before merging into the main branch. This is especially relevant for projects with a high cost of errors in security or architecture.

How the Dynamic Pipeline Works

Instead of running a static script with a hardcoded prompt, PR-AF analyzes the structure of the incoming pull request and adjusts the execution graph accordingly.

The analysis process is divided into several stages.

First, the system evaluates the diff through three different slices: semantic, mechanical, and systemic. Temporary specialized reviewer agents are created for the identified tasks.

Then the proof verification engine kicks in. If an agent believes that input validation is missing in the code, the system doesn't take its word for it. It scans the repository, extracts the AST tree, and verifies the call chain. The finding is discarded if no confirmation is found in the code.

At the next step, the falsification filter activates. The system attempts to refute its own hypothesis about a bug by checking existing safeguards and the author's intended behavior.

At the end, the composite risk synthesizer connects. Isolated minor issues in different files often combine into a critical vulnerability. The tool links such findings into a single report.

Benchmark Results and Economics

On the Martian Code-Review-Bench benchmark, the PR-AF system in combination with the open GLM-5.2 model showed a bug detection recall (golden recall) of 0.706. In the test set of 42 tools, this result placed the project first among open-source solutions.

During testing, the system independently discovered 595 confirmed errors. When using top commercial models, the results are even higher.

At the same time, the cost of one run is approximately 10 times lower than closed SaaS alternatives. You only pay for LLM tokens through your own API key, without a monthly subscription per user.

Quick Start and Working via API

You can launch PR-AF locally in a couple of minutes via Docker Compose.

git clone https://github.com/Agent-Field/pr-af.git
cd pr-af
cp .env.example .env
docker compose up --build

In the .env file, you just need to specify your OPENROUTER_API_KEY and GH_TOKEN with read and write permissions for the repository. After startup, the control node will be available on port 8080.

You can send a pull request for review with a standard HTTP request:

curl -X POST http://localhost:8080/api/v1/execute/async/pr-af.review \
  -H "Content-Type: application/json" \
  -d '{"input": {"pr_url": "https://github.com/owner/repo/pull/123"}}'

In response, you receive a final JSON with vulnerability analysis, broken down by severity level. If the bot has access to GitHub, it will automatically place comments right on the relevant lines of code with supporting evidence.

Integration with GitHub Actions

The easiest way to integrate built-in auditing is into standard CI/CD. A .github/workflows/pr-af-review.yml file is added to the repository, and the run is triggered by adding a pr-af label to the pull request.

name: AgentField PR Review

on:
  pull_request:
    types: [labeled]

jobs:
  pr-af-review:
    if: github.event.label.name == 'pr-af'
    runs-on: ubuntu-latest

    permissions:
      contents: read
      pull-requests: write

    steps:
      - name: Checkout PR-AF
        uses: actions/checkout@v4
        with:
          repository: Agent-Field/pr-af
          path: pr-af

      - name: Start AgentField & PR-AF
        working-directory: ./pr-af
        env:
          OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          docker compose up -d
          sleep 15

      - name: Execute Deep Architectural Audit
        working-directory: ./pr-af
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
        run: |
          python3 scripts/ci_runner.py

The developer attaches the label to the responsible PR, half an hour passes, and a detailed report with verified facts appears in the thread.

Under the Hood

The project's main node was recently rewritten in Go (the code lives in the go/ directory), which positively affected speed and memory consumption. The original Python implementation remains available in the repository as an alternative.

The project is distributed under the permissive Apache 2.0 license. The code is fully open, including scripts for reproducing the benchmark results.

Conclusion

PR-AF offers a sensible approach to automated review. Instead of spamming the compilation with superficial comments, the system spends time on detailed AST analysis and hypothesis verification.

The tool is ideal for teams that are tired of false positives from regular AI bots and need a reliable security filter in CI/CD. It's not suitable for quick typo fixes, but it performs excellently before merging critical features into production.