How to Make AI Find Real Code Vulnerabilities Without Tons of False Positives

If you've ever tried pointing an LLM at source code with a "find vulnerabilities" prompt, you probably remember the result. The model typically produces a wall of theoretical observations: missing OWASP checklist validation here, another layer of sanitization could be added there, and this variable is suspiciously named. In practice, 90% of such findings turn out to be noise that simply wastes developers' time.

The Cloudflare team has open-sourced security-audit-skill. It's a set of instructions and pipeline for coding agents, which they used to build their internal harness for continuous repository auditing.

What's the Main Problem with Regular AI Auditing

Most static analyzers and simple neural network prompts suffer from two issues: they don't verify exploitability and hallucinate context. The neural network sees a dangerous function but doesn't notice that input data is already filtered by three layers above.

Cloudflare took the opposite approach and built the skill on strict principles:

• Reports are only generated for what's actually exploitable. Phrases like "theoretically an attacker could" are immediately filtered out.
• The person who found the bug doesn't have the right to validate it. A separate independent agent runs in the role of devil's advocate for verification.
• The absence of a second protection layer isn't considered a vulnerability if the first layer reliably blocks the attack vector.
• Criticality assessment is based on real impact, not formal checklist matches.

How the Six-Phase Pipeline Works

Instead of one long prompt, the tool breaks down the work into six sequential stages. Parallel sub-agents work within, each with a strictly defined task.

+-------------------------------------------------------------+
| 1. Recon       -> Карта архитектуры и точек входа           |
| 2. Hunt        -> Параллельные атаки по разным векторам     |
| 3. Validate    -> Попытка опровергнуть каждую находку       |
| 4. Report      -> Формирование читаемых отчетов             |
| 5. Structured  -> Генерация findings.json со схемой         |
| 6. Verify      -> Сверка фактов со свежими агентами         |
+-------------------------------------------------------------+

1. Recon

Agents investigate the project, define trust boundaries, identify entry points, and map out the overall architecture. The result is a file architecture.md that serves as a map for attacking agents.

2. Hunt

Multiple agents test the codebase in parallel from different angles. The project is split into separate files with prompts for different attack classes:

• Injections, access control, and business logic.
• Web protocol specifics, caching, and authentication (WEB-PROTOCOL-AND-AUTH.md).
• Client-side threats like DOM injections and prototype pollution (CLIENT-SIDE.md).
• Memory safety and binary vulnerabilities for native code (MEMORY-SAFETY-AND-BINARY.md).
• LLM system issues: prompt injection, context leaks, and tool call manipulation (AI-AND-LLM.md).

Each hunting agent can spawn additional processes to dig deeper into suspicious call chains.

3. Adversarial Validation

The most useful stage. Fresh agents receive a list of potential bugs and deliberately try to prove that an attack won't work. If protection is in place or the vector is blocked by a neighboring module, the finding is ruthlessly crossed out.

4. Report & Structured Output

Files REPORT.md and FINDINGS-DETAIL.md are generated with detailed traces for Medium-level and higher vulnerabilities, along with findings.json. The structured JSON is validated by script validate-findings.cjs based on Node.js with no external dependencies.

5. Independent Verification

Final quality control. Agents with clean context line-by-line verify the claims in the report against the actual code to eliminate hallucinations in line numbers or function names.

Installation and Running

The package connects via Skills CLI to any coding agent that supports tool calls and parallel sub-agents.

Project installation:

npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit

Global installation for the entire system:

npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit --global

After that, just open the codebase in your agent and write in plain text:

security audit this codebase

or specify a specific directory and report path:

do a security review, output to ~/audits/my-project

Interesting detail: runs can be accumulated. The authors found during testing that one run finds roughly half of real issues due to randomness in traversal paths. The skill can read previous findings.json, skip already known bugs, and explore untouched code branches.

Who This Is For

The tool requires a capable model with parallel tool call support, so running it on weak local setups probably won't work.

But if you're already using agents for refactoring or writing tests, adding a role as a meticulous security auditor is a great idea before a release or major merge. The approach of splitting roles between "attacker" and "skeptic" noticeably reduces manual effort dealing with false positives.