How today's bug-finding and bug-fixing CRSs perform on CRSBench.
We compare bug-finding CRSs on 117 benchmarks (304 CPVs): a fuzzer-only baseline running each project's own fuzzer, an LLM agent (Claude Code, Opus 4.6) with basic bug-finding instructions, and, on the hardest subset, a hybrid CRS running both concurrently with a shared corpus. Each column below is one exclusive overlap class: which configurations found exactly those CPVs.
Across all 304 CPVs, the fuzzer and the LLM agent largely agree on the easy bugs and diverge on the rest: 74 of the fuzzer's 80 CPVs (93%) are also found by the agent, but the agent reaches 170 CPVs the fuzzer never triggers, and the fuzzer still finds 6 CPVs the agent misses. Neither approach subsumes the other.
| CRS | Solved CPVs | Trial success | Time to trigger | LLM $/trial |
|---|---|---|---|---|
| Fuzzer only crs-given-fuzzer | 24.6% (227/921) | 1,668s | - | |
| LLM agent crs-claude-code (Opus 4.6) | 62.9% (579/921) | 1,981s | $17.07 |
Five LLM agent-based CRSs run the full pipeline (find a vulnerability, then fix it) on a partial AFC subset of 51 vulnerabilities, using identical prompts and changing only the vendor-provided coding-agent harness. The finding stage is the main differentiator: bug-finding success spans 45–92% while conditional fix rates cluster at 70–94%, and finding dominates the LLM cost ($1.09–$10.91 per trial vs. $0.12–$1.19 for fixing).
| CRS | ||||||
|---|---|---|---|---|---|---|
| Claude Code Opus 4.6 | $10.91 | $1.19 | $9.52 | |||
| Opencode GLM-5.1 | $2.11 | $0.12 | $1.73 | |||
| Gemini CLI Gemini 3 Flash | $1.77 | $0.35 | $1.55 | |||
| Codex GPT-5.4-mini | $1.26 | $0.36 | $1.15 | |||
| Claude Code Haiku 4.5 | $1.09 | $0.42 | $0.91 |
Success = vulnerabilities solved at least once across 3 trials on the 51-vulnerability AFC subset; cost = mean per-trial LLM spend. *Fix is conditional on the vulnerability being found first. Click a column header to sort.
Three coding agents patch all benchmark vulnerabilities under two modes: delta, where the bug-introducing commit is supplied, and full, where the CRS must localize the fault in the whole project on its own. Every patch must reproduce none of the PoVs (including CRSBench's variant PoVs) and pass the project's functionality tests.
| CRS | Delta mode | Full mode | Overall | Time/trial | LLM $/trial |
|---|---|---|---|---|---|
| Claude Code Opus 4.6 | 86.3% | 607s | $1.43 | ||
| Codex GPT-5.4 | 87.3% | 589s | $1.29 | ||
| Gemini CLI Gemini 3.1 Pro | 86.6% | 1,255s | $0.89 |
We bucket all 2,736 fixing trials by properties of the ground-truth patch: changed lines, files, hunks, and whether the top line of the crash stack trace matches the patched code. Success drops as patches grow or move away from the crash site, falling to 75–81% for patches with 3+ hunks, 14+ changed lines, or no match with the top line of the crash stack trace.