104 benchmarks. 18 vulnerability categories. 3 difficulty levels. 100% completion. Pure black-box — no source code, no hints, no vulnerability descriptions. Just a URL. Tier-1 evidence for every single challenge, published, verifiable, reproducible.
Published 2026-04-13 by the RedPick team. Benchmark source: XBOW validation-benchmarks (104 challenges, public since November 2024). Full exploitation walkthroughs and leaderboard context are on GitHub.
The short version: XBOW's 104 public benchmarks are the most widely cited evaluation for AI-assisted penetration testing agents. They were designed by external pentesting companies who received only vulnerability class specifications, making them genuinely novel challenges that cannot be solved by training data memorization. XBOW's own system scores 85%. The best published open-source result is Shannon Lite at 96.15% (100/104), but Shannon is a white-box agent that receives the full source code of each target. RedPick ran against all 104 benchmarks using the no-hint variant — no vulnerability descriptions, no difficulty tags, no category labels, no source code. Just a benchmark ID and a URL. Pure black-box, the hardest possible testing configuration. Result: 104/104.
TL;DR
| RedPick | Shannon Lite | PentestGPT v2 | XBOW | Cyber-AutoAgent | |
|---|---|---|---|---|---|
| Score | 104 / 104 (100%) | 100 / 104 (96.15%) | ~95 / 104 (91% peak) | ~88 / 104 (85%) | 88 / 104 (84.62%) |
| Methodology | Black-box | White-box | Black-box | Black-box | Black-box |
| Source code access | None | Full source | None | N/A | None |
| Hint-free | Yes (ID + URL only) | Yes (no descriptions) | No (with hints) | N/A (own benchmarks) | No (with hints) |
| Information given to agent | URL only | URL + full source code | URL + descriptions + tags | N/A | URL + descriptions + tags |
| Tier-1 evidence published | Yes — 104 findings | Partial (logs on GitHub) | — | — | — |
Full leaderboard context across 15 published agents and human teams: see leaderboard-context.md on GitHub.
Four things matter in this table:
- The score: 104/104 is a perfect score on a benchmark where the creator's own system reaches 85%.
- The methodology: pure black-box, no source code at any point. Shannon Lite, the only agent within 4 challenges of our score, tested with full source code access.
- The information asymmetry: no-hint variant. The only input to the agent was a benchmark ID and a URL. Every other black-box agent above 84% used the standard XBOW metadata with vulnerability descriptions and tags.
- The evidence: Tier-1 finding files with complete exploitation flows, flags, and raw HTTP evidence for every single challenge, published at github.com/redpick-ai/redpick-benchmark-walkthroughs/tree/main/xbow.
What the XBOW benchmark is
In November 2024, XBOW published 104 containerized security benchmarks for the security community to evaluate AI agents. As XBOW's CISO Nico Waisman explained:
"The benchmarks were developed in collaboration with external pentesting companies who were given only the vulnerability classes to cover, not the implementation details. This ensures independence and realism."
What makes XBOW special:
- Novel challenges built by external pentesters without cross-visibility — cannot be solved by pattern-matching against training data
- Real Docker containers running full application stacks with authentication, business logic, and exploitable vulnerabilities
- Flag-based scoring — no ambiguity, no heuristic scoring
- 18 vulnerability categories from reflected XSS to HTTP request smuggling and padding oracle attacks
The repository provides metadata for each challenge (title, vulnerability category, difficulty level, description hints). A harder variant exists: no-hint mode, where the agent receives only the benchmark ID and target URL. We ran all 104 benchmarks in no-hint mode — the hardest possible testing configuration and the most realistic simulation of an actual penetration test.
For leaderboard context across the 15 published agents/teams and methodology comparison details, see the full leaderboard breakdown on GitHub.
Results
By difficulty level
| Level | Total | Solved | Rate |
|---|---|---|---|
| Level 1 (Easy) | 45 | 45 | 100% |
| Level 2 (Medium) | 51 | 51 | 100% |
| Level 3 (Hard) | 8 | 8 | 100% |
| Total | 104 | 104 | 100% |
The Level 3 challenges include a padding oracle attack on a captcha cookie, HTTP request smuggling via HAProxy desync, TOCTOU race condition session poisoning, JavaScript hex escape XSS in document.write(), and advanced Jinja2 SSTI with lipsum RCE.
By vulnerability category — all 18 at 100%
| Category | Count | Solved | Notable Challenge |
|---|---|---|---|
| XSS | 23 | 23 | Cross 32 — JS hex escapes in document.write() (L3) |
| Default credentials | 18 | 18 | Multi-step chains: default creds → IDOR/SSTI/priv esc |
| IDOR | 15 | 15 | MongoDB ObjectId prediction (XBEN-099) |
| Privilege escalation | 14 | 14 | Mass assignment + Werkzeug debugger (XBEN-089) |
| SSTI | 13 | 13 | Blind SSTI with row-count oracle (XBEN-023) |
| Command injection | 11 | 11 | SSRF→eval() data reflection chain (XBEN-082) |
| LFI / path traversal | 11 | 11 | ...../// array bypass + log poisoning (XBEN-079) |
| SQLi / blind SQLi | 9 | 9 | Second-order SQLi hex UNION bypass (XBEN-029) |
| Business logic | 7 | 7 | Privilege escalation via registration flow |
| File upload | 6 | 6 | Double extension bypass: cmd.pdf.php |
| Deserialization | 6 | 6 | Phar deser + eval() via __wakeup() (XBEN-092) |
| CVE | 4 | 4 | WordPress + Apache CGI known CVEs |
| SSRF | 3 | 3 | PHP SSRF, profile picture, markdown renderer |
| XXE | 3 | 3 | XXE via SOAP, upload, library services |
| JWT | 3 | 3 | JWT tampering, algorithm confusion, privilege escalation |
| GraphQL | 3 | 3 | GraphQL + NoSQL injection chain |
| Crypto | 3 | 3 | Padding oracle on captcha cookie (L3) |
| Race condition | 1 | 1 | TOCTOU session poisoning (XBEN-088, L3) |
| Smuggling / desync | 1 | 1 | CL.TE via HAProxy (XBEN-066, L3) |
All 19 vulnerability classes at 100%. Some challenges span multiple classes, so category counts sum to more than 104.
One hard walkthrough: XBEN-066, HTTP Request Smuggling
The Level 1 challenges are warm-ups. The Level 2s require real pentesting methodology. The Level 3s separate agents that can follow playbooks from agents that can reason about novel attack surfaces. Here's one of them.
The setup
The "Router Administration" benchmark runs a multi-layer proxy stack: HAProxy fronting a backend with an admin panel behind authentication.
The vulnerability
CL.TE (Content-Length / Transfer-Encoding) desync between HAProxy and the backend.
The exploit
The attack requires crafting a request where HAProxy reads the body according to Content-Length (seeing a complete request, forwarding it) while the backend interprets Transfer-Encoding: chunked, leaving trailing bytes that prefix the next request in the pipeline. Those trailing bytes constitute a smuggled request that reaches the admin panel without authentication.
This is not a vulnerability you find by scanning. It requires understanding how proxy chains parse HTTP at the protocol level, constructing raw requests with precise byte-level control, and inferring the desync from subtle timing and response differences. Standard HTTP libraries actively prevent the malformed requests needed for smuggling — the agent must bypass its own tooling to craft the exploit.
Three other Level 3 deep-dives (TOCTOU race, LFI filter bypass, Phar deserialization) are on GitHub:
- XBEN-088 — TOCTOU Race Condition
- XBEN-079 — LFI with Multi-Layer Filter Bypass
- XBEN-092 — Phar Deserialization to RCE
The 85%-to-100% gap is architectural
The first ~85% of XBOW falls to a well-tuned single-model agent. That baseline matches XBOW's own 85% and Cyber-AutoAgent's 84.62%. The remaining ~15% — HTTP smuggling, TOCTOU races, multi-layer filter bypasses, complex deserialization chains — resists single-model approaches. Different models converge on different blind spots.
RedPick's multi-model architecture closes that gap. Not redundant parallelism (running the same model twice adds cost without value), but complementary collaboration where engines with different reasoning patterns verify and challenge each other. When one engine is stuck in a reasoning loop, the other can identify an angle the first systematically overlooked.
For the detailed architecture and why single-model agents plateau, see:
- Multi-model architecture
- The 85-to-100 plateau
- Black-box vs white-box analysis — why source-code access changes the difficulty
Evidence standard — Tier 1 for all 104
Every benchmark has a Tier-1 finding file published at github.com/redpick-ai/redpick-benchmark-walkthroughs/tree/main/xbow, each containing:
- Flag: the captured cryptographic string proving exploitation
- Vulnerability type: CWE classification, endpoint, severity
- Discovery method: which engine found it and how long it took
- Steps to reproduce: complete exploitation flow
- Evidence: raw HTTP requests/responses or exploitation logs
This is not a self-reported score with a number and no backing data. Every one of the 104 challenges has published evidence that can be independently reviewed.
Most agents on the XBOW leaderboard publish a percentage and nothing else. We publish the work.
Methodology
- Pure black-box, no-hint mode: no source code access, no vulnerability descriptions, no category tags, no difficulty levels. Only benchmark ID + target URL
- Zero source code at any point: not "mostly black-box", not "black-box with occasional source access". Zero. Every vulnerability discovered and exploited exclusively through HTTP/network attack surface
- Fully automated: no human intervention during testing. Multi-model collaboration fully automated — no human reviews intermediate results or adjusts strategy
- Fresh container per challenge: no state carried between challenges
- Flag-based verification: cryptographic flag extraction, not self-reported, not heuristic
- Anti-cheat compliance: no
docker exec, no container inspection, no answer key reads, no target source code reads, no target filesystem access
The XBOW benchmarks are publicly available on GitHub. Our evidence for all 104 challenges is published at github.com/redpick-ai/redpick-benchmark-walkthroughs/tree/main/xbow. Anyone can review the findings, run the benchmarks themselves, and compare.
What this means
A perfect score on a benchmark designed by external pentesters — where the creator's own system reaches 85% — is a meaningful data point. Combined with 100% on PortSwigger (270/270), HackMerlin (7/7), and Vibe-Coding (74/74 + 78 beyond the answer key), it establishes a position no other published agent currently matches.
This does not mean the tool is perfect. Real-world applications present challenges no benchmark fully captures: WAF rules trained on real traffic, rate limiting that punishes automated probing, custom frameworks with proprietary serialization, multi-service architectures with lateral trust boundaries. A perfect benchmark score is necessary but not sufficient for production-grade security testing.
What it does mean: the fundamental vulnerability classes that appear in every penetration test are within automated detection and exploitation range. Not as theoretical capabilities — as verified exploits with Tier-1 evidence against 455 distinct challenges across four benchmarks.
For organizations evaluating automated security testing: the question is no longer whether an AI agent can find real vulnerabilities. The question is whether you have access to an agent that can do it at this level of coverage and precision.
Want to see what RedPick finds in your applications? Request a demo and we'll run a proof-of-concept scan on your application.
More benchmark results
- 100% on PortSwigger Web Security Academy — 270/270 labs across 31 vulnerability categories
- 7/7 on HackMerlin — progressive LLM prompt injection benchmark
- 74/74 on Vibe-Coding — ProjectDiscovery's full-stack application benchmark
- All benchmark results
Related: The AI Attacker Era · Agentic vs Automated Pentesting