A black-box companion to Doyensec's "Aikido vs. XBOW" comparison. RedPick ran the same two open-source applications — Photoview and Fider — with only a target URL and credentials, no source code: 62 internally confirmed findings, 55 extended true positives, and 34 valid issues outside the reconstructed answer key, at an 89.6% adjusted two-app score.
Published 2026-06-03 by the RedPick team. Reference: Doyensec, "Comparing AI Application Security Testing Platforms: Aikido vs. XBOW" (May 2026). RedPick's runs were pure black-box — target URL + credentials only, no repository, no source-code upload, no static review. Because Doyensec published their findings only as a partially redacted spreadsheet (some issues were still unfixed at publication), our scoring uses a reconstructed Doyensec-derived answer key — one item per distinct vulnerability family in the public report — not Doyensec's raw item list. The counts below are our own; they are not Doyensec-validated. Full per-app finding tables and the complete scoring methodology are on GitHub.
The short version. When Doyensec compared Aikido and XBOW on two real open-source apps, it gave the market a concrete, honest test drive — measuring setup, cost, speed, report quality, false positives, and manually validated findings. We treated that paper as an invitation to run a third data point. RedPick ran the same two application families in pure black-box mode and produced 55 extended true positives across them — and, more interestingly, 34 valid findings that live outside the benchmark's answer key entirely. The most important number here isn't a leaderboard position. It's how much real attack surface existed beyond the key. We put the three platforms side by side as positioning, not as a ranking.
This is not a claim that Doyensec evaluated RedPick — they did not — nor that our numbers sit on the same footing as Aikido's and XBOW's. Their findings were manually validated by Doyensec with source-code access. Ours were validated by our own replay, deduplication, and scoring pipeline, run without source code. We compare public counts and vulnerability families; we do not pretend this is a controlled three-way bakeoff.
TL;DR
| Metric | Aikido (Doyensec) | XBOW (Doyensec) | RedPick (black-box) |
|---|---|---|---|
| Targets | Fider + Photoview | Fider + Photoview | Fider + Photoview |
| Testing mode | Source-aware SaaS | Source-aware SaaS | Black-box: URL + creds only |
| Source-code access | GitHub repo paired | Source/doc upload | None |
| Doyensec-validated true positives | 49 | 31 | Not Doyensec-reviewed |
| False positives (Doyensec review) | 2 | 1 | Not Doyensec-reviewed |
| Internally confirmed findings | — | — | 62 |
| — mapping to key items / extras | — | — | 20 / 42 |
| Scorer-recognized extended TPs | — | — | 55 |
| Validated extras outside the key | — | — | 34 |
| Adjusted two-app score | — | — | 60.0 / 67 (89.6%) |
A quick note on the bases, because the post leans on it: internally confirmed is a finding count (deduplicated findings reproduced with evidence — 62, of which 20 map to key items and 42 are extras); found / partial / missed are key-item counts (33 reference items); and the extended true positive count fuses the two — the 21 key items found at full credit plus the 34 validated extras = 55. One finding can satisfy several key items, which is why 20 key-mapped findings cover more than 20 items.
- Relative to the published results: RedPick's 55 internally-scored extended TPs sit alongside the 49 and 31 Doyensec independently validated — the same range, counted on a different basis. We read this as positioning, not a verdict on any platform.
- On independent validation: Aikido and XBOW carry Doyensec's external review; our numbers do not. We make no claim that 55 is equivalent to, or better than, an externally-validated count — only that it lands in the same range under a stricter, source-free setup.
- On the benchmark itself: RedPick matched most of the reconstructed answer-key families and then substantiated 34 additional valid vulnerabilities outside that finite key — roughly as many as the entire key contains.
- On methodology: RedPick did this without source access, while both Doyensec-evaluated workflows included repository/source-code configuration.
What Doyensec measured
Doyensec compared two commercial AI-powered pentest platforms — Aikido's Attack AI Pentest and XBOW's Lightspeed — at the same price tier, on two open-source apps picked at random from a large pool:
- Fider — a multi-user feedback portal with Google OAuth and magic-link auth, roles, user-generated content, exports, Atom feeds, and public/private workflows.
- Photoview — a multi-user photo gallery with a GraphQL/API backend, media parsing and path handling, authentication, authorization, and configuration exposure.
The setup matters. Doyensec configured both platforms with source code: Aikido "paired GitHub to fetch our fork of the repo," and for XBOW they "were also able to upload the source code, which the guide says will improve the results." Those are legitimate commercial modes — but they are not black-box. RedPick's runs used no repository, no source upload, no static analysis, and no answer-key lookup during testing: only URL, credentials, and live HTTP behavior.
The public Doyensec results:
| Platform | Fider TP / FP | Photoview TP / FP | Total TP | Total FP |
|---|---|---|---|---|
| Aikido | 17/19 · 2 FP | 32/32 · 0 FP | 49 | 2 |
| XBOW | 24/25 · 1 FP | 7/7 · 0 FP | 31 | 1 |
Doyensec also found that overlap between the two tools was strikingly low — only three identical findings on Fider and four on Photoview. Different AI testing systems didn't just vary in count; they explored different parts of the same applications. RedPick's result reinforces that point.
RedPick on Photoview
Pure black-box, URL and credentials only against a self-hosted instance.
| Metric | RedPick Photoview |
|---|---|
| Internally confirmed findings | 28 |
| — mapping to key items / extras | 8 / 20 |
| Reconstructed key items | 16 |
| Found / partial / missed (items) | 10 / 5 / 1 |
| Strict key score | 12.5 / 16 (78.1%) |
| Validated extras outside the key | 18 |
| Adjusted score | 30.5 / 34 (89.7%) |
The strongest answer-key areas were SQL injection (Critical), IDOR and broken authorization across user boundaries (High), GraphQL exposure and SSRF, plus session, cookie, header and TLS weaknesses. The single strict miss was a rate-limiting key item — RedPick found related rate-limit primitives in a different shape, but they didn't map cleanly enough to the key item.
The 18 validated extras included genuinely high-impact issues, not just a hardening tail: GraphQL aliases amplifying the authorizeUser brute force (≈20× attempts per request), an empty password accepted on password change and login, a user minting a public share for an album they don't own, a GraphQL endpoint accepting multipart/form-data (CSRF where cookie auth is used), LIKE/wildcard filter injection in GraphQL search, a public share leaking server file paths, username enumeration via a login timing oracle, and an auth token stored in a non-HttpOnly cookie.
For context: on Photoview, Doyensec's public validated counts were 32 (Aikido) and 7 (XBOW); RedPick internally confirmed 28 — below Aikido's published figure, and produced without any source access. XBOW's lower Photoview number reflects its specific evaluated configuration in the study, not a general capability judgment; the platforms simply explored different parts of the app, which is the study's own central observation. Full Photoview finding table →
RedPick on Fider
Fider is a different kind of target — OAuth and magic-link auth, role transitions, user content, exports, Atom feeds, and several places where a vulnerability looks like "just a small workflow oddity" until the proof is packaged correctly. The run used live browser-authenticated Google sessions for the admin and member accounts; it did not use the Fider repository or source code.
| Metric | RedPick Fider |
|---|---|
| Internally confirmed findings | 34 |
| — mapping to key items / extras | 12 / 22 |
| Reconstructed key items | 17 |
| Found / partial / missed (items) | 11 / 5 / 1 |
| Strict key score | 13.0 / 17 (76.5%) |
| Validated extras outside the key | 16 |
| Adjusted score | 29.5 / 33 (89.4%) |
Deduplication produced 37 candidate findings; 3 low-quality near-duplicates did not pass the scoring gate, leaving 34. Of those, 12 map to reconstructed key items and 22 are extras beyond the key — the scorer credited 16 of the extras as materially valid. The 16 included excessive JWT session lifetime (auth cookie valid 365 days), cacheable responses carrying Set-Cookie (a runtime-observed shared-cache session-fixation risk), admin HTML with user emails served without cache-control, production source-map exposure, a known ReDoS in the bundled marked 4.3.0, missing SRI on first-party bundles, user/email enumeration, /api/v1/posts returning HTTP 500 on a negative limit, and sitemap.xml enumerable despite indexing-prevention config. Same pattern as Photoview: the reconstructed key captured representative known vulnerabilities, but the application contained more real attack surface than the key. Full Fider finding table →
The combined two-app view
| Metric | Photoview | Fider | Combined |
|---|---|---|---|
| Internally confirmed findings | 28 | 34 | 62 |
| — mapping to key items | 8 | 12 | 20 |
| — extras (outside key) | 20 | 22 | 42 |
| Reconstructed key items | 16 | 17 | 33 |
| Found / partial / missed | 10 / 5 / 1 | 11 / 5 / 1 | 21 / 10 / 2 |
| Strict key score | 12.5 / 16 | 13.0 / 17 | 25.5 / 33 (77.3%) |
| Validated extras | 18 | 16 | 34 |
| Adjusted score | 30.5 / 34 | 29.5 / 33 | 60.0 / 67 (89.6%) |
| Extended true positives | 28 | 27 | 55 |
How the adjusted score works. Each key item is worth one point — a full match scores 1.0, a partial 0.5, a miss 0. Each validated extra adds one point to both the numerator and the denominator, so the metric can't be inflated by piling on noise: an unsubstantiated extra earns nothing and never reaches the numerator. The combined result is (26.0 key points + 34 valid extras) / (33 key items + 34 valid extras) = 60.0 / 67 = 89.6%. RedPick lands at 89.6% rather than 100% purely because of the 10 partial matches and 2 misses against the key. The full formula, per-item credit, and the partial/miss breakdown are on GitHub.
The extras are the main point
Doyensec's study intentionally avoided true-negative analysis — the right call, since nobody can easily prove the complete absence of vulnerabilities in a real application. But once you accept that, a benchmark answer key becomes a floor, not a ceiling. A system that finds only the answer-key items is good at benchmark recall; one that also finds additional real vulnerabilities is doing something closer to useful application security work.
That's why the 34 validated extras matter. The two reconstructed keys contain 33 items in total, and RedPick substantiated 34 additional validated issues outside them. One honest caveat belongs right here: because we built that key ourselves — one entry per distinct vulnerability family Doyensec describes — the near one-to-one match between extras and key is an illustration, not a fixed law of the benchmark, and it doesn't prove uniqueness against Aikido or XBOW without their full normalized finding archives. But the shape holds regardless of how the key is sliced: a finite benchmark captured only about half of the real attack surface RedPick was able to substantiate.
This is where a pure black-box result becomes especially interesting. Source-aware systems can use code structure to find latent issues that are hard to exercise dynamically. RedPick found a second benchmark-sized set of issues without that advantage, by continuing to interrogate the live application after the known classes were already covered.
Where RedPick fell short
The two key misses are useful because they're concrete:
| Target | Missed item | Severity | Read |
|---|---|---|---|
| Photoview | Missing rate limiting on auth / high-value operations | Low | Related rate-limit primitives found, but not in the exact shape the key item expects. |
| Fider | RCE reachable from an injection primitive | High | No confirmed RCE proof was produced from the injection primitive. |
Fider also produced five partials — mass assignment in the sign-in binder, server-side template injection, open redirect, a privileged-workflow exposure, and a race condition — where evidence existed but didn't reach full proof. This is exactly the right kind of failure to have in a benchmark: not vague, not hidden. It tells us what to improve — mass-assignment testing must prove accepted over-posted fields; SSTI must separate template evaluation from RCE escalation; redirect testing must require an attacker-controlled external sink; race testing must prove a broken invariant under concurrency. Several of those improvements have already been fed back into RedPick's generic test runbooks. The full partial/miss ledger is on GitHub.
Small team, large signal
Aikido and XBOW are serious platforms with serious teams; Doyensec's paper shows both found real vulnerabilities at commercial scale, with Aikido producing the strongest public validated count and smoother overall experience. RedPick is much smaller. That doesn't make our findings more valid, and it doesn't turn internal validation into independent validation. But it changes how the result reads: when a small team can run a source-free agentic benchmark and land in the same range as the public true-positive totals of much larger competitors, the interesting story isn't a leaderboard position — it's leverage. Agentic systems are narrowing the gap between small expert teams and heavily resourced platforms.
What we would — and would not — claim
We would claim: RedPick ran Photoview and Fider in pure black-box mode; internally confirmed 62 findings (20 mapping to key items, 42 extras); produced 55 scorer-recognized extended true positives after hygiene, dedup, and key scoring; found 34 validated extras outside the reconstructed key; scored 60.0/67 (89.6%) adjusted; and landed in the same range as the published platform results, while operating in a stricter source-free setup.
We would not claim: that Doyensec validated RedPick; that RedPick formally beat Aikido in Doyensec's study; that every RedPick extra was absent from Aikido or XBOW; that internal replay validation equals independent expert validation; that our reconstructed key matches Doyensec's exact internal list; or that two applications generalize to all web apps. Those caveats don't weaken the result — they make it usable.
Bottom line
Doyensec's conclusion was measured: Aikido and XBOW are both capable, Aikido found more true positives and had a smoother workflow, XBOW had slightly fewer false positives, and both showed limits in setup, report quality, and severity calibration. RedPick adds a third signal — in a stricter black-box setup, with no repository and no source-code workflow, a small independent team produced a competitive two-app result: 62 internally confirmed findings, 55 extended true positives, 34 validated extras beyond the reconstructed key, and an 89.6% adjusted score.
A finite benchmark tells you whether a system can find representative known issues. A real pentest asks whether it can keep going — prove the next issue, deduplicate it, assess its severity, and explain it well enough that an engineering team can fix it. That's the bar, and these extras are the clearest part of the signal.
Want to see what RedPick finds in your applications? Request a demo and we'll run a proof-of-concept scan, or explore our managed pentest. For the full program, see all benchmark results.
Related: 104/104 on XBOW · Vibe-Coding: 152 Findings, 0 FP · Agentic vs Automated Pentesting
— the RedPick team, 2026-06-03