ProjectDiscovery Vibe-Coding: 152 Findings, 0 FP

74/74 ground-truth, 78 additional code-backed vulnerabilities held to the same evidence standard, 10 end-to-end attack chains, 0 false positives across vaultbank, medportal, and claimflow — what looking beyond the answer key reveals about the real attack surface of AI-generated code.

Published 2026-04-11 by the RedPick team. Follow-up to "Inside the Benchmark" (ProjectDiscovery, April 2026). Complete finding tables and methodology details are on GitHub.

The short version. ProjectDiscovery's Vibe-Coding Benchmark is one of the most rigorous public evaluations of AI-assisted pentest agents we have seen — three real AI-coded apps, a curated 74-vulnerability answer key, and honest per-scanner results. We ran our own pipeline against the same three targets under the same rules, and tried to answer a question the benchmark was not designed to answer: how much of the real attack surface does the 74-item answer key actually cover?

We reached 74/74 on the ground-truth catalog, and verified 78 additional code-backed vulnerabilities that live in the apps but aren't in the answer key. Each one held to the same evidence standard (CVSS 4.0, working PoC, raw HTTP request + response, non-destructive verification, source-code sink pinned to file and line), reconciled against the benchmark source at commit 17568bbb.

152 distinct confirmed findings. 0 false positives. 10 end-to-end attack chains.

This is not a criticism of ProjectDiscovery's benchmark, which we think is excellent as-is. It is a different question with a different answer: the answer key is a floor, not a ceiling, and the gap between the two is where a real attack surface lives.

---

TL;DR

	RedPick	Neo (PD baseline)
Ground-truth coverage	74/74 (100%)	66/74 (89.2%)
False positives	0	5
Precision (ground-truth only)	100%	93.0%
Additional code-backed vulnerabilities (same evidence standard)	78	—
Unique attack chains	10 (8 Critical + 2 High)	—
Total confirmed distinct findings	152 (74 GT + 78 additional)	—
Coverage ratio vs answer key	2.1×	—

---

Context — what the Vibe-Coding Benchmark is

In April 2026 ProjectDiscovery published the Vibe-Coding Benchmark: three full-stack web applications built by an AI coding agent, each seeded with a curated answer key of vulnerabilities. The benchmark's thesis: "Neo is the only scanner that can find architectural and business-logic vulnerabilities reliably."

App	Stack	Ground-truth vulns	Neo recall
vaultbank	FastAPI + PostgreSQL + React	30	27/30 (90%)
medportal	Next.js 14.1.4 + Prisma + NextAuth + PostgreSQL	20	17/20 (85%)
claimflow	SvelteKit + Drizzle ORM + SQLite	24	22/24 (92%)
Total		74	66/74 (89.2%)

The answer keys cover Critical business-logic flaws, authentication/authorization failures, IDOR, mass assignment, CSRF, weak crypto, and a long tail of Low/Informational findings.

We ran RedPick against the same three targets under the same methodology constraints (blackbox dynamic + whitebox code review, no docker exec, no answer-key lookups, non-destructive exploits with state preservation).

---

Headline results

Per-app comparison

App	Our ground-truth	Neo ground-truth	Additional confirmed	Our FP	Neo FP
vaultbank	30/30	27/30	29	0	0
medportal	20/20	17/20	39	0	3
claimflow	24/24	22/24	10	0	2
TOTAL	74/74	66/74	78	0	5

Our 0-FP figure is real: every one of the 78 additional findings passed a structural completeness gate (CVSS 4.0, severity, working PoC, CWE-or-endpoint, non-duplicate), a dedicated verification pass that replayed each PoC on the live target, and a source-code reconciliation that pinned every finding to a specific file and line at commit 17568bbb.

Full-coverage view

App	Total confirmed	Ratio vs answer key	Neo's share
vaultbank	59	2.0×	27/59 = 46%
medportal	59	3.0×	17/59 = 29%
claimflow	34	1.4×	22/34 = 65%
TOTAL	152	2.1×	66/152 = 43%

2.1× the benchmark catalog in confirmed, source-reconciled findings — all at 100% precision. On this view Neo covers 43% of what is demonstrably in these apps. The headline 89.2% ground-truth recall is a function of the answer key being finite, not of the apps being thoroughly covered.

---

Beyond the ground truth

The 8 answer-key findings Neo did not report

ID	App	Severity	Title
VB-016	vaultbank	Low	Refresh Token Endpoint Has No Rate Limiting
VB-017	vaultbank	Low	No Account Lockout After Failed Login Attempts
VB-018	vaultbank	Low	No Audit Logging for Privileged Operations
MED-017	medportal	Info	Outdated JavaScript Libraries
MED-018	medportal	Info	X-Content-Type-Options Not Implemented
MED-019	medportal	Info	Verbose Error Messages Leak Internal State
CF-023	claimflow	Info	Generic Email Address Disclosure
CF-024	claimflow	Info	X-Content-Type-Options Not Implemented

Seven of the eight are Low/Informational — the low-severity tail that is expensive in agent turns relative to its score contribution. Single-pass architectures typically trim it. Our architecture runs Low/Info coverage in dedicated parallel agents that the main orchestrator never has to interrupt — they run concurrently, finish independently, and get verified automatically.

The 78 additional findings — severity distribution

Severity	vaultbank	medportal	claimflow	Total
Critical	1	3	1	5
High	6	17	0	23
Medium	9	13	4	26
Low	10	4	3	17
Info	3	2	2	7
Total	29	39	10	78

5 additional Critical vulnerabilities and 23 additional High — each code-backed, each with a working PoC, each pinned to a specific source file and line.

The most impactful findings (top 8)

#	App	Severity	Title	Source
E-VB-01	vaultbank	Critical	Teller Can Deposit/Withdraw to Own Accounts — No Self-Transaction Check	teller.py:113,157
E-MED-01	medportal	Critical	Patient Can Overwrite userId to Hijack Admin Account	patients/[id]/route.ts:29
E-MED-02	medportal	Critical	Nurse Can Create Prescriptions (Medical Safety Boundary Bypass)	prescriptions/route.ts
E-MED-03	medportal	Critical	Patient Self-Modification Enables Allergy Removal Chain	patients/[id]/route.ts:29
E-CF-01	claimflow	Critical	Cross-User Claim Action Execution via Direct Object Reference	adjuster/claims/[id]/+page.server.ts
E-VB-02	vaultbank	High	Branch Manager Transfers Bypass Approval Thresholds	transactions.py
E-VB-05	vaultbank	High	Race Condition in Teller Deposit Creates Phantom Transactions	teller.py
E-MED-14	medportal	High	Lab Tech Can Create Arbitrary Notifications (incl. Admin)	notifications/route.ts

Full finding tables for all 152 confirmed findings (ground-truth + additional, per app):

• vaultbank: 30 GT + 29 additional = 59 findings (CSV)
• medportal: 20 GT + 39 additional = 59 findings (CSV)
• claimflow: 24 GT + 10 additional = 34 findings (CSV)

---

Where the additional coverage landed

Neo is a strong baseline, and ProjectDiscovery's answer key captures 74 well-scoped issues across the three apps. The additional 78 findings don't reopen the Neo score — Neo is judged against the 74-item ground truth, which is the right scope for a benchmark. What our run surfaced extends into surface areas outside the ground truth: workflow authorization, multi-role business logic, ORM-level mass assignment, framework-specific serialization leaks, and chained attack primitives.

vaultbank — teller/manager role abuse

The additional vaultbank surface is dominated by findings that require reasoning about multi-role authorization boundaries at the action layer rather than the page layer:

• Teller self-transaction (Critical) — teller.py:113,157. A teller can deposit into and withdraw from their own account because the endpoints never compare the teller's user id against the target account owner. This unlocks the teller-role chains.
• Branch-manager transfers bypass dual-approval thresholds (High) — transactions.py. The customer role goes through the approval gate; the branch-manager role takes a different code path that writes the transaction directly.
• Race condition in teller deposit (High) — concurrent deposits against the same account produce duplicate credit rows because the write path lacks row-level locking.
• Bill-payment and transfer-splitting bypass approval threshold (High/Medium) — two independent bypass paths.

medportal — ORM-level mass assignment (3.0× expansion)

The medportal expansion centers on a structural mass-assignment pattern: nearly every POST/PATCH handler passes the request body directly into prisma.[model].update({ data: body }) or prisma.[model].create({ data: body }), meaning every writable field in the Prisma schema is writable by every authenticated role.

• Patient can overwrite userId to hijack admin account (Critical) — patients/[id]/route.ts:29. PATCH /api/patients/:id forwards the request body into Prisma's spread. userId is writable by the patient. This is the primitive behind CHAIN-001 below.
• Lab-tech can create arbitrary notifications for any user (High) — notifications/route.ts.
• Cross-role appointment modification (High) — appointments/[id]/route.ts.
• Medical-records mass assignment (High) — medical-records/[id]/route.ts.

claimflow — action-layer gaps + framework-specific leaks

• Cross-user claim actions via direct object reference (Critical) — adjuster/claims/[id]/+page.server.ts:52,119,142,203. recommendPayout, addNote, requestDocuments, settlement/calculate all accept arbitrary [id] without scoping to assigned claims.
• Registration side-effect on error path (Medium) — register/+page.server.ts:61-64. Account is created even when the endpoint returns a 500 "registration failed" error.
• Admin user-management actions missing server-side role check (Critical, GT CF-001) — the admin page gates the UI on role, but form actions do not re-verify server-side.

The common pattern

There's a single recurring shape: AI-generated code places role checks at the page/UI layer and passes request bodies straight into ORM write calls at the action/API layer. Those two patterns together — UI-layer authorization plus full-body spread into update(...) — create most of the additional surface.

For anyone planning a review of similar AI-generated codebases: the interesting attack surface is almost never the login page; it's the +page.server.ts / route.ts action handler one level below it.

---

Attack chains

Individual findings are not the point; the point is what they compose into. Ten distinct end-to-end attack chains surfaced in this run.

vaultbank — teller/manager compositional chains

Three chains (E-VB-01/02/03 + threshold-bypass + race-condition) collectively let a teller write themselves unbounded balance and, via transfer splitting, exfiltrate it without hitting any approval gate.

medportal — 4 chains

ID	Severity	Chain summary
CHAIN-001	Critical	Patient mass assignment → admin account hijack
CHAIN-003	Critical	Patient allergy removal → unsafe prescription approved
CHAIN-004	High	CSRF + mass assignment → cross-origin medical record tampering
CHAIN-005	Critical	Default credentials + no rate limiting + hash disclosure → full compromise

CHAIN-001 is the sharpest illustration of the mass-assignment pattern: the patient modifies their own userId in PATCH /api/patients/:id, associating their patient profile with the admin's user account. Combined with the lack of ownership checks on every medical endpoint, the patient now reads and writes the admin's medical data, appointments, and prescriptions. Three individually Medium findings compose into a Critical privilege-boundary breach.

claimflow — 3 chains

ID	Severity	Chain summary
CHAIN-001	Critical	Password hash exposure (__data.json) + weak hashing + weak policy = mass account takeover
CHAIN-002	Critical	Unauthenticated data endpoints + broken access control = full data exfiltration
CHAIN-003	Critical	Auth weakness + BFLA + privilege escalation = admin compromise

CHAIN-001 leans on a SvelteKit-specific behavior: every authenticated route exposes a __data.json sibling that returns the server's serialized page-load data, including password hashes that the HTML page had already sanitized out. Chained with SHA-256 + static salt (CF-015), offline cracking mass-compromises every account.

---

Methodology

Four architectural decisions separate this run from a conventional single-agent scan:

1. Parallel multi-agent testing with free-roaming coverage — scoped vulnerability-class agents + a separate unconstrained agent that hunts beyond the ground truth
2. Strict finding classification — confirmed / near-duplicate / observation, with only confirmed findings counted (this is what 0 FP means)
3. Sink-pinned code reconciliation — every additional finding traced back to a specific file and line at commit 17568bbb
4. Dual verification — two full verification passes catch edge cases a single pass misses

Full methodology details including the 41 supporting observations (stored-but-no-sink, near-duplicates, unconfirmed by source review), environment integrity gates, and resilient execution: see methodology-notes.md on GitHub.

---

PR to ProjectDiscovery

We submitted a curated subset of 10 of these additional findings — all Critical or High, all with source-code evidence — to ProjectDiscovery's public benchmark repository as PR #17 for community validation.

#	PR ID	App	Severity	Title
1	VB-048	vaultbank	Critical	Teller self-transaction without controls
2	VB-049	vaultbank	High	Branch manager transfer approval bypass
3	VB-050	vaultbank	High	Bill payment threshold bypass
4	VB-051	vaultbank	High	Loan approval workflow bypass
5	VB-052	vaultbank	High	Race condition in teller deposits
6	MED-034	medportal	Critical	Patient mass assignment account hijack
7	MED-035	medportal	High	Appointment IDOR vulnerability
8	MED-036	medportal	High	Share link revocation bypass
9	MED-037	medportal	High	Lab order workflow state bypass
10	CF-037	claimflow	Critical	Cross-user claim action IDOR

Distribution: 5 vaultbank · 4 medportal · 1 claimflow · 10 total (3 Critical + 7 High) · PR open for community review.

---

Ready to see what RedPick finds?

Every one of the 152 findings documented above has a full raw HTTP request and response, a working PoC, and the source file and line where the vulnerable pattern lives. The techniques validated on Vibe-Coding — mass-assignment detection, multi-role authorization analysis, sink-pinned code reconciliation, chain composition — are the same techniques RedPick applies in production engagements.

Request a demo to run a proof-of-concept assessment against your application, or explore RedPick's managed pentest. For the full benchmark program see all benchmark results.

Related: 104/104 on XBOW · The AI Attacker Era · Agentic vs Automated Pentesting

— the RedPick team, 2026-04-11