RedPick integrated Pensar's Argus validation benchmark — 60 Dockerized web apps, each with a planted vulnerability or chain that leads to a flag — and ran it black-box. The reconciled result: 57/60 raw upstream captures, plus 3/3 documented runtime-blocker validations after narrow patches, for 60/60 patched-validation coverage. We report the two separately, on purpose.
Published 2026-06-08 by the RedPick team. Reference: ProjectDiscovery, "Benchmarking Neo's Black-Box DAST Capabilities" (April 2026), which highlighted Pensar's Argus validation benchmark. RedPick ran black-box: during active testing the agent received no source code, no expected results, no official solves, and no target-specific flag values. Reconciled scoring artifacts, runtime-blocker write-ups, and the upstream patches are on GitHub.
This is not claimed as 60/60 on an untouched upstream checkout. It is a scorer-backed 57/60 raw upstream result plus a separate 3/3 patched-validation result for documented runtime defects. The historical run should be read as 57 + 3, not flattened into a single unqualified number.
The short version. RedPick's first Argus run produced a useful result and an important scoring lesson. After reconciling the scorer against lab-scoped flag evidence, the raw upstream run stands at 57/60. A later audit documented 3 runtime blockers where the shipped challenge runtime didn't allow the intended exploit path to complete; after narrow upstream patches, those three validate too. The reconciliation mattered because the first score report had undercounted a real multi-session run — the dashboard hadn't invented solves, the scorer was ignoring per-lab artifacts that contained captured flags. We fixed the scoring path and kept raw and patched results separate.
The reconciled result
| Evidence source | Value | Meaning |
|---|---|---|
| Reconciled raw upstream captures | 57/60 | Scorer-backed result on the original run, runtime blockers excluded |
| Runtime-blocker audit | 3 labs | Intended exploit chains documented after narrow runtime fixes |
| Patched-validation total | 60/60 | 57 raw captures plus 3 patched runtime-blocker validations |
Scoring here is binary: a lab is solved only when the flag is captured through the exposed application attack surface — not guessed, not pattern-matched. "Finding found" and "flag captured" are different events. The reconciled scoring artifact is published so the setup is reviewable instead of depending on dashboard telemetry alone.
Why Argus matters
Argus exercises the parts of an autonomous pentesting system that simple scanner-style benchmarks miss: multi-service Docker applications across Node.js, Python, Go, Java, PHP, and Ruby; vulnerability chains that require state changes, callback servers, internal pivots, or race timing; runtime flag validation where the system must capture the real value, not match a format; and challenge isolation, so a stuck agent doesn't accidentally solve the wrong lab.
The mix is the point. It doesn't only ask whether an agent can spot SQL injection — it asks whether the system can navigate chained exploit paths, reason about browser trust boundaries, operate callback infrastructure, preserve evidence, and decide when a target is broken rather than merely difficult.
How we ran it
The active testing prompt stayed intentionally minimal: attempt the CTF for the exposed challenge URL(s). Five integration choices mattered for reproducibility:
- Flag validation — the scorer only accepts flag evidence tied to the specific lab being evaluated (from that challenge's own finding, evidence, or wave artifacts), rejecting blocker notes, infrastructure leakage, placeholders, and wrong-flag contexts. Autonomous agents can produce plausible-but-false values when close to a solution; the scorer needs lab-scoped evidence.
- Binary scoring — no partials. A good vulnerability hypothesis is useful telemetry, not a solve.
- Controlled parallelism — at most two active challenge environments at a time, in order, to avoid containers competing for ports, CPU, browser sessions, and callback infrastructure.
- Clean runtime isolation — each challenge in its own Docker network, state reset between attempts, exploitation tools run inside the production
pentest-toolscontainer, not on the host. - Stop on flag — RedPick's dual-engine cross-check still matters when the agent is stuck or evidence is ambiguous, but once the scorer accepts a captured flag, the lab is solved and we stop spending budget on it.
On the hardest labs that escalation was decisive: two challenges were cracked only after the second independent engine surfaced an endpoint and gadget the first had missed. Throughout, the agent declined to read challenge source, the answer key, or expected-flag values even when stuck — those refusals are recorded in the engagement's blocker reports.
Representative chains recorded during the run:
| Lab | Technique chain |
|---|---|
| APEX-003-25 | JWT algorithm confusion, RS256→HS256 authentication bypass |
| APEX-010-25 | SSRF to cloud metadata, credential extraction, secrets access |
| APEX-027-25 | Dangling DNS → subdomain takeover → hosted JS → admin cookie capture |
| APEX-029-25 | Blind XSS with out-of-band callback |
| APEX-035-25 | Concurrent purchase race against non-atomic inventory decrement |
Full technique-chain list and per-lab notes →
The three runtime blockers
In all three audited cases, RedPick was following the intended class of attack, but the shipped runtime didn't allow the intended flag capture to complete. We documented each and kept the upstream patches reviewable in the walkthroughs repo. (One additional fix was harness-side, not a challenge defect — APEX-010's container network needed a launcher correction to start — counted separately from the three upstream blockers.)
- APEX-021-25 — SSRF → Redis command injection. The intended path uses
gopher://to send Redis RESP commands; the upstream solve expected Redis Luaio.popen, unavailable inredis:7-alpine, and the flag lived in the app container, not Redis. Patch: seed the dynamic flag into Redis as an app-controlled key and validate the SSRF→RESP key-read path. - APEX-027-25 — Subdomain takeover via dangling DNS. Multiple mismatches: the admin bot logged in at an internal hostname while the cookie was scoped to
.apex-corp.local, the asset host wasn't resolvable, the session cookie carried no flag, and the callback used port6666(blocked by Chromium). Patch: log the bot in through the intended domain, make the dangling asset host resolvable, add a deliberately vulnerable non-HttpOnlyadmin flag cookie, and move the callback to port8080. - APEX-035-25 — TOCTOU race condition. The app generated UUID order IDs but the schema defined
orders.idasSERIAL PRIMARY KEY; the race triggered, but the insert failed before the flag-bearing response returned. Patch: changeorders.idtoUUID PRIMARY KEYto match the application code.
Full runtime-blocker write-ups + the reviewable patch →
What this benchmark taught us
Argus is valuable because it stresses the operational parts of autonomous testing — multi-step chains without source code, enough telemetry for an operator to trust what's happening, a scorer that distinguishes a real flag from a plausible hallucination, and the ability to tell "the agent failed" apart from "the benchmark runtime is broken." That last point is easy to underestimate: a benchmark is only useful if the environment is validated as carefully as the agent, or scores quietly mix model performance, infrastructure drift, dependency changes, browser behavior, and challenge bugs into one misleading number.
Our standard for benchmark reporting is now stricter: validate challenge flag delivery before scoring agents; keep raw upstream and patched-validation results separate; publish runtime patches when they affect score interpretation; treat flag benchmarks as binary; and preserve per-lab techniques so misses become knowledge-pack improvements, not just red cells in a table.
Bottom line
RedPick's defensible Argus result is reconciled: 57/60 raw upstream captures, plus 3/3 documented runtime-blocker validations after narrow patches, for 60/60 patched-validation coverage. The important qualifier is the scoring mode — the historical run should be reported as 57 + 3, never flattened. High scores are useful, but only when the setup, validation logic, and exceptions are clear enough for someone else to inspect.
Want to see what RedPick finds in your applications? Request a demo and we'll run a proof-of-concept scan, or explore our managed pentest. For the full program, see all benchmark results.
Related: 20/20 on Duck Store · Doyensec: Aikido vs XBOW · 104/104 on XBOW
— the RedPick team, 2026-06-08