The way attackers find vulnerabilities is changing — and the change is accelerating. AI agents that were research curiosities a year ago are now finding real zero-days in production software. Defenders still relying on yesterday's tools are increasingly testing against yesterday's attackers.
This post walks through what is publicly known about AI-powered offensive security, what it means for defenders, and why building defenses on the same kind of AI reasoning is no longer optional.
AI Offensive Tools Are Real
Three publicly documented research efforts make the picture clear.
Anthropic — Mythos. Anthropic has been publicly demonstrating the security research capabilities of its frontier models, including a research effort referred to as Mythos. The company has discussed the responsible-disclosure implications of releasing models capable of finding deep vulnerabilities in widely-used software, and has chosen restricted access for the most capable variants. The takeaway is not the marketing — it is the fact that a major frontier lab considers its model dangerous enough to restrict.
Google — Big Sleep / Project Naptime. Google's Project Zero published Project Naptime, a framework for evaluating LLM offensive capabilities. The follow-up effort, Big Sleep, was credited with finding an exploitable buffer underflow in SQLite — a real, production-grade C codebase. This was not a CTF challenge. It was an unmodified open-source project that is shipped on billions of devices.
OpenAI — Aardvark. OpenAI introduced Aardvark, an autonomous security research agent that has identified real CVEs in open-source software. The success rate published by OpenAI is high enough that the gap between "AI helps a researcher" and "AI does the research" is narrowing fast.
Academic validation. A 2024 paper, "LLM Agents can Autonomously Exploit One-day Vulnerabilities", demonstrated that GPT-4-class agents can autonomously exploit a high percentage of disclosed one-day vulnerabilities given only the public CVE description. This is not about finding new bugs — it is about closing the window between disclosure and exploitation, automatically.
Each of these is a separate project, from a separate organization, validating the same trajectory: AI agents are crossing from "interesting demo" to "produces real findings on real software."
What This Means for Defenders
The defender's playbook for the last two decades has been built around two assumptions:
- Time. There is a window between vulnerability disclosure and weaponized exploitation. Patch cycles, scan cycles, and audit cycles all assume that window is measured in days or weeks.
- Asymmetry of effort. Attackers spending months on a target was the cost of doing business. The cost was a natural rate-limiter on how many targets could be compromised per attacker.
Both assumptions are eroding.
The time window is shrinking. When disclosed CVEs can be exploited automatically by an agent that ingests the description, the gap between "patch released" and "patch needed" closes immediately. Defenders cannot rely on slow patch cycles. They cannot rely on annual penetration tests. They cannot rely on quarterly scans.
The asymmetry is flipping. An AI agent capable of running autonomous reconnaissance, fuzzing, exploitation, and lateral movement does not have the human cost that limited traditional attackers. Once the methodology exists and works, it scales. The same agent that tests one application can test thousands in parallel.
The implication is uncomfortable but unavoidable: defensive testing has to operate at the same speed and reasoning depth as the offensive AI it is preparing for. Manual testing alone cannot keep up. Pattern-matching scanners cannot keep up. Annual point-in-time assessments cannot keep up.
Why RedPick Is Built for This
RedPick was designed from day one around the same agentic reasoning model that powers offensive AI research. Rather than being a scanner with pattern rules, it is a system of specialist AI agents that reason about applications, plan multi-step attacks, and verify findings with working exploits.
Three properties matter for the AI-attacker era:
Reasoning over rules. Pattern-matching DAST scanners fail on anything that requires understanding context — business logic flaws, authorization chains, multi-step exploits. RedPick's agents reason about the application like a human pentester, but at the speed of automation.
Specialist agents per category. RedPick deploys specialist AI agents for each of 17 vulnerability categories, in parallel. Injection, authentication, access control, business logic, server-side request forgery — each is handled by an agent that understands the nuances of that class.
Verification, not probability. Every finding is verified with a working proof-of-concept. There are no "potential issues" or scanner alerts to triage. If RedPick reports a vulnerability, it has reproduced the exploitation. This is the same standard the offensive AI tools above are converging on, and it is the only way to keep pace.
The benchmark proof matters here. RedPick has scored 100% on every public benchmark for autonomous penetration testing it has been submitted to:
- 100% on XBOW — 104/104 challenges, black-box no-hint mode
- 100% on PortSwigger Web Security Academy — 270/270 labs across 31 vulnerability categories
- 100% on HackBench — 16/16 challenges
- 74/74 on ProjectDiscovery's Benchmark — plus 78 additional code-backed vulnerabilities beyond the answer key, zero false positives
These are not cherry-picked. They are the public benchmarks that exist for this category, and RedPick is built to keep pace with the frontier.
For LLM-specific security testing, RedPick also achieved 7/7 on HackMerlin — a progressive prompt injection benchmark with 4-layer defense at the highest tier.
For organizations evaluating defensive AI testing, our platform overview documents the full agentic methodology, and pricing covers the managed option, where senior pentesters run manual testing in parallel with the AI engine, along with how findings map to PCI DSS, NIS2, DORA, and SOC 2. For LLM-integrated applications, the new attack surface that didn't exist three years ago, see our application coverage.
What Defenders Should Do Tomorrow
The uncomfortable part of this shift is that the corrective actions are not new ideas — they are old recommendations that suddenly became load-bearing. The comfortable timelines that let teams defer them are gone. Five concrete moves matter now.
1. Inventory the attack surface you actually have, not the one in the asset register
AI-powered attackers enumerate faster than any defender's spreadsheet stays current. Dev environments exposed to the public internet, forgotten subdomains pointing at decommissioned services, staging clusters with production data, APIs deployed by a team that has since reorganized — these are the entry points that show up first in automated recon. The remediation is continuous external attack surface discovery, run on a schedule short enough that a new subdomain appearing on Monday is flagged before an AI agent finds it on Wednesday.
This is the easiest recommendation to implement and the most commonly neglected. Every successful breach retrospective includes some version of "we didn't know that host was there."
2. Move from annual to continuous application testing
Annual pentests are a compliance artifact. They produce a PDF that is stale by the time it is signed. In a threat model where disclosed CVEs are weaponized within hours of publication and business logic flaws can be reasoned about by agents operating autonomously, annual coverage is several orders of magnitude too slow.
Continuous does not mean more pentesters. It means automating the coverage that does not require human judgment (the 80% of findings that pattern-match known vulnerability classes), running it on every merge or on a daily cadence, and reserving human expertise for the 20% that requires creative threat modeling or specific business context. The economics only work if the automated layer is reasoning-based rather than signature-based — otherwise the noise drowns the signal.
3. Adopt reasoning-based testing for business logic and chained attacks
The vulnerabilities AI attackers find first are the ones traditional scanners cannot see — authorization chains, workflow bypasses, race conditions, multi-step exploits that require composing primitives into an outcome. If your pipeline catches SQL injection and missing security headers but not IDOR-chained-to-SSRF-to-privilege-escalation, you are testing for the 2015 threat model against the 2026 attacker.
The practical test: can your current tooling find a vulnerability that requires understanding that endpoint A's output is endpoint B's input, with a role boundary in between? If the answer is no, the gap is structural. See our agentic vs automated pentesting breakdown for the full picture on why signature-based tools cannot close this gap regardless of tuning.
4. Treat verification as non-negotiable
False positives are the silent multiplier on security team overhead. A pipeline that reports 500 findings where 50 are real wastes ten times the triage effort and, worse, trains the team to ignore alerts. The offensive AI tools that are setting the frontier (Big Sleep, Aardvark, Project Naptime) all converge on the same standard: a finding is not a finding until it is reproduced with a working exploit. Defenders should match that standard.
If your scanner cannot produce a working PoC for every alert, your pipeline is producing hypotheses, not findings. Engineering time spent chasing hypotheses does not reduce risk.
5. Close the LLM and agent surface now, not after the first incident
The fastest-growing attack surface in the next 18 months is LLM-integrated features — chatbots, AI assistants, RAG pipelines, agents with tool access. Most of these were shipped by product teams without security review, because the risk vocabulary ("prompt injection," "excessive agency," "indirect injection") is still outside the standard threat model playbook.
Every production LLM feature needs an adversarial assessment covering the OWASP LLM Top 10, tool permission audit, and end-to-end red team of at least one indirect-injection vector. Defer this and the first finding will come from an external researcher — or an attacker. Our LLM security testing guide has the full checklist.
These five moves compress into a single directive: assume that within a quarter, unknown actors will be running continuously-improving AI agents against your attack surface. The program that can keep pace is one where recon is continuous, application testing is automated with reasoning depth, findings are verified with exploits, and the LLM surface is audited before the first complaint.
The Choice Defenders Face
The era of AI-powered attackers is not arriving. It is here. Frontier labs are restricting their own tools because of how dangerous they are. Open-source AI security agents are finding real CVEs on a regular cadence. Academic papers are demonstrating one-day exploitation at high success rates.
Defenders have a choice. They can continue testing against yesterday's threat model — periodic scans, annual pentests, manual review for the most critical apps — and accept that the gap between attacker capability and defender testing will keep widening. Or they can adopt defensive AI testing that operates with the same reasoning, the same speed, and the same verification standards.
The first option is comfortable. The second option is the one that keeps the gap closed.
If you would like to see what that looks like for your applications, request a demo or explore RedPick's managed pentest option, where senior pentesters deliver complete engagements with the AI engine running in parallel.