AI Pentesting Buyer's Checklist (2026)

Twelve questions to ask before you sign. Written for security teams buying a platform and for pentest firms deciding which engine to put behind their own name.

Before you buy an AI pentesting tool, ask for working exploit proof, the testing mode behind every benchmark figure, a documented false-positive method, results you can reproduce, production-safe controls and a retest path.

The agentic pentesting market has become crowded quickly. A few recent examples:

• XBOW launched Autonomous Exposure Management on September 24, 2026. Agents test an organization's external application estate in parallel, starting with no source code and no credentials.
• Hadrian raised a $40M Series B on October 6, 2026, for a platform that combines exposure management and agentic penetration testing.
• Edgescan introduced Atomic on October 7, 2026, an autonomous pentesting capability whose actions pass through a control layer the company calls the Edgescan Harness.
• Invicti published an agentic pentest benchmark on October 8, 2026, using the same Fider and Photoview versions Doyensec used to compare Aikido and XBOW.
• Antigen, as Founderland reported on October 9, 2026, has launched tCell, an agent that finds and exploits issues, proposes fixes as pull requests and then retests. The same report lists Snyk, Synack, YesWeHack and Pentera among vendors that announced similar products between May and August 2026.

With this many products using the same vocabulary ("AI", "agentic", "validated"), the labels alone no longer help a buyer tell them apart. The questions below are the ones we would ask if we were buying, and the ones we expect to be asked about RedPick.

---

1. Is every finding backed by a working exploit?

This is the most important question. A scanner usually reports that something looks vulnerable, although some scanners confirm a subset of issues automatically. A pentest is expected to prove that something is exploitable. Ask for a sample finding and check that it contains:

• the exact HTTP request(s) that trigger the issue
• the response that shows the impact, such as returned data or a changed state, rather than an error message alone
• a baseline comparison: the same request made by a legitimate user, so the difference is visible

If the only "proof" is a severity score and a CWE link, the finding hasn't been verified, whatever the product is called.

Some valid findings have no exploit to show, such as a missing security header, and some shouldn't be exploited in production, such as an injection in a DELETE statement (see question 8). For those, the report should say so and state what evidence the finding relies on instead.

For example, suppose the vendor reports an insecure direct object reference on GET /api/invoices/{id}. A finding you can trust shows three requests side by side: user A requesting invoice 1001 (their own) and getting 200; user B, a different tenant, requesting the same invoice 1001 and also getting 200 with A's customer name and amount in the body; and user B requesting an invoice of their own, to show the session works normally. Anyone on your team can replay those three requests in five minutes and see the problem. A finding that says "Possible IDOR detected on /api/invoices (High)" leaves your developers to work out whether the issue is real.

NIST draws the same distinction in SP 800-115, its 2008 technical guide to security testing: vulnerability scanners check only for the possible existence of a vulnerability, while a penetration test exploits it to confirm that it exists.

2. Which testing mode produced the headline number?

Benchmarks have become a common way to market these products, and a score means little without the testing mode behind it. The same application tested black-box (a URL, sometimes with test accounts), grey-box (credentials plus some documentation) or source-assisted (the repository is paired or uploaded) is a different exercise each time.

Doyensec's Aikido vs XBOW comparison configured both platforms with source code and user credentials. Invicti's October benchmark combined authenticated black-box runs with source-assisted ones. RedPick's run on the same two apps was black-box with credentials only. All three are legitimate setups, but their numbers can't be compared directly. Ask the vendor to state the mode next to every number.

The mode changes the result in predictable ways. With source access, an engine can read the routing table, spot an unauthenticated handler and go straight to it. Without source, it has to discover the same endpoint by crawling, guessing or reading client-side JavaScript. Source access usually broadens coverage, and the PCI SSC guidance cited in question 3 notes that white-box and grey-box tests generally give more accurate results than pure black-box ones. Black-box testing is closer to what an outside attacker sees, and it is the only option when the code can't be shared. The mistake to avoid is comparing a source-assisted score from vendor A with a black-box score from vendor B and concluding that A is stronger.

3. Does it work without your source code?

Many organizations can't hand a repository to a third-party SaaS, because of contractual restrictions, acquired code, vendor-supplied applications or internal policy. Ask whether the product delivers useful results in pure black-box mode, and ask for a benchmark that was run that way.

Also ask what "black-box" means to them. Usage varies: the PCI SSC's penetration testing guidance keeps the term for tests where the client provides no information in advance, while many vendors, including us, say "black-box with credentials" for a run that gets a URL and test accounts but no source code or documentation. Some vendors also expect an OpenAPI file and a role matrix. That is a reasonable setup, but it is closer to grey-box. Make sure the setup in your trial matches the setup behind the numbers you were shown.

4. How are false positives measured, and by whom?

A claim of "zero false positives" is only as good as the way it was measured: who reviewed the findings, and against what. Ask:

• Who validated the findings: the vendor, a third party, or the benchmark author?
• Were findings outside the answer key checked one by one, or counted as true or false positives without review?
• Is the raw data (findings list, evidence) available?
• How are duplicates handled? Ten reports of the same missing check on ten endpoints can be one root cause or ten findings, and the choice moves the totals a lot.

During a trial you can measure this yourself. Take every finding rated High or Critical, give it to one engineer for an afternoon, and record how many reproduce from the evidence alone. That ratio is a better guide than any headline figure, and it also shows how much triage time the tool will cost your team.

We apply the same standard to ourselves. Our benchmark posts report raw and patched results separately (Argus) and say where our numbers are self-validated rather than externally reviewed (Doyensec apps).

5. Can I reproduce the benchmark?

Public benchmarks such as XBOW's validation suite, PortSwigger Web Security Academy labs and HackBench can be run by anyone (our results: XBOW, PortSwigger, HackBench). A score on a private, vendor-built benchmark is much harder to check, so prefer results on targets you could run yourself.

Keep in mind that most public targets come with public solutions, which are likely to be in a model's training data. XBOW now describes its own suite as saturated. A good score on a public benchmark shows that a tool works; it says less about how the tool handles an application it has never seen.

That is why the best trial target is an application you know well: a staging build of your own product that still contains a few issues found in past pentests. Because you know the answer key, you can check both what the tool finds and what it gets wrong. If you can't use your own code, OWASP Juice Shop is a common choice, with the caveat that its solutions are published too.

6. Does it find business logic flaws, or just injection?

Injection, reflected XSS and known CVEs are the classes scanners handle best. Many of the most damaging application flaws sit in authorization and workflow logic:

• one tenant reading another tenant's data
• a teller who can approve their own transfer
• a race condition in a payment flow

Broken access control is the top category in both the 2021 and 2025 editions of the OWASP Top 10, and it is also where pattern matching helps least, because the request looks normal and only the context makes it wrong. Ask for example findings in the access-control and business-logic classes specifically. Our vibe-coding benchmark write-up shows what that looks like in practice.

7. Does it chain findings?

Several lower-severity findings can add up to a serious one. Ask whether the tool combines findings into attack chains and rates the chain as a whole, or lists isolated issues and leaves that work to you.

A typical example: an endpoint returns any user's profile, including email address and phone number, when given that user's ID, but the IDs are random UUIDs, so the issue is often rated Low. Separately, the comments API includes each author's UUID in its responses (Informational). Taken one at a time, these go into the backlog. Together, they let any logged-in user collect UUIDs from the comments and read every commenter's profile, which is a large-scale exposure of personal data. A tool that reports only the separate pieces leaves the most important conclusion out of the report.

8. What are the production-safety guarantees?

Because agents choose their own payloads, ask specifically:

• How is SQL injection confirmed? The safe technique depends on the statement the input ends up in: a boolean condition that is harmless in a SELECT can rewrite or delete every row when the vulnerable query is an UPDATE or DELETE. Ask how the tool works out the statement type before choosing a payload, and whether stacked queries are ever sent to production.
• What payloads are used to confirm XSS?
• Does testing back off when the application throttles?
• What happens to artifacts (uploaded files, stored payloads) after testing?
• How is scope enforced? What stops an agent from following a link off-scope?

Edgescan, for example, describes its harness as a layer between the model's reasoning and the actions it requests, enforcing scope, permissions and approved operations through deterministic controls. Whatever the design, ask where those limits are enforced and whether each action is logged.

Write the answers into the contract or the rules of engagement. The UK NCSC's guidance on penetration testing recommends agreeing scope and constraints in a scoping phase before testing, and keeping a technical point of contact available throughout the test. The same applies when the tester is software. In practice, that means a named contact on your side, a way to stop the test that you control, and a defined window for anything that touches production.

9. Is there an audit trail?

For regulated buyers, an audit trail is the simplest way to show what was tested and how. Under DORA, financial entities other than microenterprises must ensure, at least yearly, that appropriate tests are conducted on all ICT systems and applications supporting critical or important functions (Regulation (EU) 2022/2554, Article 24(6)), as part of a documented testing programme. Ask whether every agent action is logged and exportable, and whether the report is fit to hand to an auditor.

DORA is not the only driver. PCI DSS v4.0.1 requirement 11.4 asks for a defined penetration testing methodology, testing at least once every 12 months and after any significant change, and retention of the results for at least 12 months. An exportable log of what was tested, when and with which identity helps an assessor see that the methodology was followed. It supports the assessor's review; it doesn't replace the qualified, independent tester the requirement calls for.

10. What does the report look like for each audience?

Each audience needs something different from the report: developers need reproduction steps and remediation guidance, management needs an executive summary and a view of the risk, and pentest firms reselling the service need a report they can review, edit and sign. Ask for a sample report from a real, sanitized engagement.

Check the details: does every finding map to a standard reference such as a CWE or an OWASP WSTG test case? Is the severity explained, or just a color? Can you export findings to your tracker (JSON, CSV or a Jira integration) rather than only as a PDF? If you are a pentest firm, can you add your own branding and commentary, and is it clear which parts were reviewed by a human?

11. Which application types does it actually cover?

Coverage claims often refer to web applications only. If you also ship APIs (REST, GraphQL), mobile apps, desktop clients or LLM-powered features, ask for evidence for each type. For LLM features, ask how the product tests the OWASP Top 10 for LLM Applications (our LLM security testing guide maps the tests) and whether it has results on a live agent target such as HackTheAgent.

12. How does it fit the way you already work?

Ask how findings reach the people who fix them: CI/CD triggers, re-testing after a fix, and whether retest results are tracked against the original finding.

Retesting is often the weak point. A ticket gets closed, nobody repeats the original attack, and it later turns out that the fix was incomplete or has regressed. A good tool reruns the original exploit, tries close variants of it (other parameters, encodings or roles), and attaches the new requests and responses as evidence that the issue is closed.

---

Red flags during a demo

A few answers deserve a closer look:

• "Our AI doesn't produce false positives," with no method behind it. Any tool can report something that doesn't hold up. A useful answer explains how findings are filtered, who checked them, on which targets, and how many got through.
• Only aggregate numbers. If you can't see a single finding with full evidence, you can't judge quality.
• A benchmark with no named targets or mode. "We found 3x more vulnerabilities" means little without the application, version and setup.
• No way to stop a running test. For anything that touches production, you need to be able to stop the test at any time.

How to run a two-week evaluation

The plan below fits a two-week trial for a busy security team:

1. Day 1–2: pick two targets: one you know well (a staging build that still contains issues from past pentests) and, as a common reference point, one public app such as Juice Shop, keeping in mind that its solutions are published. Agree scope and rules of engagement in writing.
2. Day 3–7: run each shortlisted tool in the same mode and scope, with equivalent accounts for the same roles, one tool at a time, resetting the target between runs. Don't tune one tool more than the others.
3. Day 8–10: have an engineer try to reproduce every High and Critical finding from the evidence alone. Note how long each one takes.
4. Day 11–12: fix one or two real issues and ask each tool to retest. Check that it reruns the original exploit and tries variants of it.
5. Day 13–14: score each tool against the table below and share the evidence with whoever signs the purchase.

---

A one-page scorecard

#	Question	Evidence to request
1	Exploit evidence per finding	Sample finding with request/response and baseline
2	Testing mode per number	Mode stated next to every benchmark figure
3	Works without source	Black-box benchmark result
4	False-positive methodology	Who validated, against what
5	Reproducibility	Trial on your own application, plus public benchmark results
6	Business logic coverage	Example access-control and workflow findings
7	Chaining	Example chain with composed severity
8	Production safety	Written description of verification techniques
9	Audit trail	Exportable action log
10	Reporting	Sanitized sample report
11	Application types	Per-type evidence
12	Workflow fit	CI/CD and retest flow demo

---

Related: Platform · Benchmarks · LLM security testing guide