Use this checklist to compare application security testing options: pentest firms, scanners and AI pentesting agents. The short version: ask what the tool can test while signed in as different users, how each finding is proven, what you get to recheck yourself, and how fixes reach your code. Price and report format matter, but they are easy to compare. Coverage and evidence are not, so most of the questions below are about those.
Updated October 2026
Coverage
- [ ] Which vulnerability classes are tested? Ask for a list, not "the OWASP Top 10."
- [ ] Does it test while signed in? How many roles and accounts at once? See authenticated testing.
- [ ] Does it test cross-tenant access if your app is multi-tenant?
- [ ] Does it cover APIs (REST, GraphQL) as well as pages?
- [ ] Can it work from source code, the running app, or both?
- [ ] Does the report state what was covered: which roles, pages and endpoints?
Access control and business logic depth
Broken access control is the top OWASP risk and where tools differ most.
- [ ] How does it find IDOR? Does it use a second real account, or only guess IDs?
- [ ] How does it decide whether a response is a leak or an expected result?
- [ ] Can it find privilege escalation between roles and tenants?
- [ ] Does it test business logic and race conditions, or only injection?
Evidence and false positives
- [ ] Does each finding include exact steps, requests and responses?
- [ ] Is each finding reproduced by a separate step before you see it?
- [ ] Is there a control case showing the difference is real?
- [ ] Is severity given with a standard score such as CVSS 3.1, with the vector?
- [ ] In a trial, what share of findings did your team reject?
Fixing and retesting
- [ ] Does it suggest a fix in your code, and can it open a pull request?
- [ ] Can you retest a single finding after a fix, without waiting for the next full test?
- [ ] Does it send findings to your ticketing and chat tools?
Frequency and timing
- [ ] Can it run on every pull request, nightly or when new hosts appear?
- [ ] How long from start to first finding?
- [ ] For a human pentest: how far ahead must you book, and how long is the test window? See continuous vs annual pentesting.
Safety and data
- [ ] How is scope enforced? What stops traffic to hosts you did not approve?
- [ ] Are there rate limits to protect your app?
- [ ] Where do test credentials and captured traffic live, and how are customers isolated?
- [ ] Which security certifications and documents can the vendor share today? Ask for the documents, not a logo.
Commercials
- [ ] What is the pricing unit: per test, per app, per seat, per year?
- [ ] What does a retest cost?
- [ ] Is a trial possible on your own app? See pentest cost in 2026 for public price ranges.
How to run a fair trial
Pick one staging app. Create the same test accounts (two per role, two tenants) for every vendor. If you can, plant two or three known bugs, including one IDOR. Run each option for the same period and compare found, missed and rejected findings side by side.
Red flags
- The vendor cannot say which roles and endpoints were tested.
- Findings arrive without requests and responses, only a description.
- Every finding is rated critical or high.
- Access control testing means "we try sequential IDs" with one account.
- Retests are slow or need a new engagement.
How Wakefield helps
Wakefield agents sign in as every role, build a model of intended access, and try to cross user, tenant and role lines. A separate verifier replays each finding with a live exploit and a negative control; anything it cannot reproduce does not reach you. Fixes come with a confidence level and open as a pull request in one click. See the platform.