How to read a pentest report without being impressed
A report with twenty critical findings does not necessarily describe an insecure organization. It may describe a badly configured scanner. Six questions for judging the quality of what you were handed.
If your organization buys penetration tests, at some point you will compare two reports from two different vendors and reach an uncomfortable conclusion: the thicker report with more critical findings is usually the worse of the two.
It is not a paradox. It is what happens when the work is billed by volume of findings rather than depth of analysis. These are the six questions we use to judge report quality — including our own.
1. Was each finding demonstrated, or only detected?
A scanner reports that a server exposes a version with a known vulnerability. That is a detection. A real test confirms that the vulnerability is exploitable in that specific configuration and documents the result. That is a demonstration.
The difference matters because a high proportion of automated detections are not exploitable in context: the service sits behind another control, the vulnerable function is not in use, or the version was patched without changing the banner the scanner reads.
What to look for: every high or critical finding should carry concrete reproduction steps and evidence of the outcome. If it says “we recommend upgrading to the latest version” and nothing more, that is scanner output.
2. Do the severities reflect your business, or a generic score?
CVSS is useful as a common reference, but it is agnostic to your context. The same vulnerability can be critical on one system and practically irrelevant on another.
A good report adjusts severity for real exposure, asset value, and existing compensating controls — and explains the adjustment. If every severity in the report matches the CVE base score exactly, nobody thought about you.
3. Is there any chaining of findings?
This is where real work separates from automated work. Three medium findings, in isolation, look manageable. The same three chained — an information disclosure that reveals a username, a weak password policy that lets you guess it, and an admin interface without a second factor — constitute a direct path to administrator.
A scanner never sees that chain, because it evaluates each finding separately. If the report contains no chained attack narrative at all, probably nobody moved past the scanning phase.
4. Is the executive summary useful or decorative?
The test: hand it to someone in finance and ask them to tell you what needs fixing first and why. If they cannot, the summary is not doing its job.
A good executive summary is not the technical report with fewer words. It answers three things: what could happen to the business, how likely it is, and what to do in what order. No CVE identifiers in the first paragraph.
5. Is what was not tested stated plainly?
Every engagement has limits. Out-of-scope systems, restricted testing windows, environments that were down during execution, credentials delivered late. An honest report states its limitations explicitly, because an unprepared reader can take “no findings were reported” to mean “there are no findings.”
The absence of a limitations section does not mean there were none. It means nobody told you about them.
6. Is retesting included?
A remediated finding is not closed until someone verifies it. Partial fixes are common: the reported instance is fixed while three identical ones remain on systems that were not reviewed, or the patch introduces a different problem.
If retesting is quoted as a separate engagement, price it into the total from the start — because without it, the report describes a moment in the past and nothing more.
On the number of findings
It is worth saying directly: more findings is not better work.
A report with forty findings, thirty of which are minor HTTP header configurations, will consume weeks of your infrastructure team’s time to reduce very little real risk. A report with six findings, two of which describe complete paths to your most sensitive data, is worth considerably more — and is harder to produce.
When evaluating proposals, ask for an anonymized sample report. The structure of the deliverable tells you more about how a vendor works than any certification listed in the commercial proposal.