Penetration testing has long been held up as proof that an organisation is serious about security. Yet the way most pen tests are scoped, delivered, and consumed today often leaves businesses with a false sense of confidence.
We see it time and time again at Chaleit. An organisation commissions a test, gets back a long list of findings, ties up their IT and development teams for months and still remains vulnerable to the threats that matter most.
The problem isn’t intent but scope. Or as David O’Neil, an accomplished CISO we recently spoke with, calls it: “the murky scope.”
The false comfort of the checkbox
Organisations ask for pen tests for many reasons. Some are driven by security compliance: ISO, SOC, customer demands. Others simply want reassurance, proof that they’re not missing anything obvious.
David explains the situation well:
“I use the traditional term, which is to have somebody try to hack us. But what’s really wanted is an understanding of where we’re vulnerable and what the actual problem is.”
That distinction matters. A compliance-driven pen test may satisfy auditors, but it rarely mirrors how attackers actually behave. A skilled attacker isn’t going to follow a scripted sequence of steps. They’ll look for the easiest and fastest way in, the lowest effort route to the biggest payoff.
When the scope is defined by compliance templates or vague instructions like “test the application,” the outcome is predictable: lots of findings, few of which map to real-world risk.
Noise over insight
The result of a poorly defined test is almost always noise. Dozens or even hundreds of findings, many of which the internal team already knows about.
“If I’m just finding the same things that your team isn’t fixing because they’re not prioritising, or you’re not actually scanning the right things, then I’m identifying process gaps, not meaningful vulnerabilities,” David points out.
This is why development teams lose faith in pen testing reports. They’re handed stacks of issues that consume time but don’t address the real risks. It’s a cycle that creates frustration without reducing exposure.
Chaleit’s own experience backs this up. In one recent case, we found that a global client’s multi-million-dollar identity platform wasn’t enforcing multi-factor authentication correctly. On paper, MFA was in place. In reality, it might as well not have existed. That misconfiguration wouldn’t have shown up in a standard application-focused test, but it created an open door for attackers.
The wrong questions
So why does this keep happening? Partly because pen testing has become commoditised. Vendors package it into “standard SKUs”, predefined scopes that are easy to sell and deliver, but often irrelevant to the customer’s actual risks.
David has seen this first-hand:
“Out of the three larger vendors I went to, not one gave me a scope after four weeks. Why? Because it wasn’t a standard SKU. If it’s not a predefined product, they can’t respond.”
That reliance on predefined packages means clients aren’t asked the most important question: what is really bothering you?
When we took the time to ask one enterprise CISO that question, the answer wasn’t about APIs or applications. It was about identity. Working with them, we uncovered critical weaknesses affecting 50,000 users — weaknesses that would have been invisible in a “murky scope” engagement.
Context matters. Asking the right questions matters. Read more about this in our guide to modern penetration testing methodology.
Scope failure
The danger of mis-scoping plays out in live environments every day.
One FTSE 250 company we worked with had invested heavily in security. They had the technology, people, and processes. After four years of testing and hardening, they believed themselves almost unhackable.
Then we tried a different angle. Within hours, our team had gained domain administrator access — undetected. Their Security Operations Centre, equipped with the latest tools, only triggered two low-level alerts. In a real attack, the business could have been taken offline entirely. The SOC was the parachute, and it didn’t open.
This was a failure of scope. The “standard” tests had long since found nothing new. Only by working on the harder problem did the real exposure surface.
Too much data
Even when vulnerabilities are identified, another problem quickly emerges: volume.
Organisations are flooded with findings. Every scanner, monitoring platform, and pen test report adds more.
But as David points out, volume isn’t value:
“I can’t throw hundreds of vulnerabilities over the fence to developers and tell them to fix everything. The real issue is focus: understanding where the attackers can actually do damage.”
Frameworks like CVSS or DREAD attempt to prioritise findings, but they lack organisational context. A low-severity issue in the wrong place can be more dangerous than a high-severity one that isn’t exploitable. Without that context, teams are left spinning their wheels.
“Good enough” isn’t safe
For many executives, pen testing is a way of achieving “good enough.” The test is done, the report is filed, and the compliance box is ticked.
But that mentality is dangerous.
“From an executive level, there’s the concept that we can overinvest in security, so we settle on good enough. But if good enough means compliance is working, then we’ve missed the point,” David explains.
History offers plenty of examples. A decade ago, organisations rushed to buy data loss prevention tools because frameworks required them. The tools generated hundreds of thousands of alerts. Teams turned them off, but kept the products in place to maintain compliance. Security wasn’t improved; only the checkbox was satisfied.
Key takeaways
Pen testing, when done right, remains one of the most powerful tools for improving security. But only if it’s scoped to uncover the issues that actually matter.
Scoping is everything. Poorly defined tests generate noise and miss what matters.
Compliance isn’t security. Attestation letters don’t stop attackers. Real tests should mirror real-world threats.
Attackers think differently. They don’t follow scripts; they look for the quickest path to value. Pen testing should reflect that mindset.
Volume isn’t value. Hundreds of findings overwhelm teams. The priority must be what attackers can actually use.
Good enough isn’t safe. Compliance-driven testing gives comfort but not protection. Real security requires uplift, not just a report.
That’s the business we’re in at Chaleit: finding what others miss, and fixing it properly.
