The annual penetration test arrives like clockwork. Your security team books the engagement, defines the scope, and waits for the report filled with colour-coded vulnerabilities.
A few weeks later, you file the document away, tick the compliance box, and breathe a sigh of relief.
Job done, right?
Not quite. This ritual, repeated across countless organisations, represents one of cyber security's most persistent problems: treating penetration testing as a bureaucratic exercise rather than a strategic security tool.
To understand why this happens and what can be done about it, we spoke with Jim Newman, CISO at Capco, who has spent years wrestling with these challenges and developing better ways to make security testing genuinely helpful.
When pen testing becomes pointless
Most security frameworks mandate penetration testing with reassuring precision: applications and systems must be tested annually, and after major changes. What sounds like diligent security practice actually creates what Jim calls "minimum viable testing": organisations doing just enough to satisfy auditors rather than actually securing their systems.
This compliance-driven mindset creates a "single point in time view" of your cyber security posture.
Organisations design tightly scoped tests to generate favourable results, missing the broader picture of their actual risk exposure.
Meanwhile, the rapid pace of DevOps means that "major changes are happening under the hood all of the time" through continuous deployment pipelines, making annual assessments increasingly irrelevant. Cyber security expert Dan Haagman explores this timing problem in depth, questioning whether yearly pen tests are still relevant.
The result is security testing that satisfies auditors but leaves real vulnerabilities unaddressed.
However, perhaps more damaging than infrequent testing is the lack of context that plagues traditional penetration testing engagements. When Jim discusses security findings with his team or external auditors, his first words are often: "Context is key."
"Without context, you've just got a list of findings," he explains. "You can't go into vulnerability management just looking at findings, because that's overwhelming and without meaning. It doesn't tell you anything about the risk. Where is it running? How is it running? What else is there?"
Traditional pen testing reports arrive as isolated snapshots, divorced from the operational reality of how systems actually work together. Without understanding the broader architecture, business processes, and existing security controls, even critical-sounding vulnerabilities might pose minimal real-world risk, while seemingly minor issues could represent significant threats.
The scope assumption trap
One of the most dangerous aspects of traditional penetration testing lies in unstated assumptions about what will be tested. Both clients and testing firms operate under different expectations, creating blind spots that can prove catastrophic.
Picture this post-breach conversation:
"We got hacked. But you tested us." "We didn't test that bit." "Why not?" "You didn't ask us to... it wasn't in the scope." This scenario plays out more often than anyone wants to admit, highlighting how rigid scoping creates dangerous blind spots, Jim warns.
These gaps aren't just theoretical. Modern applications often include Lambda functions, APIs, and third-party integrations that might seem peripheral but can provide attack vectors. A tightly scoped application test might miss the publicly accessible Lambda functions that could drain cloud budgets through what security professionals call "denial of wallet" attacks.
Traditional penetration testing struggles particularly with business logic vulnerabilities: flaws that exploit intended functionality in unintended ways. These issues can't be detected by automated scanning tools and require a deep understanding of both the application's purpose and creative thinking about edge cases.
The reality is that attackers don't respect artificial boundaries. They exploit whatever they can find, regardless of whether it was included in your testing scope or considered during your threat modelling sessions.
If you're struggling with these scope and procurement challenges, our comprehensive Guide to Penetration Testing 2025 explores practical strategies for getting better results from your security testing investments.
The partnership alternative
Rather than accepting these limitations, some organisations are rethinking their entire relationship with penetration testing providers. Instead of discrete, project-based engagements, they're establishing ongoing partnerships that function more like having extended security team members.
"The engagement changes on a month-by-month basis based on what we need," Jim explains. This flexibility allows security teams to address emerging threats, validate fixes immediately, and provide development teams with direct access to security expertise.
This partnership model addresses several critical gaps in traditional testing:
Immediate response capability — When urgent security questions arise, established relationships eliminate procurement delays and ensure the testing team already understands your environment.
Developer collaboration — Rather than security being "done to" development teams, collaborative relationships enable direct dialogue between developers and security experts. "By working with you, we're also providing those teams with input and the opportunity to discuss what the solutions are," Jim observes.
Continuous context building — Long-term relationships mean testing teams develop a deep understanding of your infrastructure, applications, and business processes. This knowledge proves invaluable when interpreting new findings or assessing emerging threats.
Real-time validation — When development teams implement fixes, they can immediately validate solutions rather than waiting for the next annual assessment cycle.
The difference becomes clear when there is an urgent security question. Traditional procurement processes require finding available consultants, defining scope, and navigating approval workflows. Partnership arrangements eliminate these delays entirely.
For example, when security teams need immediate interpretation of vulnerability scan outputs or threat intelligence, they can access expert analysis straight away.
As Jim puts it: "Our penetration testing firm has already tested the exploitability of some of these things, and this is not an issue. They've come back and told us what to do in terms of prioritising fixes."
Making the change
Converting from compliance-driven testing to strategic security partnership requires rethinking both procurement and engagement models. Rather than annual projects with fixed scopes, consider retainer arrangements that provide flexible access to security expertise when needed.
The key is ensuring your penetration testing provider becomes "an extension of what we have," as Jim says. This means establishing regular communication channels, involving testers in architectural discussions, and creating feedback loops between development teams and security experts.
Most importantly, resist the temptation to scope tests so narrowly that they become meaningless. "If you're using it for value, then you make sure the test is broad enough," Jim advises.
Six key lessons
Annual compliance testing creates dangerous security gaps in rapidly changing environments.
Context transforms vulnerability lists into actionable risk intelligence.
Unstated scope assumptions leave critical systems untested.
Partnership models provide greater value than project-based engagements.
Business logic vulnerabilities require ongoing collaboration between developers and security experts.
Flexible retainer arrangements enable an immediate response to emerging threats.
Traditional penetration testing leaves organisations vulnerable while providing false confidence. If you're tired of checkbox exercises that fail to address real risks, it's time for a different conversation.
At Chaleit, we work as partners, not vendors. Our concierge penetration testing service provides the ongoing expertise and collaborative relationship your security program deserves. No more waiting months for results. No more confusion about findings. Just smart security testing that delivers genuine value.
Let's discuss how strategic penetration testing can strengthen your security posture.
