What's the Point of Pen Testing if Nothing Gets Fixed?

Everyone in security knows the drill: commission a penetration test, wait weeks for results, receive a hefty PDF report, then scramble to address dozens of findings before the next audit cycle.

Rinse and repeat.

Yet organisations continue to get breached, often through vulnerabilities that multiple penetration tests failed to properly contextualise or prioritise.

We've said it before and we'll say it again: traditional penetration testing isn't just ineffective, it's actively creating security theatre that diverts resources from real protection.

We sat down with Aaron Katz, a seasoned CISO who's worked across multiple industries, to understand why the problem persists and how to make things better.

Broken security

A wake-up call from Aaron:

"By the time people start actually fixing the issues, it's time for the next pen test. Nothing actually gets fixed."

Here's how things are currently unfolding. Cyber crime has industrialised into specialised services. One group obtains initial access, sells it to brokers, who then sell to malware deployers, who might sell portions to ransomware operators. The barrier to entry for attacks has plummeted while defensive testing remains expensive and fragmented.

Meanwhile, large organisations often weather breaches through cash reserves and insurance.

"Look at every large company that's been breached. What happens? They shuffle their C-suite a little bit, and their stock does better," Aaron observes.

This results in misaligned incentives where security testing becomes mere compliance rather than genuine risk reduction.

CVSS trap

The dysfunction becomes clear in vulnerability scoring. The industry's reliance on CVSS scores without context creates a peculiar form of security paralysis.

Aaron recalls a telling example:

"I've seen tests where a Log4j vulnerability was identified, which sounds bad, but the function being used wasn't actually enabled, meaning there was nothing that could be exploited. Instead of assessing the actual risk, they just focused on the CVSS score, leading them to waste time and resources."

This scenario plays out repeatedly across organisations. Insurance companies demand remediation based on scores, while security teams comply without understanding the business context, and engineering resources are diverted to fixing non-issues, leaving real vulnerabilities unaddressed.

The solution isn't abandoning CVSS entirely, but rather implementing temporal and environmental scoring that considers production status, exploitability, and business impact. A "critical" vulnerability in a sandboxed development system shouldn't receive the same priority as a medium-severity issue in customer-facing production infrastructure.

Read more about the scoring problem and what to do instead in our guide to modern penetration testing methodology.

Missing link

How should findings be handled? Not every vulnerability requires immediate technical remediation — some should trigger broader business initiatives.

Consider Active Directory security. A typical pen test might recommend: "Review your Active Directory environment for misconfigurations." But Aaron notes the absurdity: "That's at least a year-long effort. Is that so I can close that?"

Smart security teams handle this differently. They address specific tactical findings while converting strategic recommendations into proper business cases. "I'm going to close out this finding about looking into Active Directory, because that now needs to be an initiative for the company," Aaron explains. This might require funding, multiple teams, and formal project management processes.

Crucially, this approach includes a resolution status rarely seen in security: "won't do." "I'm saying I'm not doing this pen test item. I have acknowledged it. I know it's an issue," but the organisation has made a risk-based decision not to address it immediately.

This is particularly relevant when broader changes make current fixes obsolete. Why invest heavily in on-premise Active Directory hardening when the organisation is migrating to cloud-based identity management?

Adversarial model

The most damaging aspect of current pen testing might be how it reinforces adversarial relationships between security and development teams. Security throws findings "over the fence" without understanding implementation constraints or business priorities.

"I've seen times where a security team would say, You have these vulnerabilities; go fix them. And then someone had to explain how Kubernetes worked, or how technology worked to show it's not an issue," Aaron recounts. "And then their impression is the security team just regurgitates tools and doesn't really provide value."

Effective security requires a partnership, not policing.

This means security professionals need sufficient technical depth to engage meaningfully with development and infrastructure teams. It means sharing tools and knowledge rather than hoarding them. Most importantly, it means understanding that "the majority of people who take pride in their work want to do things the right way."

Read our guide on how to buy penetration testing that actually works to learn how to identify providers who help you get more secure, not just tick compliance boxes.

Communication and integration

How many times have security leaders experienced this scenario: testing begins Monday, Thursday arrives with no communication, and two weeks later, they're told the report is "in QA" for another week? Meanwhile, critical vulnerabilities remain unaddressed simply because nobody knew they existed.

This communication gap isn't just frustrating, it's dangerous. A critical vulnerability with genuine business impact warrants immediate escalation, not being buried in a delayed report.

Modern pen testing should include real-time collaboration, immediate escalation of critical findings, and continuous dialogue throughout the engagement.

Security testing works best when integrated into broader development and risk management processes. This means security teams must understand how their organisations generate revenue and what disruptions would genuinely impact operations.

"If you don't understand how your company generates revenue," Aaron emphasises, "a suddenly reflected cross-site scripting attack might not seem as important as ensuring continuous availability." He argues that this foundational business context is essential for properly prioritising risks and allocating resources effectively.

Integration also means adapting security practices to organisational culture and workflows. The same penetration testing firm may operate differently across different clients, depending on their internal processes, risk tolerance, and technical constraints.

So what's the point, really?

The point of pen testing isn't the test itself, it's what happens next. When done properly, security testing becomes a catalyst for meaningful change: better development practices, improved risk understanding, and stronger partnerships between security and engineering teams.

The pen tests that matter are the ones that help organisations ask better questions:

What are we actually trying to protect?
What would genuinely disrupt our business?
How do we build security into our DNA rather than bolting it on afterwards?

At Chaleit, we've reimagined how security testing should work: real-time collaboration, business-focused prioritisation, and seamless integration with your existing workflows.

No more waiting weeks for reports that gather dust. No more wondering what happens after the test. Just security testing that delivers genuine value and fits your organisation's rhythm.

Let's discuss your security challenges and explore how we can effectively address them.

Key takeaways

For security leaders:

Implement temporal and environmental CVSS scoring to provide meaningful context.
Develop project management processes that convert strategic findings into business initiatives.
Build cyber security partnerships with development and infrastructure teams rather than adversarial relationships.
Ensure your team understands the business model and revenue generation.

For organisations:

Demand real-time communication during security testing.
Require findings that integrate with existing project management tools and workflows.
Evaluate pen testing providers on their ability to provide business context, not just technical findings.
Stay away from one-size-fits-all approaches and look for providers who can understand your unique security challenges.