You're running penetration tests, but are you actually getting more secure?
Many organisations today find themselves trapped in an inefficient cycle of checkbox security. They commission annual penetration tests, receive hefty PDF reports, fix a handful of issues, and repeat the process next year, only to find many of the same vulnerabilities resurfacing. Meanwhile, the real threats to your business continue to escalate. This isn't just inefficient — it's dangerous.
The hard truth is that traditional penetration testing has failed to keep pace with security challenges.
While you're busy scheduling your annual test six months in advance, attackers are moving faster, finding new ways in, and targeting the blind spots that standardised assessments consistently miss. APIs, cloud configurations, and production environments — where real attacks happen — often fall outside the rigid scope of traditional tests.
This penetration testing guide cuts through the noise with a clear, straightforward approach that actually works. You'll discover:
What real-world penetration testing looks like beyond compliance checkboxes
Why the traditional annual testing model creates security bottlenecks
How leading organisations have cut remediation costs by 50% with a better approach
Practical strategies for aligning security testing with your business rhythm
Five principles for penetration testing that deliver genuine security improvements
Whether you're a security leader or technical practitioner, this guide provides the clarity you need to transform penetration testing from a periodic burden into a strategic advantage.
Security done right requires clarity, not complexity. Let's get started.
What penetration testing is (and isn't)
Penetration testing is the deliberate simulation of adversarial techniques to identify vulnerabilities, validate controls, and measure your real exposure. It's not vulnerability scanning or bug hunting. It's a risk-driven, methodical process that answers specific threat questions that matter to your business.
The concept dates back to the 1960s, when NSA experts first used "penetration" to describe attacks against computer systems. Today's approach is far more sophisticated and necessary.
Why penetration testing matters in 2025
The 2024 IBM global report puts the average cost of a data breach at $4.88 million, 10% higher than 2023 and the highest ever recorded. The top attack vectors: phishing, stolen credentials, and cloud misconfigurations.
Most organisations confuse scanning with testing. The difference? One finds known issues; the other reveals how those issues create exploitation paths that impact your business.
As we noted in a recent article, "Penetration testing has developed a reputation problem. What should be a valuable security exercise has too often become a mere checkbox activity — something organisations do twice a year because compliance demands it, not because it genuinely improves their security posture."
Let's define what matters:
Vulnerability assessment: Broad evaluation that identifies known weaknesses without exploitation. Useful but incomplete.
Penetration testing: Simulates real attacks to exploit weaknesses. The value comes from a defined scope and finding risks relevant to your environment.
PTaaS (Penetration Testing as a Service): Embeds testing into your development lifecycle with faster feedback loops and better visibility.
Red teaming: Goal-driven approach simulating persistent threats to test detection and response capabilities.
Purple teaming: Collaborative effort between offensive and defensive teams. Drives practical improvements across detection, prevention, and response.
Pen testing and compliance: Breaking the checkbox cycle
Many organisations approach penetration testing primarily as a security compliance exercise. It's understandable — regulations demand it, auditors check for it, and frameworks require it:
ISO/IEC 27001: Control validation and effectiveness
PCI DSS: Required testing under DSS v4.0
NIST 800-53/800-115: Technical control assessment
GDPR: Assurance for Article 32 technical controls
But this compliance-first mindset often undermines security value.
Robin Fewster, Senior Manager of Security Testing at Hargreaves Lansdown, acknowledges that while compliance is important for establishing security standards, it can sometimes stifle innovation:
"The focus on meeting specific compliance requirements can lead to a tick-box exercise approach, leaving little room for creative thinking about potential security threats."
Jacob Thampi, VP and Divisional Information Security Officer at QBE North America, emphasises that poor pen tests have real-world consequences:
“In the event of a security incident, the fact that we’ve performed pen tests will lead to questions like: Wasn’t the purpose of those tests to prevent exactly this kind of thing?”. To avoid that, “We need to deliver pen testing with context. While compliance is a major driver, simply providing the output without that context isn’t sufficient.”
The challenge is clear: How can companies meet compliance requirements while also deriving genuine security value from penetration testing? The answer begins with understanding why traditional approaches fail.
Problems with traditional penetration testing
Most organisations are stuck in a cycle of annual penetration testing that satisfies compliance but often fails to improve security.
Dan Haagman, Chaleit's CEO, puts it bluntly in an article about pen testing scopes:
"Pen testing applications alone cannot make a company secure."
He highlights several issues with traditional penetration testing:
Narrow scopes miss broader attack surfaces. Too many providers focus on limited test targets rather than the entire attack surface. Attackers won't politely respect your scope boundaries — they'll breach corporate infrastructure to access application data through the path of least resistance.
Annual cycles create bottlenecks. The Q4 rush creates a security traffic jam. Companies scramble to complete tests before year-end, creating a concertina effect where everything backs up at once, followed by a flood of remediation work hitting teams in January.
Blind spots in critical areas. Today's SaaS-driven world leaves dangerous gaps. While we test lower environments thoroughly, production environments and APIs often get overlooked entirely.
The solution isn't just more testing but smarter testing.
Organisations must mature beyond the yearly checkbox mentality and rigid scopes. Quick, targeted assessments might not be exhaustive, but often reveal critical vulnerabilities that justify more focused testing where it matters most.
Types of penetration testing: Matching methods to threats
Selecting the right test isn't about following industry trends. It's about aligning with your business objectives and addressing threats relevant to your environment.
Our goal at Chaleit: reduce risk. Everything else is secondary.
Most security teams struggle with test selection because they work from standardised templates rather than business needs. As Robin Fewster notes:
"The industry is full of extremely smart people that apply critical thinking on a daily basis. It's just that sometimes they're forced to deliver services in a certain way."
To break free from the checkbox mentality, you need a framework for structuring tests that actually match your threat landscape. Here's how to think about it:
Penetration testing by origin: Where attacks come from
The entry point of an attack determines what gets tested and how. Different origins require different testing approaches:
External: Tests from the internet, showing what outsiders can access.
Internal: Reveals what happens when perimeter defences fail.
Cloud: Targets cloud-specific assets and misconfigurations.
IoT: Examines embedded systems and their communication paths.
Mobile: Evaluates mobile apps and device-side protections.
Penetration testing by methodology: How much visibility testers have
The level of system knowledge given to testers dramatically changes what they'll find:
Black box: No prior knowledge. Shows what determined attackers see.
Grey box: Partial knowledge. Simulates insider threats.
White box: Full disclosure. Delivers thorough, cooperative assessment.
These approaches work best when co-designed with your team. Your internal knowledge makes testing more relevant and drives better outcomes.
Penetration testing by engagement model: How testers interact with defenders
The relationship between testers and security teams shapes the learning experience:
Blind testing: Testers know the target; defenders don't.
Double blind: Neither side knows. Tests real-world detection capability.
Targeted testing: Both sides collaborate. Useful for control validation.
Effective penetration testing requires deliberate choices in each of these dimensions. One-size-fits-all approaches consistently underdeliver because they're not tailored to your specific needs and threats.
Attack surface priorities: Breaking free from standard checklists
Now that you understand how to structure your testing approach, the next critical decision is where to focus your efforts. Most security programs struggle because they apply the same generic testing pattern across drastically different technologies and environments.
Modern attack surfaces span far beyond traditional web applications. Each area requires specialised knowledge, tools, and techniques to properly assess:
Web applications: SQLi, XSS, CSRF, IDOR, authentication flaws
Mobile platforms: Local data storage, API communication, reverse engineering
Code: Logic flows, security flaws in source code
APIs: REST/GraphQL endpoint security
Infrastructure: Network vulnerabilities, misconfigurations
Cloud: IAM issues, exposed services, architecture flaws
Containers: Escape paths, insecure images
Human factor: Phishing, social engineering, physical access
AI systems: Algorithmic and model-level security
Cryptography: Weaknesses in encryption implementation
The key is aligning your testing priorities with your most critical assets and likely attack vectors. A financial institution might need deeper API and transaction security testing, while a healthcare provider might prioritise data access controls and third-party integrations.
A targeted approach sets the stage for outgrowing simple vulnerability identification and forming a more comprehensive security strategy, one that integrates additional threat intelligence and contextual inputs.
Towards context-driven testing
Prioritising specific attack surfaces is just the beginning. The most effective security programs integrate additional inputs that provide real-world context to testing activities. Without this layer, even well-structured tests can miss critical threats.
Think of it this way: vulnerabilities exist within ecosystems, not in isolation. The same technical flaw might be trivial in one environment but catastrophic in another. Here's how to build that critical context:
OSINT: Open-source intelligence reveals your visible attack surface — what adversaries can see before they even launch an attack.
Threat modelling: Aligns testing with attacker TTPs and most likely breach paths specific to your industry and organisation.
Threat intelligence: Grounds scenarios in relevant real-world data, focusing on threats actually targeting your sector.
Architecture review: Identifies systemic design issues that enable exploitation across multiple systems.
Kevin O'Sullivan, experienced cyber security strategy consultant, emphasises the importance of collaboration in this approach:
"If you're acting as a white hat coming in ethically, you're teaming up together to strengthen your cyber defences against the real bad actors."
The industry needs to adopt a more collaborative mindset when approaching security testing. By integrating intelligence sources and fostering cooperation between offensive and defensive teams, penetration testing evolves from an isolated activity into a strategic capability that continually enhances the security posture.
Read more about how to move towards context-driven penetration testing, in our practical guide to modern penetration testing methodology.
Penetration testing cadence: Low and slow wins the race
Instead of the "big-bang yearly event," Dan Haagman advocates for:
"Thinking low and slow – like a good barbecue. This approach allows us to match multiple critical paths, integrate better with the business, focus on more important parts of the exploit chain, and address issues as they arise rather than batching them up."
This distributed model has several benefits:
Release alignment: Testing coincides with your development cycles.
Business integration: Security becomes part of the process, not an interruption.
Focused coverage: Attention on high-risk areas when they matter most.
Immediate remediation: Fix issues as they appear, not months later.
Clyde Netto, Director of Technology and Cyber Security, reinforces this concern about traditional approaches:
"A yearly pen test might not provide sufficient coverage, visibility, or even legal liability protection across the entire scope of a company's security needs."
The right cadence isn't just about frequency, it's about aligning security with your business rhythm and addressing real threats when they matter most.
Root causes of breaches: What vulnerability scores miss
After understanding how to structure tests and prioritise attack surfaces, we need to focus on what actually leads to security incidents.
While CVEs and CVSS scores get the headlines, they rarely tell the complete story of organisational risk.
The limitation of vulnerability metrics
Vulnerability metrics have their place, but they're starting points, not conclusions:
CVE: Identifies what the vulnerability is
CVSS: Indicates relative severity (0.0-10.0)
Both exist in isolation from your environment. They don't account for mitigating controls, architectural protections, or business context. Real-world impact requires understanding how vulnerabilities interact with your specific systems and the defensive measures you have in place.
Five root causes of breaches
The most devastating breaches typically stem from five fundamental problems that standard vulnerability metrics often overlook:
Misconfigurations: Incorrect settings that create unnecessary exposure — often the simplest issues with the largest impact.
Detection gaps: Blind spots in monitoring that allow attacks to proceed unnoticed for months.
Poor access control: Excessive privileges enabling lateral movement once initial access is gained.
Patch management gaps: Delayed updates leaving known vulnerabilities open long after fixes are available.
Response failures: Breakdowns in containment and communication during incidents, which can turn minor breaches into major ones.
These issues are typically not isolated technical flaws but symptoms of broader operational challenges within organisations. They're also the areas where targeted, intelligence-driven testing provides the most value.
Focusing on remediation efforts rather than chasing the latest headline vulnerability can help organisations achieve meaningful security improvements that address the actual pathways attackers use most frequently. This leads us to see how penetration testing can be transformed from a technical exercise into a business enabler.
Penetration testing success in practice
Understanding root causes is essential, but how do we implement these insights? Let's look at a pen testing case study and examine how the Chaleit team helped a stock exchange-listed financial services company move away from traditional penetration testing with remarkable results:
Reduced penetration testing costs by 30%
Cut remediation costs by 50%
Decreased retest time from days to minutes
Eliminated report preparation overhead
Enabled real-time issue resolution
These were significant changes that came from rejecting the standard model. Here are five things we implemented differently:
No traditional reports — Digital reporting through existing platforms
Direct communication — Real-time collaboration via Teams/Slack
Flexible engagement — No burdensome procurement processes
Integration with existing tools — Work within client systems
Variable depth testing — Right-sized for actual needs
This pen testing 2.0 model demolished the artificial barriers between security and development teams. Instead of treating penetration testing as a periodic audit, it became a streamlined, continuous process embedded in their workflow.
Perhaps most importantly, this client shifted metrics from counting vulnerabilities to measuring resolution speed.
Security isn't about finding the most flaws — it's about fixing the right ones quickly. The model we proposed created a foundation for the broader changes in how security findings are communicated and addressed.
From dusty reports to real results
Building on the real-world example, it's clear that the traditional deliverable — the PDF report — often fails to drive meaningful change. Security findings locked away in static documents rarely translate into actual improvements.
A report collecting dust helps no one. We skip the lengthy reports and create tickets directly in your system for immediate action. This approach saves time and ensures vulnerabilities get fixed rather than filed away.
When security testing becomes a true cyber security partnership, success metrics change dramatically. Instead of counting vulnerabilities found, organisations measure actual security improvements.
Kevin O'Sullivan explains:
"Through offensive security collaboration, we might identify that certain detection rules aren't properly catching indicators of compromise. We tune them and immediately gain better visibility around our detection capabilities. That's what I want to see."
This creates what we call the "penetration testing value triangle" that connects:
Internal teams — Your defenders and developers with deep system knowledge.
Security providers — Specialists bringing external perspective and expertise.
Business objectives — Actual security improvements that matter to your organisation.
This triangle focuses everyone on outcomes rather than outputs. The result: reduced penetration testing costs, faster remediation, and genuine security improvements.
Security teams require several key elements to make this integrated approach work:
Direct task integration: Findings that flow directly into your workflow tools.
Executive briefings: Risk-oriented summaries highlighting material issues.
Hands-on remediation: Expert guidance when you need it most.
Verification testing: Confirmation that fixes actually work.
"We're nerds. We're experts in security and hacking. And we want to help people," Dan Haagman highlights.
This perspective drives Chaleit's approach. We find what others miss, then fix it properly, working alongside you, not just dropping findings and leaving.
The best security partners deliver clarity through the entire penetration testing process, not just the assessment. This partnership approach naturally leads to the next important question.
What should you look for in a penetration testing provider?
Finding the right penetration testing partner shouldn't be complicated. Look for these five qualities:
Quiet competence over rock star egos.
Speed measured in days, not weeks or months.
Easy onboarding that simplifies procurement.
Dynamic reporting directly into your systems.
Genuine aftercare focused on making you more secure.
We discuss this more in our handbook on how to buy penetration testing that works. It includes 30 provider evaluation questions, budget guidance, red flags to avoid, and how to structure engagements for measurable security improvements.
Traditional penetration testing treats security like a commodity — a predictable, standardised output regardless of context. We believe security is a concierge service, tailored to your specific environment and needs.
Effective penetration testing isn't about finding everything but about finding what matters before attackers do.
Simple isn't easy. But that's exactly why we're here.
Key takeaways: Penetration testing done right
Drawing from our extensive work with clients and discussions with industry leaders, here are five principles that define effective penetration testing:
Rethink the relationship. Position offensive security as a support network for your internal teams, not as adversaries.
Reverse the testing order. Start with collaborative, announced testing to build capabilities together before moving to surprise testing.
Communicate in real time. Replace lengthy after-action reports with ongoing communication and immediate remediation of critical findings.
Co-design your programme. Work with pen testing partners to design a programme that delivers strategic value, not just technical assessments.
Focus on outcomes. Measure success by improved security capabilities, not just vulnerabilities identified.
We hope you found this guide to penetration testing useful and that it helped you understand why you should not view security testing as just a periodic checkbox exercise.
The most effective programmes view offensive security as an integral part of their overall security strategy, working alongside defensive measures to create a more resilient posture.
At Chaleit, we've helped organisations across industries move beyond compliance to build genuine security resilience. Our approach has reduced remediation costs while creating stronger, more effective security postures.
If you're ready to discuss how a different approach to penetration testing could deliver better results for your organisation, our team is here to help.
