Artificial intelligence is both a powerful tool and a potential vulnerability across disciplines.
As organisations rush to integrate AI capabilities into their applications, many are only beginning to understand the unique security challenges this technology presents.
At Chaleit, we believe that genuine security expertise — not an overreliance on tools, automation, or AI itself — is the foundation of effective cyber security.
Our approach prioritises human intelligence, critical thinking, and deep technical knowledge, complemented by technology rather than replaced by it. While many security providers increasingly depend on automated scanning and AI-driven solutions, our security consultants apply their expert judgment to provide bespoke solutions tailored to each client's unique needs.
Nevertheless, we remain at the cutting edge of technological developments, including AI-related penetration testing, ensuring our clients benefit from both human expertise and forward-thinking methodologies for optimal outcomes.
This article is part of our Behind the Scenes series, where we share insights into how our team approaches complex security challenges in the real world.
Expanded security scopes
AI-related penetration testing brings a fundamental shift in application security.
When an organisation integrates an AI model into its application, the security scope expands dramatically. No longer is the assessment limited to traditional components like authentication, authorisation, and data handling — now it must also evaluate how the AI model itself can be manipulated, what information it might inadvertently expose, and whether it operates within its intended parameters.
Avinash Thapa, VP of Technical Services, explains, "Essentially, the assessment is circular: we're evaluating a web application, but since it contains an AI component, we're also assessing that AI, and they're inherently linked."
Consider a familiar example: ChatGPT. On its surface, it's a website with standard components: authentication, authorisation, and settings. However, behind these familiar elements lies the AI model.
"The chatbot is based on an AI model developed by OpenAI. Our testing aims to verify that the model performs according to the developers' specifications and doesn't produce unexpected or anomalous results," Avinash says.
This expanded attack surface creates significant challenges. Traditional applications operate within defined parameters — inputs are processed according to fixed rules, producing predictable outputs.
AI models, however, are designed to be flexible, interpret natural language, and generate novel responses. The flexibility creates opportunities for what security professionals call "instructional manipulation."
AI manipulation
"The goal is to understand how to manipulate the AI model, effectively changing its behaviour from its intended function, much like influencing someone's actions in a conversation," Avinash explains.
This manipulation can take various forms, from "jailbreaking" techniques that bypass AI guardrails to carefully crafted prompts that trick models into performing unintended actions.
"For example, I could try to manipulate the AI model to create a virtual machine within the chatbot and download all its resources. If successful, this would be a significant security breach. And it does happen," he illustrates.
Consider the implications. What was designed as a helpful language model or data processing tool can potentially be transformed into an attack vector, revealing sensitive information or executing malicious commands.
New attack vectors in AI applications
The security challenges don't end with the direct manipulation of AI models. Our team has identified several additional vulnerability patterns:
Cross-site scripting via AI. AI models can reflect user-provided data, including malicious scripts. If the application's user interface lacks proper security, these scripts can be executed, leading to cross-site scripting vulnerabilities.
System information disclosure. AI models may reveal sensitive system information, such as operating system details, user accounts, or file contents if not properly secured. For example, they could disclose the server's operating system and user details or potentially even expose sensitive files if prompted.
Authorisation control bypass. AI models may process data even when users have restricted access. This can result in the AI retrieving and processing information that should be blocked by authorisation controls, violating intended security policies.
These vulnerabilities are just the beginning of what security professionals must consider when evaluating AI-enhanced applications.
AI-on-AI: The next evolution in security testing
Perhaps most intriguing is the emerging approach of using AI to test AI — essentially, using one model to help circumvent the security measures of another.
"We can use external AI to attack an AI system within a client's environment. Essentially, we can leverage a model like ChatGPT to generate attack prompts, which are then sent to the target AI to exploit vulnerabilities," Avinash explains.
It's both a challenge and an opportunity. While attackers can use publicly available AI models to help develop sophisticated attacks, security professionals can leverage these same tools to enhance their testing methodologies.
Practical lessons and recommendations
Based on Chaleit's experience in AI security testing, organisations should consider the following best practices to ensure a smooth and safe integration of these new technologies:
Understand your AI model's intended scope. Clearly define what your AI component should and should not do, creating a baseline for security testing.
Implement robust input sanitisation. AI models process user inputs differently than traditional applications, requiring specialised sanitation approaches.
Test for instructional manipulation. Include scenarios where testers attempt to "jailbreak" or manipulate the AI model through carefully crafted prompts.
Apply the principle of least privilege. Ensure your AI component has access only to the systems and data it absolutely requires to function.
Implement content filtering. Both inputs to and outputs from AI models should be filtered to prevent malicious content from being processed or generated.
Consider AI's impact on traditional vulnerabilities. Evaluate how the AI component might exacerbate existing application security issues such as XSS, CSRF, or authorisation bypasses.
These recommendations reflect our approach to AI security: combining established security principles with innovative testing methodologies specifically tailored to the unique challenges of our clients.
Your AI security journey
Organisations are increasingly adopting AI capabilities, while the security implications become more complex. The potential vulnerabilities discussed here represent the beginning of what security professionals must consider.
Is your organisation developing or deploying AI-enhanced applications? Are you confident in your security posture against these new threat vectors?
We believe each client deserves a tailored approach rather than a one-size-fits-all solution. Our experienced penetration testing team combines deep human expertise with cutting-edge methodologies to help you get proactive protection.
Start with a security health check. This initial no-frills assessment provides a clear, realistic view of your current security posture against emerging AI threats. Know exactly where you stand and what specific risks your AI applications face, so you can make informed decisions about protecting your most innovative assets.
For further insights into this topic, explore the discussion between Josh Fulford (Chaleit) and Bilge Kayali (Q Investment Bank) on AI's role in cyber security and the need to adapt to thrive.
