Insufficient AI security controls demonstrated through penetration testing
Summary
Hide â˛
Show âŧ
Penetration testing by NCC Group revealed critical flaws in AI security approaches, particularly in large language models (LLMs). Organizations rely too heavily on guardrails, which are insufficient against sophisticated attacks. These vulnerabilities can lead to arbitrary code execution, password exfiltration, and database dumps. The recommendation is to shift from object-based to data-based permission models and implement proper architectural controls to mitigate risks. David Brauchler, technical director and AI/ML security practice lead at NCC Group, emphasizes the need for a fundamental shift in AI security strategies. He advises that AI systems with high-privilege access should not be exposed to untrusted data, and systems processing untrusted data should not have high-privilege functionality.
Timeline
-
21.08.2025 21:41 đ° 1 articles
AI security vulnerabilities demonstrated through penetration testing
Penetration testing by NCC Group revealed critical flaws in AI security approaches, particularly in large language models (LLMs). Organizations relying solely on guardrails are at risk of sophisticated attacks that can execute arbitrary code, exfiltrate passwords, and dump entire databases. The recommendation is to implement architectural controls that separate high-privilege access from untrusted data, and vice versa. This involves shifting from object-based to data-based permission models to better secure AI systems.
Show sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
Information Snippets
-
NCC Group demonstrated vulnerabilities in AI systems through penetration testing.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
Organizations rely too heavily on guardrails for AI security, which are insufficient against sophisticated attacks.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
Vulnerabilities in AI systems can lead to arbitrary code execution, password exfiltration, and database dumps.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
A shift from object-based to data-based permission models is recommended for improving AI security.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
AI systems with high-privilege access should not be exposed to untrusted data.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
Systems processing untrusted data should not have high-privilege functionality.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
-
Effective security strategies exist but require recognition of the unique security paradigm that AI systems present.
First reported: 21.08.2025 21:41đ° 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap â www.darkreading.com â 21.08.2025 21:41
Similar Happenings
Cursor AI editor autoruns malicious code in repositories
A flaw in the Cursor AI editor allows malicious code in repositories to autorun on developer devices. This vulnerability can lead to malware execution, environment hijacking, and credential theft. The issue arises from Cursor disabling the Workspace Trust feature from VS Code, which prevents automatic task execution without explicit user consent. The flaw affects one million users who generate over a billion lines of code daily. The Cursor team has decided not to fix the issue, citing the need to maintain AI and other features. They recommend users enable Workspace Trust manually or use basic text editors for unknown projects. The flaw is part of a broader trend of prompt injections and jailbreaks affecting AI-powered coding tools.
Critical SAP NetWeaver Command Execution Vulnerabilities Patched
SAP has patched three critical vulnerabilities in NetWeaver, its middleware for business applications. The most severe flaw, CVE-2025-42944, allows unauthenticated attackers to execute arbitrary OS commands via insecure deserialization. Two other critical issues, CVE-2025-42922 and CVE-2025-42958, enable authenticated users to upload arbitrary files and unauthorized users to access administrative functions. These vulnerabilities affect SAP's ERP, CRM, SRM, and SCM applications, widely used in large enterprise networks. The patches come amid ongoing exploitation of another critical SAP vulnerability, CVE-2025-42957, which affects S/4HANA, Business One, and NetWeaver products. SAP released 21 new and four updated security notes on September 2025 patch day, including updates for NetWeaver AS ABAP and other SAP products. SAP has also released a patch for a high-severity missing input validation bug in SAP S/4HANA (CVE-2025-42916, CVSS score: 8.1).
AI systems vulnerable to data-theft via hidden prompts in downscaled images
Researchers at Trail of Bits have demonstrated a new attack method that exploits image downscaling in AI systems to steal user data. The attack injects hidden prompts in full-resolution images that become visible when the images are resampled to lower quality. These prompts are interpreted by AI models as user instructions, potentially leading to data leakage or unauthorized actions. The vulnerability affects multiple AI systems, including Google Gemini CLI, Vertex AI Studio, Google Assistant on Android, and Genspark. The attack works by embedding instructions in images that are only revealed when the images are downscaled using specific resampling algorithms. The AI model then interprets these hidden instructions as part of the user's input, executing them without the user's knowledge. The researchers have developed an open-source tool, Anamorpher, to create images for testing this vulnerability. To mitigate the risk, Trail of Bits recommends implementing dimension restrictions on image uploads, providing users with previews of downscaled images, and requiring explicit user confirmation for sensitive tool calls.
PromptFix exploit enables AI browser deception
A new prompt injection technique, PromptFix, tricks AI-driven browsers into executing malicious actions by embedding hidden instructions in web pages. The exploit targets AI browsers like Perplexity's Comet, Microsoft Edge with Copilot, and OpenAI's upcoming 'Aura', which automate tasks such as online shopping and email management. PromptFix can deceive AI models into interacting with phishing sites or fraudulent storefronts, potentially leading to unauthorized purchases or credential theft. The technique exploits the AI's design goal to assist users quickly and without hesitation, creating a new scam landscape called Scamlexity. Researchers from Guardio Labs demonstrated the exploit by tricking Comet into adding items to a cart and auto-filling payment details on fake shopping sites. Similar attacks can manipulate AI browsers into parsing spam emails and entering credentials on phishing pages. PromptFix can also bypass CAPTCHA checks to download malicious payloads without user involvement. The exploit highlights the need for robust defenses in AI systems to anticipate and neutralize such attacks, including phishing detection, URL reputation checks, and domain spoofing protections. Until security matures, users should avoid assigning sensitive tasks to AI browsers and manually input sensitive data when needed. AI browser agents from major AI firms failed to reliably detect the signs of a phishing site. AI agents are gullible and servile, making them vulnerable to attacks in an adversarial setting. Companies should move from "trust, but verify" to "doubt, and double verify" until an AI agent has shown it can always complete a workflow properly. AI companies are not expected to pause developing more functionality to improve security. Companies should hold off on putting AI agents into any business process that requires reliability until AI-agent makers offer better visibility, control, and security. Securing AI requires gaining visibility into all AI use by company workers and creating an AI usage policy and a list of approved tools.