CyberHappenings logo
☰

Insufficient AI security controls demonstrated through penetration testing

First reported
Last updated
📰 1 unique sources, 1 articles

Summary

Hide ▲

Penetration testing by NCC Group revealed critical flaws in AI security approaches, particularly in large language models (LLMs). Organizations rely too heavily on guardrails, which are insufficient against sophisticated attacks. These vulnerabilities can lead to arbitrary code execution, password exfiltration, and database dumps. The recommendation is to shift from object-based to data-based permission models and implement proper architectural controls to mitigate risks. David Brauchler, technical director and AI/ML security practice lead at NCC Group, emphasizes the need for a fundamental shift in AI security strategies. He advises that AI systems with high-privilege access should not be exposed to untrusted data, and systems processing untrusted data should not have high-privilege functionality.

Timeline

  1. 21.08.2025 21:41 📰 1 articles

    AI security vulnerabilities demonstrated through penetration testing

    Penetration testing by NCC Group revealed critical flaws in AI security approaches, particularly in large language models (LLMs). Organizations relying solely on guardrails are at risk of sophisticated attacks that can execute arbitrary code, exfiltrate passwords, and dump entire databases. The recommendation is to implement architectural controls that separate high-privilege access from untrusted data, and vice versa. This involves shifting from object-based to data-based permission models to better secure AI systems.

    Show sources

Information Snippets

Similar Happenings

Cursor AI editor autoruns malicious code in repositories

A flaw in the Cursor AI editor allows malicious code in repositories to autorun on developer devices. This vulnerability can lead to malware execution, environment hijacking, and credential theft. The issue arises from Cursor disabling the Workspace Trust feature from VS Code, which prevents automatic task execution without explicit user consent. The flaw affects one million users who generate over a billion lines of code daily. The Cursor team has decided not to fix the issue, citing the need to maintain AI and other features. They recommend users enable Workspace Trust manually or use basic text editors for unknown projects. The flaw is part of a broader trend of prompt injections and jailbreaks affecting AI-powered coding tools.

Critical SAP NetWeaver Command Execution Vulnerabilities Patched

SAP has patched three critical vulnerabilities in NetWeaver, its middleware for business applications. The most severe flaw, CVE-2025-42944, allows unauthenticated attackers to execute arbitrary OS commands via insecure deserialization. Two other critical issues, CVE-2025-42922 and CVE-2025-42958, enable authenticated users to upload arbitrary files and unauthorized users to access administrative functions. These vulnerabilities affect SAP's ERP, CRM, SRM, and SCM applications, widely used in large enterprise networks. The patches come amid ongoing exploitation of another critical SAP vulnerability, CVE-2025-42957, which affects S/4HANA, Business One, and NetWeaver products. SAP released 21 new and four updated security notes on September 2025 patch day, including updates for NetWeaver AS ABAP and other SAP products. SAP has also released a patch for a high-severity missing input validation bug in SAP S/4HANA (CVE-2025-42916, CVSS score: 8.1).

AI systems vulnerable to data-theft via hidden prompts in downscaled images

Researchers at Trail of Bits have demonstrated a new attack method that exploits image downscaling in AI systems to steal user data. The attack injects hidden prompts in full-resolution images that become visible when the images are resampled to lower quality. These prompts are interpreted by AI models as user instructions, potentially leading to data leakage or unauthorized actions. The vulnerability affects multiple AI systems, including Google Gemini CLI, Vertex AI Studio, Google Assistant on Android, and Genspark. The attack works by embedding instructions in images that are only revealed when the images are downscaled using specific resampling algorithms. The AI model then interprets these hidden instructions as part of the user's input, executing them without the user's knowledge. The researchers have developed an open-source tool, Anamorpher, to create images for testing this vulnerability. To mitigate the risk, Trail of Bits recommends implementing dimension restrictions on image uploads, providing users with previews of downscaled images, and requiring explicit user confirmation for sensitive tool calls.

PromptFix exploit enables AI browser deception

A new prompt injection technique, PromptFix, tricks AI-driven browsers into executing malicious actions by embedding hidden instructions in web pages. The exploit targets AI browsers like Perplexity's Comet, Microsoft Edge with Copilot, and OpenAI's upcoming 'Aura', which automate tasks such as online shopping and email management. PromptFix can deceive AI models into interacting with phishing sites or fraudulent storefronts, potentially leading to unauthorized purchases or credential theft. The technique exploits the AI's design goal to assist users quickly and without hesitation, creating a new scam landscape called Scamlexity. Researchers from Guardio Labs demonstrated the exploit by tricking Comet into adding items to a cart and auto-filling payment details on fake shopping sites. Similar attacks can manipulate AI browsers into parsing spam emails and entering credentials on phishing pages. PromptFix can also bypass CAPTCHA checks to download malicious payloads without user involvement. The exploit highlights the need for robust defenses in AI systems to anticipate and neutralize such attacks, including phishing detection, URL reputation checks, and domain spoofing protections. Until security matures, users should avoid assigning sensitive tasks to AI browsers and manually input sensitive data when needed. AI browser agents from major AI firms failed to reliably detect the signs of a phishing site. AI agents are gullible and servile, making them vulnerable to attacks in an adversarial setting. Companies should move from "trust, but verify" to "doubt, and double verify" until an AI agent has shown it can always complete a workflow properly. AI companies are not expected to pause developing more functionality to improve security. Companies should hold off on putting AI agents into any business process that requires reliability until AI-agent makers offer better visibility, control, and security. Securing AI requires gaining visibility into all AI use by company workers and creating an AI usage policy and a list of approved tools.