Inadequate AI Security Controls Expose Systems to Arbitrary Code Execution, Data Exfiltration
Summary
Hide ▲
Show ▼
Organizations are overly reliant on guardrails as the primary security control for large language models (LLMs), which is insufficient against sophisticated attacks. Penetration testing has demonstrated that AI systems with inadequate security boundaries can be manipulated to execute arbitrary code, exfiltrate passwords, and dump entire databases. Experts recommend establishing proper architectural controls to mitigate these risks. The primary security concern is that LLMs with high-privilege access should never be exposed to untrusted data, and systems processing untrusted data should never have high-privilege functionality. This approach aims to reduce the attack surface and prevent potential data breaches. The discussion highlights the need for a fundamental shift from object-based permission models to data-based permissions when implementing AI systems.
Timeline
-
21.08.2025 21:41 📰 1 articles · ⏱ 26d ago
AI Security Vulnerabilities Highlighted at Black Hat USA 2025
At Black Hat USA 2025, David Brauchler, technical director and AI/ML security practice lead at NCC Group, discussed critical flaws in current AI security approaches. He emphasized that organizations are overly reliant on guardrails as the primary security control for LLMs, which is insufficient against sophisticated attacks. Penetration testing demonstrated that AI systems with inadequate security boundaries can be manipulated to execute arbitrary code, exfiltrate passwords, and dump entire databases. Brauchler recommended establishing proper architectural controls, including ensuring that high-privilege AI systems are not exposed to untrusted data and that systems processing untrusted data do not have high-privilege functionality. He also advocated for a shift from object-based permission models to data-based permissions.
Show sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
Information Snippets
-
Guardrails are insufficient as the primary security control for LLMs.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
-
AI systems with inadequate security boundaries can be manipulated to execute arbitrary code, exfiltrate passwords, and dump entire databases.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
-
Proper architectural controls are recommended to mitigate risks associated with AI systems.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
-
High-privilege AI systems should not be exposed to untrusted data.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
-
Systems processing untrusted data should not have high-privilege functionality.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
-
A shift from object-based permission models to data-based permissions is necessary for AI system security.
First reported: 21.08.2025 21:41📰 1 source, 1 articleShow sources
- How Architectural Controls Help Can Fill the AI Security Gap — www.darkreading.com — 21.08.2025 21:41
Similar Happenings
Critical SAP S/4HANA Command Injection Vulnerability Exploited
A critical command injection vulnerability in SAP S/4HANA (CVE-2025-42957) is being actively exploited in the wild. The flaw, with a CVSS score of 9.9, allows attackers with low-privileged user access to execute arbitrary ABAP code, bypass authorization checks, and fully compromise the SAP environment. This can lead to data theft, fraud, or ransomware installation. The vulnerability affects both on-premise and Private Cloud editions of SAP S/4HANA, as well as several other SAP products and versions. SecurityBridge Threat Research Labs discovered the vulnerability and reported it to SAP on June 27, 2025. The vendor fixed the vulnerability on August 11, 2025, but several systems have not applied the available security updates and are now being targeted by hackers. Exploitation activity surged dramatically after the patch was released. Organizations are advised to apply patches immediately, monitor logs for suspicious activity, and implement additional security measures.
Active Exploitation of FreePBX Zero-Day Vulnerability CVE-2025-57819
A zero-day vulnerability in FreePBX, identified as CVE-2025-57819, is being actively exploited. The flaw allows unauthenticated access to the FreePBX Administrator, leading to arbitrary database manipulation and remote code execution. The vulnerability affects specific versions of FreePBX, and exploitation began on or before August 21, 2025. Sangoma has released emergency patches for the vulnerability. Users are advised to update to the latest versions, restrict public access to the administrator control panel, and follow additional security recommendations. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has added CVE-2025-57819 to its Known Exploited Vulnerabilities (KEV) catalog, mandating federal agencies to apply patches by September 19, 2025.
Citrix NetScaler ADC and Gateway vulnerabilities patched and actively exploited in the wild
Citrix has released patches for three vulnerabilities in NetScaler ADC and NetScaler Gateway. One of these vulnerabilities, CVE-2025-7775, is actively exploited in the wild. The flaws include memory overflow vulnerabilities and improper access control issues. The vulnerabilities affect specific configurations of NetScaler ADC and NetScaler Gateway, including unsupported, end-of-life versions. Citrix has confirmed active exploitation of CVE-2025-7775, which can lead to remote code execution or denial-of-service. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has added CVE-2025-7775 to its Known Exploited Vulnerabilities (KEV) catalog, requiring federal agencies to remediate within 48 hours. Nearly 20% of NetScaler assets identified are on unsupported, end-of-life versions, with a significant concentration in North America and the APAC region. CISA lists 10 NetScaler flaws in its KEV catalog, with six discovered in the last two years. Threat actors are using HexStrike AI, an AI-driven security platform, to exploit the Citrix vulnerabilities, significantly reducing the time between disclosure and mass exploitation. HexStrike-AI was created by cybersecurity researcher Muhammad Osama and has been open-source and available on GitHub for the last month, where it has already garnered 1,800 stars and over 400 forks.
AI systems vulnerable to data-theft prompts in downscaled images
Researchers have demonstrated a new attack method that steals user data by embedding malicious prompts in images. These prompts are invisible in full-resolution images but become visible when the images are downscaled by AI systems. The attack exploits aliasing artifacts introduced by resampling algorithms, allowing hidden text to emerge and be interpreted as user instructions by the AI model. This can lead to data leakage or unauthorized actions. The method has been successfully tested against several AI systems, including Google Gemini CLI, Vertex AI Studio, Gemini's web interface, Gemini's API, Google Assistant on Android, and Genspark. The attack was developed by Kikimora Morozova and Suha Sabi Hussain from Trail of Bits, building on a 2020 theory presented in a USENIX paper. The researchers have also released an open-source tool, Anamorpher, to create images for testing the attack. They recommend implementing dimension restrictions and user confirmation for sensitive tool calls as mitigation strategies.
PromptFix Exploit Targets AI Browsers for Malicious Prompts
Researchers from Guardio Labs have demonstrated a new prompt injection technique called PromptFix. This exploit tricks generative AI (GenAI) models into executing malicious instructions embedded within fake CAPTCHA checks on web pages. The attack targets AI-driven browsers like Perplexity's Comet, which automate tasks such as shopping and email management. The exploit misleads AI models into interacting with phishing pages or fraudulent sites without user intervention, leading to potential data breaches and financial losses. The technique, dubbed Scamlexity, represents a new era of scams where AI convenience collides with invisible scam surfaces, making humans collateral damage. The exploit can trick AI models into purchasing items on fake websites, entering credentials on phishing pages, or downloading malicious payloads. The findings underscore the need for robust defenses in AI systems to anticipate, detect, and neutralize such attacks. Microsoft Edge is embedding agentic browsing features through a Copilot integration, and OpenAI is developing an agentic AI browser platform codenamed 'Aura'. Comet is quickly penetrating the mainstream consumer market. Agentic AI browsers were released with inadequate security safeguards against known and novel attacks. Guardio advises against assigning sensitive tasks to agentic AI browsers until their security matures. AI browser agents from major AI firms failed to reliably detect the signs of a phishing site. Comet often added items to a shopping cart, filled out credit-card details, and clicked the buy button on a fake Walmart site. AI browsers with access to email will read and act on prompts embedded in the messages. AI companies need stronger sanitation and guardrails against these attacks. Nearly all companies (96%) claim to want to expand their use of AI agents in the next year, but most are not prepared for the new risks posed by AI agents in a business environment. A fundamental issue is how to discern actions taken through a browser by a user versus those taken by an agent. AI agents need to be experts at not just getting things done, but at sussing out and blocking potential security threats to workers and company data. Companies should move from "trust, but verify" to "doubt, and double verify"—essentially hobbling automation until an AI agent has shown it can always complete a workflow properly. Defective AI operations continue to be a major problem, and security represents another layer on top of those issues. Companies should hold off on putting AI agents into any business process that requires reliability until AI-agent makers offer better visibility, control, and security. Companies that intend to push their use of AI into agent-based workflows should focus on a comprehensive strategy, including inventorying all AI services used by employees and creating an AI usage policy. Employees need to understand the basics of AI safety and what it means to give these bots information or privileges to do things on their behalf.