CyberHappenings logo
☰
🏠 Home 📂 Browse ℹī¸ About

General-purpose LLMs show limited effectiveness in offensive security tasks

First reported
Last updated
📰 1 unique sources, 1 articles

Summary

Hide ▲

General-purpose large language models (LLMs) demonstrate limited effectiveness in offensive security tasks, particularly for complex vulnerabilities and exploits. While some LLMs can identify simple vulnerabilities and create basic exploits, they often fail at more intricate tasks. Tailor-made AI systems remain more effective for penetration testers and offensive researchers. The research involved testing 50 different LLMs, including commercial and experimental models, on vulnerability research and exploit development tasks. The findings highlight the current limitations of general-purpose LLMs in offensive security, despite their potential for initial triaging and development support. The investigation underscores the need for human oversight in AI-driven security efforts to mitigate errors and ensure accurate results.

Timeline

  1. 13.08.2025 22:08 📰 1 articles

    Research reveals limitations of general-purpose LLMs in offensive security

    An investigation into the offensive-security capabilities of 50 different large language models (LLMs) found that while some models can handle simple vulnerabilities and exploits, they struggle with more complex tasks. Commercial LLMs showed better performance in vulnerability research tasks, but many experimental models were non-functional or behind paywalls. The research highlights the need for human oversight in AI-driven security efforts to ensure accuracy and effectiveness. The findings also underscore the potential of AI in penetration testing and application security, with expectations that AI systems will significantly impact these fields in the near future.

    Show sources
    Open in new tab

Information Snippets

Similar Happenings

AI systems vulnerable to data-theft via hidden prompts in downscaled images

Researchers at Trail of Bits have demonstrated a new attack method that exploits image downscaling in AI systems to steal user data. The attack injects hidden prompts in full-resolution images that become visible when the images are resampled to lower quality. These prompts are interpreted by AI models as user instructions, potentially leading to data leakage or unauthorized actions. The vulnerability affects multiple AI systems, including Google Gemini CLI, Vertex AI Studio, Google Assistant on Android, and Genspark. The attack works by embedding instructions in images that are only revealed when the images are downscaled using specific resampling algorithms. The AI model then interprets these hidden instructions as part of the user's input, executing them without the user's knowledge. The researchers have developed an open-source tool, Anamorpher, to create images for testing this vulnerability. To mitigate the risk, Trail of Bits recommends implementing dimension restrictions on image uploads, providing users with previews of downscaled images, and requiring explicit user confirmation for sensitive tool calls.

Open in new tab

Black Hat NOC Enhances AI in Security Operations

The Black Hat USA 2025 Network Operations Center (NOC) team expanded its use of AI and machine learning (ML) to manage and secure the conference network. The team monitored and mitigated malicious activities, distinguishing between legitimate and illegal activities, and identified vulnerabilities in applications and misconfigurations in security tools. The NOC team also observed trends such as increased self-hosting and insecure data transmission, which pose risks to organizations. The NOC team leveraged AI for risk scoring and categorization, ensuring that legitimate training activities were not flagged as malicious. They also identified and alerted attendees to security issues, such as misconfigured security tools and vulnerable applications, which could expose sensitive data. Additionally, high school students Sasha Zyuzin and Ruikai Peng presented a new vulnerability discovery framework at Black Hat USA 2025, combining static analysis with AI. Their framework, "Tree of AST," uses Google DeepMind's Tree of Thoughts methodology to automate vulnerability hunting while maintaining human oversight. The presenters discussed the double-edged nature of AI in security, noting that while LLMs can improve code quality, they can also introduce security risks.

Open in new tab