Open Source Benchmark Framework b3 for LLM Security in AI Agents
Summary
Hide ▲
Show ▼
The UK AI Security Institute (AISI) has launched an open-source benchmark framework, b3, in collaboration with Check Point and Lakera. This tool aims to enhance the security of large language models (LLMs) that power AI agents by identifying vulnerabilities in individual LLM calls. The b3 framework uses 'threat snapshots' to test for various attack vectors, including system prompt exfiltration, phishing link insertion, and malicious code injection. The framework is designed to make LLM security measurable and comparable across different models and applications. The b3 benchmark includes 10 representative agent threat snapshots and a dataset of 19,433 adversarial attacks from Lakera’s Gandalf initiative. It provides developers with a way to assess and improve the security posture of their models. The framework reveals that models with step-by-step reasoning tend to be more secure, and open-weight models are rapidly closing the security gap with closed systems.
Timeline
-
29.10.2025 12:45 1 articles · 12d ago
Open Source b3 Benchmark Framework Launched for LLM Security
The UK AI Security Institute (AISI) has released the b3 benchmark framework in collaboration with Check Point and Lakera. This open-source tool is designed to improve the security of large language models (LLMs) that power AI agents. The framework uses 'threat snapshots' to identify vulnerabilities in individual LLM calls and includes a dataset of 19,433 adversarial attacks. It aims to make LLM security measurable and comparable across different models and applications.
Show sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
Information Snippets
-
The b3 benchmark framework is an open-source tool developed by the UK AI Security Institute (AISI), Check Point, and Lakera.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
The framework focuses on the security of large language models (LLMs) that power AI agents.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
b3 uses 'threat snapshots' to identify vulnerabilities in individual LLM calls within AI agent workflows.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
The framework includes 10 representative agent threat snapshots and a dataset of 19,433 adversarial attacks.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
The b3 benchmark tests for various attack vectors, including system prompt exfiltration, phishing link insertion, malicious code injection, denial-of-service, and unauthorized tool calls.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
The framework aims to make LLM security measurable, reproducible, and comparable across different models and applications.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
The b3 benchmark reveals that models with step-by-step reasoning tend to be more secure.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
-
Open-weight models are closing the security gap with closed systems faster than expected.
First reported: 29.10.2025 12:451 source, 1 articleShow sources
- Open Source “b3” Benchmark to Boost LLM Security for Agents — www.infosecurity-magazine.com — 29.10.2025 12:45
Similar Happenings
AI-Powered Malware Families Deployed in the Wild
Google's Threat Intelligence Group (GTIG) has identified new malware families that leverage artificial intelligence (AI) and large language models (LLMs) for dynamic self-modification during execution. These malware families, including PromptFlux, PromptSteal, FruitShell, QuietVault, and PromptLock, demonstrate advanced capabilities for evading detection and maintaining persistence. PromptFlux, an experimental VBScript dropper, uses Google's LLM Gemini to generate obfuscated VBScript variants and evade antivirus software. It attempts persistence via Startup folder entries and spreads laterally on removable drives and mapped network shares. The malware is under development or testing phase and is assessed to be financially motivated. PromptSteal is a data miner written in Python that queries the LLM Qwen2.5-Coder-32B-Instruct to generate one-line Windows commands to collect information and documents in specific folders and send the data to a command-and-control (C2) server. It is used by the Russian state-sponsored actor APT28 in attacks targeting Ukraine. The use of AI in malware enables adversaries to create more versatile and adaptive threats, posing significant challenges for cybersecurity defenses. Various threat actors, including those from China, Iran, and North Korea, have been observed abusing AI models like Gemini across different stages of the attack lifecycle. The underground market for AI-powered cybercrime tools is also growing, with offerings ranging from deepfake generation to malware development and vulnerability exploitation.