The SaferAI Roundup #6: The Implications of AI for Cybersecurity

Teams of LLM Agents can Exploit Zero-Day Vulnerabilities & CYBER SECEVAL3

Welcome to "The SaferAI Roundup". Each fortnight, we will publish LLM-generated summaries of 2-3 papers that we consider consequential in the fields of AI governance, safety, and risk management. These summaries are curated and lightly edited by the SaferAI team, an AI risk management organization. Our goal is to enable you to stay up-to-date with this fast-evolving literature by delivering concise, accessible summaries in your inbox, helping you stay informed about critical advancements in AI safety and governance without having to go through numerous academic papers.

---

In this sixth edition, we explore two papers focused on one of the most important near-term risks posed by AI models: Cybersecurity.

• The first paper, "Teams of LLM Agents can Exploit Zero-Day Vulnerabilities," reveals how AI-powered systems can be used to identify and exploit previously unknown software vulnerabilities. This research highlights the double-edged nature of AI in cybersecurity, showcasing its potential for both defensive and offensive applications.

• The second paper, "CYBER SECEVAL3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models," introduces a framework and benchmark for assessing the cybersecurity implications of LLMs.

Teams of LLM Agents can Exploit Zero-Day Vulnerabilities

Richard Fang et al. (2024)

TLDR: Teams of LLM agents can successfully exploit real-world zero-day vulnerabilities, outperforming single agents and traditional vulnerability scanners.

• Novel multi-agent architecture: The paper introduces HPTSA (Hierarchical Planning and Task-Specific Agents), a system comprising a planning agent that explores the target system and dispatches specialized subagents for specific vulnerability types. This approach resolves long-term planning issues and allows for more comprehensive vulnerability exploration.

• Benchmark of real-world vulnerabilities: The authors create a benchmark of 15 recent real-world vulnerabilities, all occurring after the knowledge cutoff date of the LLM used (GPT-4). This ensures a true zero-day setting and includes various vulnerability types like XSS, CSRF, and SQLi, with severities ranging from medium to critical.

• Significant performance improvements: HPTSA achieves a 53% success rate on the benchmark, outperforming a single GPT-4 agent without vulnerability description by 2.7x. It also comes within 1.4x of an agent with full vulnerability information, while traditional scanners like ZAP and MetaSploit fail completely (0% success rate).

• Cost analysis and implications: The average cost per successful exploit is estimated at $24.39, which is higher than human experts but expected to decrease rapidly. The authors suggest this could lead to more frequent and cost-effective penetration testing, but also warns of potential misuse by malicious actors, highlighting the need for careful consideration in AI deployment.

Link to the paper: https://arxiv.org/pdf/2406.01637v1

CYBER SECEVAL3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

Shengye Wan et al. (2024)

TLDR: Meta releases CYBER SECEVAL3, a new suite of security benchmarks for assessing cybersecurity risks and capabilities in large language models (LLMs). The paper evaluates Llama 3 models against these benchmarks and compares them to other state-of-the-art LLMs.

• Risk assessment framework: CYBER SECEVAL3 evaluates 8 different risks across two broad categories: risks to third parties (e.g., automated social engineering, scaling manual offensive cyber operations) and risks to application developers/end users (e.g., prompt injection, insecure code generation). This framework provides a structured approach to empirically measure LLM cybersecurity risks and capabilities.

• Llama 3 performance and comparisons: The study found that Llama 3 models exhibit capabilities that could potentially be employed in cyber-attacks, but the associated risks are comparable to other state-of-the-art open and closed source models. For example, Llama 3 405B demonstrated the capability to automate moderately persuasive multi-turn spear-phishing attacks, similar to GPT-4 Turbo when evaluated by humans.

• Mitigation strategies and guardrails: The paper introduces several guardrails developed to mitigate identified risks, including Prompt Guard (for reducing prompt injection attacks), Code Shield (for reducing insecure code suggestions), and Llama Guard 3 (for reducing compliance with malicious prompts). These guardrails are publicly released and shown to significantly reduce various risks when implemented. One caveat is that since Meta open-sources all its models, malicious actors could use Llama 3 without guardrails, potentially rendering these mitigations ineffective in such cases.

• Limitations and future work: The authors acknowledge limitations in their assessment, such as the need for maximizing model efficacy through techniques like agent scaffolding, scaling up cross-checks between human and LLM judges, and implementing continuous assessment of model risk over time. They encourage further development of public benchmarks and risk mitigation strategies by the broader research community.

Link to the paper: https://arxiv.org/pdf/2408.01605