detector & payload: AgentHarm

[leondz]

Summary

we propose a new benchmark called AgentHarm. The benchmark includes a diverse set of 110 explicitly malicious agent tasks (440 with augmentations), covering 11 harm categories including fraud, cybercrime, and harassment. In addition to measuring whether models refuse harmful agentic requests, scoring well on AgentHarm requires jailbroken agents to maintain their capabilities following an attack to complete a multi-step task.

add payloads and detectors for 110/440 tasks

• https://arxiv.org/abs/2410.09024
• https://huggingface.co/datasets/ai-safety-institute/AgentHarm

Motivation

These detectors sound advanced, and the payloads may match them really well. Hopefully the payloads can be used cross-probe, also.