ASMR-Bench: Auditing for Sabotage in ML Research

Dimensions overall score 7.0

• safely deploying potentially subversive AI systems, and control evaluations test these protocols in specific settings such as code generation [27, 22], agentic tasks [28], AI R&D [11], and diffuse settings [29]. Sandbagging is a related phenomenon where models strategically perform worse than they are capable of [30]claim 82%
• detect: no auditor, whether LLM or human, reliably identified sabotaged codebases or pinpointed the specific sabotage. We found that predicting experiment outcomes is likely the most important 11 skill for reliably producing sabotages. LLMs are also capable of generating sabotages that evade similarly capable modelsclaim 82%
• coordinated human auditing, and wrote the paper. VH conceived the project and helped define its scope and requirements. JS and BS advised the project throughout, AB advised the project at its inception. References [1] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Pressclaim 82%
• results. Each sabotage modifies implementation details, such as hyperparameters, training data, or evaluation code, while preserving the high-level methodology described in the paperclaim 82%
• pruning [44], adversarial robustness [45, 46, 47], ML for COVID-19 [48], and ML reproducibility more broadly [49]. If unintentional design flaws of this kind are common enough to go undetected by peer review, intentional design sabotages that mimic them may be unincriminating and even harder to catchclaim 82%
• same-capability LLM auditors. We release ASMR-Bench to support research on monitoring and auditing techniques for AI-conducted research. 1 Introductionclaim 82%