WoW-bench

WoW-bench is a novel benchmark specifically created to rigorously test the capabilities of frontier large language models (LLMs) when deployed as autonomous agents within intricate enterprise environments. It addresses a critical gap in existing evaluation methods, which often overlook the unique challenges of enterprise systems, such as limited observability, vast database states, and hidden workflows that produce cascading side effects. The benchmark utilizes a realistic ServiceNow-based environment, meticulously incorporating over 4,000 business rules and 55 active workflows. Through 234 distinct tasks, WoW-bench evaluates an agent's proficiency in constrained task completion and its capacity for enterprise dynamics modeling. This is crucial for developing reliable AI agents, as LLMs often suffer from 'dynamics blindness,' failing to predict the invisible consequences of their actions, leading to silent constraint violations. Researchers and ML engineers working on enterprise AI, autonomous agents, and system reliability are the primary users of this benchmark.