Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• EVMbench's evaluation scope is narrow, testing only 14 agent configurations with most models tested on their vendor scaffold only.limitation 90%
• EVMbench relies on audit-contest data published before every model's release that models may have seen during training.limitation 90%
• Agents' detection results are not stable, with rankings shifting across configurations, tasks, and datasets.result 90%
• On real-world incidents, no agent succeeds at end-to-end exploitation across all 110 agent-incident pairs despite detecting up to 65% of vulnerabilities.result 90%
• Scaffolding materially affects results, with an open-source scaffold outperforming vendor alternatives by up to 5 percentage points.result 90%
• The paper's findings challenge the narrative that fully automated AI auditing is imminent.market 85%
• Agents reliably catch well-known patterns and respond strongly to human-provided context, but cannot replace human judgment.technical 85%
• For audit firms, agents are most effective within a human-in-the-loop workflow where AI handles breadth and human auditors contribute protocol-specific knowledge and adversarial reasoning.market 80%

Competitive landscape

Competitor map is still being generated for this paper. Enable generation or check back soon.

Startup potential card