Re-Evaluating EVMBench: Are AI Agents Ready for Smart Contract Security?

Dimensions overall score 8.0

GitHub Code Pulse

No public code linked for this paper yet.

• EVMbench's evaluation scope is narrow, testing only 14 agent configurations with most models tested on their vendor scaffold only.limitation 90%
  Evidencepartial

  its narrow evaluation scope (14 agent configurations, most models tested on only their vendor scaffold)

  Implicationpartial

  Directly stated in abstract as an identified limitation

  Verificationpartial

  partial

• EVMbench relies on audit-contest data published before every model's release that models may have seen during training.limitation 90%
  Evidencepartial

  its reliance on audit-contest data published before every model's release that models may have seen during training

  Implicationpartial

  Directly stated in abstract as an identified limitation

  Verificationpartial

  partial

• Agents' detection results are not stable, with rankings shifting across configurations, tasks, and datasets.result 90%
  Evidencepartial

  agents' detection results are not stable, with rankings shifting across configurations, tasks, and datasets

  Implicationpartial

  Directly stated as finding (1) in abstract with supporting evaluation

  Verificationpartial

  partial

• On real-world incidents, no agent succeeds at end-to-end exploitation across all 110 agent-incident pairs despite detecting up to 65% of vulnerabilities.result 90%
  Evidencepartial

  on real-world incidents, no agent succeeds at end-to-end exploitation across all 110 agent-incident pairs despite detecting up to 65% of vulnerabilities

  Implicationpartial

  Directly stated as finding (2) in abstract with specific numeric evidence

  Verificationpartial

  partial

• Scaffolding materially affects results, with an open-source scaffold outperforming vendor alternatives by up to 5 percentage points.result 90%
  Evidencepartial

  scaffolding materially affects results, with an open-source scaffold outperforming vendor alternatives by up to 5 percentage points

  Implicationpartial

  Directly stated as finding (3) in abstract with specific numeric evidence

  Verificationpartial

  partial

• The paper's findings challenge the narrative that fully automated AI auditing is imminent.market 85%
  Evidencepartial

  These findings challenge the narrative that fully automated AI auditing is imminent

  Implicationpartial

  Directly stated conclusion in abstract, though 'narrative' interpretation requires some inference

  Verificationpartial

  partial

• Agents reliably catch well-known patterns and respond strongly to human-provided context, but cannot replace human judgment.technical 85%
  Evidencepartial

  Agents reliably catch well-known patterns and respond strongly to human-provided context, but cannot replace human judgment

  Implicationpartial

  Directly stated conclusion in abstract about agent capabilities and limitations

  Verificationpartial

  partial

• For audit firms, agents are most effective within a human-in-the-loop workflow where AI handles breadth and human auditors contribute protocol-specific knowledge and adversarial reasoning.market 80%
  Evidencepartial

  For audit firms, agents are most effective within a human-in-the-loop workflow where AI handles breadth and human auditors contribute protocol-specific knowledge and adversarial reasoning

  Implicationpartial

  Directly stated recommendation in abstract, though effectiveness claim requires some inference

  Verificationpartial

  partial

Startup potential card