Executive Summary
In partnership with Paradigm, a new benchmark named EVMbench has been introduced to evaluate the ability of AI agents to interact with smart contract vulnerabilities. EVMbench provides a structured environment for measuring an AI's performance across three key security tasks: detecting flaws, patching code to fix them, and actively exploiting them. The initiative aims to track emerging AI-driven cyber risks and encourage the development of AI systems for defensive security in the high-value blockchain ecosystem.
Key Takeaways
* Product: EVMbench, a benchmark for evaluating AI agents on smart contract security.
* Core Functionality: It assesses AI performance in three modes:
* Detect: Finding ground-truth vulnerabilities in a smart contract repository.
* Patch: Modifying vulnerable contracts to eliminate exploitability while preserving functionality.
* Exploit: Executing fund-draining attacks against deployed contracts in a sandboxed environment.
* Dataset: The benchmark is built on 120 curated, high-severity vulnerabilities sourced from 40 real-world security audits and competitions.
* Initial Findings: A frontier model, GPT-5.3-Codex, achieved a 72.2% success rate in the 'exploit' mode, a significant improvement over prior models. Agent performance was weaker on the more nuanced 'detect' and 'patch' tasks.
* Availability: The benchmark's tasks, evaluation framework, and tooling are being released to the research community.
Strategic Importance
This announcement establishes a standard for measuring AI's dual-use capabilities in the critical domain of blockchain security. It positions the company as a leader in both advancing powerful AI systems and responsibly guiding their application toward defensive cybersecurity.