Executive Summary
OpenAI has announced it will no longer use or report scores for the SWE-bench Verified benchmark, a standard for measuring AI coding capabilities. An internal analysis revealed two critical issues: data contamination, where models show evidence of having been trained on the benchmark's problems and solutions, and flawed test cases. The company found that nearly 60% of audited problems had tests that were either too specific, rejecting correct solutions, or too broad, requiring unspecified functionality. OpenAI now recommends the community use SWE-bench Pro as an interim measure while new, uncontaminated evaluations are developed.
Key Takeaways
* Benchmark Deprecation: OpenAI is ceasing its use of SWE-bench Verified for measuring frontier model performance in software engineering tasks and recommends other developers do the same.
* Data Contamination: Analysis indicates that frontier models have been trained on the benchmark's source problems, as they can reproduce original human-written solutions verbatim. This means high scores may reflect memorization rather than true coding ability.
* Flawed Test Cases: An audit of difficult problems revealed that 59.4% contain significant flaws. These include:
* Narrow Tests (35.5%): Test cases enforce specific, unstated implementation details (e.g., a precise function name), causing functionally correct code to fail.
* Wide Tests (18.8%): Tests check for functionality that was not included in the original problem description.
* Recommended Alternative: Until better evaluations are created, OpenAI recommends the industry report results on SWE-bench Pro.
* Future Goal: The company is focused on building new, uncontaminated benchmarks to more accurately track the real-world coding capabilities of AI models.
Strategic Importance
This announcement highlights the growing challenge of creating robust, un-gamed benchmarks for increasingly powerful AI models. By publicly deprecating its own popular benchmark, OpenAI signals a commitment to rigorous evaluation and pushes the industry to address the critical issue of training data contamination to ensure that reported progress reflects genuine advances in capability.