OpenAI Finds About 30% of SWE-Bench Pro Tasks Are Flawed
The finding undermines the reliability of reported AI coding performance gains and questions the validity of popular benchmarks used to measure AI progress.
Reporting from 1 source: GIGAZINE.
OpenAI analyzed the SWE-Bench Pro coding benchmark and found that approximately 30% of its tasks have evaluation problems. Issues include overly strict tests, insufficient explanations, narrow test coverage, and misleading prompts. The company withdrew its earlier recommendation to use SWE-Bench Pro and called for new benchmarks designed by experienced software engineers.
OpenAI used a pipeline combining automated filters, a Codex-based AI agent, researcher judgment, and five experienced software engineers. They first identified 286 suspicious tasks, then manually reviewed them. The automated pipeline flagged 27.4% of tasks as broken, while human reviewers found 34.1% problematic. The company estimates roughly 30% of SWE-Bench Pro tasks are flawed. OpenAI noted that GitHub issues and pull requests, which form the basis of the benchmark, are designed for human collaboration, not for clean AI evaluation. The company has retracted its previous recommendation to adopt SWE-Bench Pro and now says new benchmarks must be designed by experienced developers specifically for AI evaluation.
Synthesized by Yomimono from the 1 cited source below, including Japanese-language reporting where cited, then editorially reviewed before publishing.