OpenAI and the SWE-bench authors released SWE-bench Verified, a human-filtered subset of 500 unambiguous, solvable issues that eliminated underspecified prompts and flaky tests—turning SWE-bench Verified into the industry's primary coding-agent yardstick.
SWE-bench
Also known as: SWE-bench Verified · SWE-bench Lite
Plain English
The standard benchmark for testing whether an AI coding agent can actually fix real bugs in real open-source GitHub repositories and pass the project's test suite.
Technical Overview
Created by Princeton NLP researchers, SWE-bench evaluates models and agent harnesses on real Python GitHub issues paired with repository snapshots and hidden unit tests inside isolated Docker sandboxes. Its human-audited 500-problem SWE-bench Verified subset became the headline leaderboard for frontier coding models (Claude, GPT, Gemini) and CLI agents (Claude Code, Codex).
Chronological Entries (2)
Jimenez et al. published SWE-bench, introducing 2,294 real-world software engineering tasks drawn from 12 popular Python repositories where candidate patches are graded strictly by executing the repository's actual unit tests.
Direct Relations (4)
- Part ofEvals & Benchmarks
- UsesAgent Harness
- UsesClaude Code
- UsesCodex