IndexMapAbout
Tool / RuntimeLLM

SWE-bench

Also known as: SWE-bench Verified · SWE-bench Lite

Plain English

The standard benchmark for testing whether an AI coding agent can actually fix real bugs in real open-source GitHub repositories and pass the project's test suite.

Technical Overview

Created by Princeton NLP researchers, SWE-bench evaluates models and agent harnesses on real Python GitHub issues paired with repository snapshots and hidden unit tests inside isolated Docker sandboxes. Its human-audited 500-problem SWE-bench Verified subset became the headline leaderboard for frontier coding models (Claude, GPT, Gemini) and CLI agents (Claude Code, Codex).

Chronological Entries (2)

#2Release
AI-Assisted

OpenAI and the SWE-bench authors released SWE-bench Verified, a human-filtered subset of 500 unambiguous, solvable issues that eliminated underspecified prompts and flaky tests—turning SWE-bench Verified into the industry's primary coding-agent yardstick.

Direct Relations (4)