A new model comes out, and the launch post shows a coding score a few points above the competition. Should your team switch? The number alone cannot tell you. You need to know which tasks stand behind it, how the result was produced and how much it can vary by chance.
This is part 4 of Level 4 (Expert) in our AI Development path. You will learn what SWE-bench Verified, Aider Polyglot and Terminal-Bench measure, how their tasks are scored, where the numbers mislead and how to build a small benchmark from your own repository. Level 0 introduced the first of these benchmarks in what AI coding assistants do; the methods here build on testing AI features with evals from Level 2.
What a coding benchmark measures
A coding benchmark is a fixed set of programming tasks with an automatic check, almost always tests. The score is the share of tasks solved, usually with one attempt per task (pass@1). Two more terms matter for everything that follows. The harness, also called the scaffold, is the software around the model: it gives the model tools such as a shell or a file editor, runs the loop and limits steps and time. Contamination means that tasks or their solutions were part of the model’s training data.
The three benchmarks from our LLM data test different kinds of work:
None of them measures what a solved task costs, how long it takes or whether your colleagues can maintain the code. Keep that in mind whenever you read a leaderboard.
SWE-bench Verified: real issues, hidden tests
SWE-bench was introduced in 2023 (Source: Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, ICLR 2024). It contains 2,294 tasks built from real GitHub issues in 12 popular Python repositories. Each task gives the model the issue text and the repository as it was before the fix. The model, usually inside an agent harness, edits the code and hands in a patch: its change as a diff. Then two groups of tests decide:
- Fail-to-pass tests came with the real fix: they failed before it and passed after it. They stay hidden from the model, and its patch must make them pass.
- Pass-to-pass tests are existing tests that passed before the fix. They must keep passing, so the patch breaks nothing else.
Only if both groups pass does the task count as resolved. The original set had a weakness: some issues were too vague to solve, and some tests rejected valid solutions. In August 2024, OpenAI released SWE-bench Verified. For it, 93 software developers experienced in Python screened 1,699 tasks for vague descriptions and unfair tests; the 500 tasks of SWE-bench Verified passed this check. The annotators also estimated the effort for an experienced developer: 196 tasks take less than 15 minutes, and only 45 take more than an hour (Source: OpenAI, “Introducing SWE-bench Verified”, 2024, openai.com). So the benchmark measures small, well-described fixes, not the design of a new feature.
In our data, all SWE-bench Verified results come from Epoch AI’s own runs with one fixed harness. That makes them comparable with each other, and often lower than the numbers in vendor announcements:
| # | Model | Released | Result |
|---|---|---|---|
| 1 | Claude Opus 4.7 (max) Anthropic | Apr 16, 2026 | 83.5% |
| 2 | GPT-5.5 (xhigh) OpenAI | Apr 23, 2026 | 80.6% |
| 3 | Gemini 3.5 Flash Google DeepMind | May 19, 2026 | 79.3% |
| 4 | Claude Opus 4.6 (no thinking) Anthropic | Feb 5, 2026 | 78.7% |
| 5 | GLM-5.2 Z.ai (Zhipu AI) | Jun 16, 2026 | 78.7% |
500 real issues from open-source Python projects, checked by humans. The model has to change the code so that the project's tests pass. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Aider Polyglot and Terminal-Bench
Aider Polyglot: editing code in six languages
The makers of the open-source coding tool Aider published the Polyglot benchmark in December 2024. It uses exercises from the learning platform Exercism in C++, Go, Java, JavaScript, Python and Rust. Of 697 exercises, it keeps the 225 that at most three of seven leading models could solve at the time (Source: Aider, “o1 tops aider’s new polyglot leaderboard”, 2024, aider.chat). The model has to edit the given files in a fixed edit format. If the unit tests fail, it sees their output and gets a second attempt. Besides the share of solved exercises, the leaderboard reports how often a model used the edit format correctly. That matters for every tool that applies AI edits automatically.
What Aider Polyglot does not cover: the exercises are small and self-contained, there is no large codebase to understand, and the solutions are public. These results come from the Aider leaderboard, collected by Epoch AI:
| # | Model | Released | Result |
|---|---|---|---|
| 1 | GPT-5 (high) OpenAI | Aug 7, 2025 | 88.0% |
| 2 | o3-pro OpenAI | Jun 10, 2025 | 84.9% |
| 3 | Gemini 2.5 Pro Preview (Jun 2025) Google DeepMind | Jun 5, 2025 | 83.1% |
| 4 | o3 (high) OpenAI | Apr 16, 2025 | 81.3% |
| 5 | Grok 4 xAI | Jul 9, 2025 | 79.6% |
225 difficult programming exercises in C++, Go, Java, JavaScript, Python and Rust. It measures whether a model can edit existing code correctly, with a second attempt after seeing the test output. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Terminal-Bench: an agent in a real command line
Terminal-Bench was built by Stanford University and the Laude Institute. Version 2.0 from November 2025 contains 89 tasks. Each task has its own environment in a container, a reference solution written by a person and tests that check the result (Source: Merrill et al., “Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces”, 2026, arxiv.org). The agent has to set up software, repair a system or process data: work that takes many commands in a row.
Here, the score belongs to a pair: a model and the agent software that drives it. The same model can rank quite differently with two agents. Our table shows the best result per model:
| # | Model | Released | Result |
|---|---|---|---|
| 1 | GPT-5.5 OpenAI | Apr 23, 2026 | 84.7% |
| 2 | GPT-5.4 OpenAI | Mar 5, 2026 | 81.8% |
| 3 | Gemini 3.1 Pro Google DeepMind | Feb 19, 2026 | 80.2% |
| 4 | Claude Opus 4.7 Anthropic | Apr 16, 2026 | 80.2% |
| 5 | Claude Opus 4.6 Anthropic | Feb 5, 2026 | 79.8% |
Tasks that an agent has to solve in a real command line: set up software, fix a system, process data. The score belongs to a model together with its agent software; we show the best result per model. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Not every new model is tested on every benchmark, so the newest models in our LLM database may be missing from these tables.
How fast the records moved
The chart shows the record of each benchmark over time: the best result in our data, by the release date of the models.
Three lessons follow. First, results rose steeply on all three benchmarks within months, so a record from a year ago tells you little about today’s models. Second, a benchmark close to its ceiling stops separating models: the last unsolved tasks are often the ones with flawed tests. Third, a line that ends does not mean progress stopped. It means that nobody has run the benchmark on newer models yet.
Where the numbers mislead
Contamination and flawed tests. SWE-bench tasks come from public repositories, and the fixes are public too. In February 2026, OpenAI stopped reporting SWE-bench Verified. It had audited the tasks that models often failed, and at least 59.4 percent of them had flawed tests that can reject correct solutions. It also found that frontier models could reproduce the original fix or the issue text from the task ID and a short hint (Source: OpenAI, “Why we no longer evaluate SWE-bench Verified”, 2026, openai.com). OpenAI then recommended SWE-bench Pro from Scale AI. In July 2026, it estimated that about 30 percent of SWE-bench Pro’s tasks are broken as well and withdrew the recommendation. Its explanation applies to every benchmark built this way: issues and pull requests were written for people working together, not as self-contained test tasks (Source: OpenAI, “Separating signal from noise in coding evaluations”, 2026, openai.com).
Different harnesses. A vendor’s number and an independent number for the same model are often not comparable. Vendors may use their own harness, more attempts or more compute. Anthropic, for example, reported one SWE-bench Verified result for Claude 3.7 Sonnet with its standard setup and a clearly higher one with a custom scaffold (Source: Anthropic, “Claude 3.7 Sonnet and Claude Code”, 2025, anthropic.com). Read the footnotes before you compare.
Chance. Every benchmark is a sample of tasks. The fewer tasks, the more a score can vary by chance:
With 500 tasks, two models that are 2 points apart may be equally good. With the 50 tasks of a typical private eval, only large differences count. Evan Miller of Anthropic recommends reporting error bars and comparing models on the same tasks, question by question (Source: Miller, “Adding Error Bars to Evals”, 2024, arxiv.org).
The table lists the traps and what to check before you trust a number:
| Trap | What happens | What to check |
|---|---|---|
| Contamination | The model saw tasks or fixes during training | Are the tasks public? Are there results on new, unpublished tasks? |
| Flawed tests | Correct solutions fail, or wrong ones pass | Was the task set audited? How close is the top score to the ceiling? |
| Harness | Vendor numbers use other tools, attempts or compute | Who ran the test, with which harness and how many attempts? |
| Chance | Small gaps come from the task sample | How many tasks? Are error bars reported? |
| Missing costs | The score ignores tokens, time and review effort | What does one solved task cost, and how long does it take? |
| Missing models | Not every model has run every benchmark | Is the model you care about in the data at all? |
Your repository is the real test
Public benchmarks make a good first filter: they show which models are worth testing. The decision belongs to a private benchmark built from your own code, with the same idea as SWE-bench:
- Collect tasks. Take recently closed issues with a merged fix and tests. Fifty is a useful start; more is better. Recent tasks are less likely to be in any model’s training data.
- Freeze the environment. For each task, prepare a container with the repository at the commit before the fix and everything the tests need.
- Hide the decisive tests. Remove the tests that came with the fix and keep them for scoring as fail-to-pass tests. The existing test suite serves as pass-to-pass tests.
- Use your real harness. Run each candidate model in the agent or IDE tool your team actually uses, with the same limits. Run each task several times.
- Measure more than the pass rate. Record the cost per resolved task, the time, the size of the diff and a reviewer’s verdict on a sample: would you merge this?
- Compare task by task. Look at the tasks where the models differ, not only at the totals.
The cost per resolved task often separates models more clearly than the pass rate. Part 5 of this level shows how to measure and lower the cost per successful task. Whichever model wins, its code still passes the gates of your AI-assisted coding workflow.
Tools can help with parts of this. According to the vendor, TestDriver.ai runs pull requests in a desktop sandbox and builds regression tests, which helps when old issues lack tests. If your shortlist includes open-weight coding models, a GPU cloud such as Runpod can run them for the test. App builders such as Emergent also write code with AI models, so their output deserves the same checks. We have not tested these tools.
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.