A new model comes out, and the announcement is full of numbers: a record on one test, a big jump on another. A few weeks later, a competitor claims the top spot. If you choose AI tools or models for yourself or your team, you need to know what these numbers can tell you — and what they cannot.
This is part 4 of Level 4 in our AI Productivity path. You will learn what a benchmark really measures, why tests stop being useful, what contamination and test settings do to a score, how composite indices and agent tests work, and why the last test must always be your own. All model results in this article come straight from our LLM database, which is based on data from Epoch AI.
What a benchmark actually is
A benchmark is a fixed set of tasks with a fixed scoring rule. Most scores are the share of tasks a model solved, in percent. Some are different: the METR Time Horizon is a duration, and the Epoch Capabilities Index is an index. Our LLM pages track 16 benchmarks for 343 models; the data was last retrieved on October 17, 2026.
Every percentage needs a reference point. On tests with four answer options, such as MMLU or GPQA Diamond, pure guessing already gives 25%. A score of 30% on such a test is close to chance.
| Group | Examples in our data | A high score shows | It does not show |
|---|---|---|---|
| Knowledge and reasoning | MMLU, GPQA Diamond, Humanity’s Last Exam, SimpleQA Verified | Strong results in written expert exams | That the model knows your company or today’s news |
| Mathematics | GSM8K, MATH Level 5, FrontierMath | Multi-step calculation and proof | That your spreadsheet formulas will be right |
| Coding and agents | SWE-bench Verified, Aider Polyglot, Terminal-Bench | Fixing real code, working in a command line | That it works safely in your systems |
| Abstraction | ARC-AGI, ARC-AGI-2 | Learning a new rule from a few examples | General intelligence |
| Overall | Epoch Capabilities Index, METR Time Horizon | Broad capability; length of tasks an agent can finish | Fit for your specific task |
In the ranking tables of this article, each row shows the model, its developer, the release date and the result. Watch the release date: a model that came out last week may simply not have been tested on every benchmark yet. And watch the brackets after a model name, such as “(high)” or “(max)”: they name the setting used for that result.
A score is never just a property of the model. It depends on many choices made by the people who ran the test:
Saturation: when a test stops telling models apart
A benchmark is saturated when the best models score so close to the maximum that the remaining differences mean little. MMLU — multiple-choice questions from 57 subjects — was the standard knowledge test for years. Today the best models are so close to its ceiling that hardly any new results are published.
The ceiling is also lower than it looks. Researchers who re-checked MMLU estimated that 6.49% of its questions contain errors; in the virology subset, 57% of the questions they analyzed were flawed. (Source: Gema et al., “Are We Done with MMLU?”, 2024, arxiv.org) Above a certain level, a higher score can mean that a model matches the wrong answer key.
Harder tests take over. GPQA Diamond has 198 graduate-level science questions that non-experts fail even with web access. Humanity’s Last Exam (HLE) has 2,500 expert-written questions from many fields and was built by the Center for AI Safety and Scale AI. FrontierMath uses unpublished math problems that Epoch AI keeps private. The chart shows the pattern: each new test starts low and climbs, while the older ones flatten near the top.
These are the current leaders on Humanity’s Last Exam:
| # | Model | Released | Result |
|---|---|---|---|
| 1 | GPT 6 Astra OpenAI | Sep 3, 2026 | 54.8% |
| 2 | Fable 5.1 (xhigh) Anthropic | Sep 1, 2026 | 46.5% |
| 3 | Gemini 3.1 Pro Preview (thinking high) Google DeepMind | Feb 19, 2026 | 46.4% |
| 4 | Sonnet 5.5 (xhigh) Anthropic | Sep 28, 2026 | 46.1% |
| 5 | Gemini 3.8 Flash Google DeepMind | Sep 2, 2026 | 44.5% |
Expert-written questions at the edge of human knowledge across many fields, built to stay hard after older tests were solved. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Contamination: when the model has seen the test
Contamination means that test questions, or their answers, ended up in a model’s training data. The model may then remember instead of solve. Every test that is published on the internet is at risk, and the risk grows with its age.
Typical warning signs: a model does much better on an old public test than on a newer one that measures the same skill; or its answers repeat the exact wording of the official solutions. The best protection is a test that was never published: private problem sets such as FrontierMath, fresh questions — or your own tasks from your own work.
You can run a simple check yourself. Take a few questions from a public test and reword them: change the names, the numbers and the order of the answer options, but keep the task the same. If a model does much worse on the reworded versions, it probably relied on memory rather than on skill.
Settings: same model, different score
The same model can get very different scores, depending on how the test was run. Five settings matter most:
- Reasoning effort. Many models can “think” longer before they answer. In our data you will see model names with “(high)” or “(max)” — the same model with more thinking time, and often a higher score.
- Attempts. One answer per task is not the same as the best of several tries.
- Tools. SimpleQA Verified is answered without tools; a model with web search would score differently.
- Agent software. On Terminal-Bench, the score belongs to a model together with the agent software that runs it. On SWE-bench Verified, our data comes from Epoch AI’s own runs with one fixed setup; vendors often report higher numbers with their own tooling.
- Test size. GPQA Diamond has 198 questions, so one question more or less moves the score by about half a percentage point. A gap of one or two points between two models is often noise.
Composite indices and agent tests
Not every model is tested on every benchmark. The chart shows how many models in our database have results for each test. Some tests cover more than two hundred models, others only a few dozen — so a missing score is not a zero, and a ranking on a rare test leaves most models out.
The Epoch Capabilities Index (ECI) solves this. It combines the results of more than 50 benchmarks into one number per model, so that models can be compared even if they never took the same tests. It is an index, not a percentage: Epoch AI scales it so that Claude 3.5 Sonnet stands at 130 and GPT-5 at 150. An index is good for a first overview, but it hides strengths and weaknesses in single areas.
The METR Time Horizon measures something else: the length of tasks — counted in the time a human expert needs — that a model completes correctly in about half of the attempts. METR found that this horizon has been doubling roughly every seven months since 2019. (Source: METR, “Measuring AI Ability to Complete Long Tasks”, 2025, arxiv.org) Keep the 50% in mind: at that task length, the model also fails about half of the time. METR’s tasks come from software work, so the horizon says most about that kind of task. A horizon of two hours does not mean that a model can work on your project for two hours without supervision.
| # | Model | Released | Result |
|---|---|---|---|
| 1 | Claude Mythos Preview (Early) Anthropic | Apr 7, 2026 | 17 h 25 min |
| 2 | Claude Opus 4.6 (unknown thinking) Anthropic | Feb 5, 2026 | 11 h 59 min |
| 3 | Gemini 3.1 Pro Preview Google DeepMind | Feb 19, 2026 | 6 h 24 min |
| 4 | GPT-5.2 (high) OpenAI | Dec 11, 2025 | 5 h 52 min |
| 5 | GPT-5.3 Codex OpenAI | Feb 5, 2026 | 5 h 50 min |
The length of the tasks a model completes correctly about half of the time, measured by how long a human expert needs for them. The number is a duration, not a score: a longer horizon means the model can work on its own for longer. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
From leaderboard to your own decision
Benchmarks are a filter, not a verdict. Use them to build a shortlist, then test the shortlist on your own work. Level 3 shows how to choose the right LLM for the task, and part 5 of this level how to test and evaluate your prompts with a test set.
Seven questions help you judge any benchmark claim — in a vendor announcement, a news article or a ranking:
| Question | Why it matters | Red flag |
|---|---|---|
| Is the test saturated? | Top scores near the maximum say little | Records on MMLU, GSM8K or HellaSwag |
| Could the model have seen the test? | Contamination inflates scores | Only old, public tests are reported |
| Who ran the test? | Vendors often use their own setup | No independent results |
| Which settings were used? | Thinking time, attempts and tools change scores | Settings are not stated |
| How large is the gap? | Small tests are noisy | A ranking decided by one or two points |
| Does the test match your task? | Each benchmark measures specific skills | A coding score used to pick a writing model |
| Do several tests agree? | A single test can mislead | One record, no other results |