{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Productivity"],"content":{"markdown":"A new model comes out, and the announcement is full of numbers: a record on one test, a big jump on another. A few weeks later, a competitor claims the top spot. If you choose AI tools or models for yourself or your team, you need to know what these numbers can tell you — and what they cannot.\n\nThis is part 4 of Level 4 in our AI Productivity path. You will learn what a benchmark really measures, why tests stop being useful, what contamination and test settings do to a score, how composite indices and agent tests work, and why the last test must always be your own. All model results in this article come straight from our [LLM database](/en/llm/), which is based on data from Epoch AI.\n\n## What a benchmark actually is\n\nA **benchmark** is a fixed set of tasks with a fixed scoring rule. Most scores are the share of tasks a model solved, in percent. Some are different: the METR Time Horizon is a duration, and the Epoch Capabilities Index is an index. Our LLM pages track {{\u003c stat \"benchmarks\" \u003e}} benchmarks for {{\u003c stat \"models\" \u003e}} models; the data was last retrieved on {{\u003c stat \"retrieved\" \u003e}}.\n\nEvery percentage needs a reference point. On tests with four answer options, such as MMLU or GPQA Diamond, pure **guessing** already gives 25%. A score of 30% on such a test is close to chance.\n\n| Group | Examples in our data | A high score shows | It does not show |\n|---|---|---|---|\n| Knowledge and reasoning | MMLU, GPQA Diamond, Humanity's Last Exam, SimpleQA Verified | Strong results in written expert exams | That the model knows your company or today's news |\n| Mathematics | GSM8K, MATH Level 5, FrontierMath | Multi-step calculation and proof | That your spreadsheet formulas will be right |\n| Coding and agents | SWE-bench Verified, Aider Polyglot, Terminal-Bench | Fixing real code, working in a command line | That it works safely in your systems |\n| Abstraction | ARC-AGI, ARC-AGI-2 | Learning a new rule from a few examples | General intelligence |\n| Overall | Epoch Capabilities Index, METR Time Horizon | Broad capability; length of tasks an agent can finish | Fit for your specific task |\n\nIn the ranking tables of this article, each row shows the model, its developer, the release date and the result. Watch the release date: a model that came out last week may simply not have been tested on every benchmark yet. And watch the brackets after a model name, such as \"(high)\" or \"(max)\": they name the setting used for that result.\n\nA score is never just a property of the model. It depends on many choices made by the people who ran the test:\n\n![Six cards: tasks, scoring, model version, settings, tools and agent software, and who ran the test](/images/learn/en/read-llm-benchmarks-like-an-expert-behind-score.webp \"One number, many choices: before you compare two scores, check that these six things match.\")\n\n## Saturation: when a test stops telling models apart\n\nA benchmark is **saturated** when the best models score so close to the maximum that the remaining differences mean little. MMLU — multiple-choice questions from 57 subjects — was the standard knowledge test for years. Today the best models are so close to its ceiling that hardly any new results are published.\n\nThe ceiling is also lower than it looks. Researchers who re-checked MMLU estimated that 6.49% of its questions contain errors; in the virology subset, 57% of the questions they analyzed were flawed. (Source: Gema et al., \"Are We Done with MMLU?\", 2024, arxiv.org) Above a certain level, a higher score can mean that a model matches the wrong answer key.\n\nHarder tests take over. **GPQA Diamond** has 198 graduate-level science questions that non-experts fail even with web access. **Humanity's Last Exam** (HLE) has 2,500 expert-written questions from many fields and was built by the Center for AI Safety and Scale AI. **FrontierMath** uses unpublished math problems that Epoch AI keeps private. The chart shows the pattern: each new test starts low and climbs, while the older ones flatten near the top.\n\n![Line chart: record scores over time for MMLU, GPQA Diamond and Humanity's Last Exam, with the guessing level for four-option questions](/images/learn/en/read-llm-benchmarks-like-an-expert-saturation.webp \"Record scores over time: MMLU flattened first, GPQA Diamond is close to its ceiling, Humanity's Last Exam is still rising. Data: Epoch AI, CC BY 4.0.\")\n\nThese are the current leaders on Humanity's Last Exam:\n\n{{\u003c llm-top bench=\"hle\" n=\"5\" \u003e}}\n\n## Contamination: when the model has seen the test\n\n**Contamination** means that test questions, or their answers, ended up in a model's training data. The model may then remember instead of solve. Every test that is published on the internet is at risk, and the risk grows with its age.\n\nTypical warning signs: a model does much better on an old public test than on a newer one that measures the same skill; or its answers repeat the exact wording of the official solutions. The best protection is a test that was never published: private problem sets such as FrontierMath, fresh questions — or your own tasks from your own work.\n\nYou can run a simple check yourself. Take a few questions from a public test and reword them: change the names, the numbers and the order of the answer options, but keep the task the same. If a model does much worse on the reworded versions, it probably relied on memory rather than on skill.\n\n## Settings: same model, different score\n\nThe same model can get very different scores, depending on how the test was run. Five settings matter most:\n\n- **Reasoning effort.** Many models can \"think\" longer before they answer. In our data you will see model names with \"(high)\" or \"(max)\" — the same model with more thinking time, and often a higher score.\n- **Attempts.** One answer per task is not the same as the best of several tries.\n- **Tools.** SimpleQA Verified is answered without tools; a model with web search would score differently.\n- **Agent software.** On Terminal-Bench, the score belongs to a model together with the agent software that runs it. On SWE-bench Verified, our data comes from Epoch AI's own runs with one fixed setup; vendors often report higher numbers with their own tooling.\n- **Test size.** GPQA Diamond has 198 questions, so one question more or less moves the score by about half a percentage point. A gap of one or two points between two models is often noise.\n\n{{\u003c callout type=\"tip\" title=\"Compare like with like\" \u003e}}\nBefore you compare two scores, check three things: the same benchmark version, the same settings, and the same source. A vendor's own number next to an independent result is not a fair comparison.\n{{\u003c /callout \u003e}}\n\n## Composite indices and agent tests\n\nNot every model is tested on every benchmark. The chart shows how many models in our database have results for each test. Some tests cover more than two hundred models, others only a few dozen — so a missing score is not a zero, and a ranking on a rare test leaves most models out.\n\n![Bar chart: number of models with results per benchmark in our LLM database](/images/learn/en/read-llm-benchmarks-like-an-expert-coverage.webp \"How many models have results for each benchmark in our LLM database. Data: Epoch AI, CC BY 4.0.\")\n\nThe **Epoch Capabilities Index** (ECI) solves this. It combines the results of more than 50 benchmarks into one number per model, so that models can be compared even if they never took the same tests. It is an index, not a percentage: Epoch AI scales it so that Claude 3.5 Sonnet stands at 130 and GPT-5 at 150. An index is good for a first overview, but it hides strengths and weaknesses in single areas.\n\nThe **METR Time Horizon** measures something else: the length of tasks — counted in the time a human expert needs — that a model completes correctly in about half of the attempts. METR found that this horizon has been doubling roughly every seven months since 2019. (Source: METR, \"Measuring AI Ability to Complete Long Tasks\", 2025, arxiv.org) Keep the 50% in mind: at that task length, the model also fails about half of the time. METR's tasks come from software work, so the horizon says most about that kind of task. A horizon of two hours does not mean that a model can work on your project for two hours without supervision.\n\n{{\u003c llm-top bench=\"metr-time-horizon\" n=\"5\" \u003e}}\n\n## From leaderboard to your own decision\n\nBenchmarks are a filter, not a verdict. Use them to build a shortlist, then test the shortlist on your own work. Level 3 shows how to [choose the right LLM for the task](/en/ai-productivity/learn/choose-the-right-llm-for-the-task/), and part 5 of this level how to [test and evaluate your prompts](/en/ai-productivity/learn/test-and-evaluate-your-prompts/) with a test set.\n\n![Funnel from public leaderboards to a shortlist, your own test set, a pilot with real users and a decision](/images/learn/en/read-llm-benchmarks-like-an-expert-funnel.webp \"Public benchmarks narrow the field; your own test set and a pilot make the decision.\")\n\nSeven questions help you judge any benchmark claim — in a vendor announcement, a news article or a ranking:\n\n| Question | Why it matters | Red flag |\n|---|---|---|\n| Is the test saturated? | Top scores near the maximum say little | Records on MMLU, GSM8K or HellaSwag |\n| Could the model have seen the test? | Contamination inflates scores | Only old, public tests are reported |\n| Who ran the test? | Vendors often use their own setup | No independent results |\n| Which settings were used? | Thinking time, attempts and tools change scores | Settings are not stated |\n| How large is the gap? | Small tests are noisy | A ranking decided by one or two points |\n| Does the test match your task? | Each benchmark measures specific skills | A coding score used to pick a writing model |\n| Do several tests agree? | A single test can mislead | One record, no other results |","text":"A new model comes out, and the announcement is full of numbers: a record on one test, a big jump on another. A few weeks later, a competitor claims the top spot. If you choose AI tools or models for yourself or your team, you need to know what these numbers can tell you — and what they cannot.\nThis is part 4 of Level 4 in our AI Productivity path. You will learn what a benchmark really measures, why tests stop being useful, what contamination and test settings do to a score, how composite indices and agent tests work, and why the last test must always be your own. All model results in this article come straight from our LLM database, which is based on data from Epoch AI.\nWhat a benchmark actually is A benchmark is a fixed set of tasks with a fixed scoring rule. Most scores are the share of tasks a model solved, in percent. Some are different: the METR Time Horizon is a duration, and the Epoch Capabilities Index is an index. Our LLM pages track 16 benchmarks for 343 models; the data was last retrieved on October 17, 2026.\nEvery percentage needs a reference point. On tests with four answer options, such as MMLU or GPQA Diamond, pure guessing already gives 25%. A score of 30% on such a test is close to chance.\nRead LLM Benchmarks Like an Expert Group Examples in our data A high score shows It does not show Knowledge and reasoning MMLU, GPQA Diamond, Humanity\u0026rsquo;s Last Exam, SimpleQA Verified Strong results in written expert exams That the model knows your company or today\u0026rsquo;s news Mathematics GSM8K, MATH Level 5, FrontierMath Multi-step calculation and proof That your spreadsheet formulas will be right Coding and agents SWE-bench Verified, Aider Polyglot, Terminal-Bench Fixing real code, working in a command line That it works safely in your systems Abstraction ARC-AGI, ARC-AGI-2 Learning a new rule from a few examples General intelligence Overall Epoch Capabilities Index, METR Time Horizon Broad capability; length of tasks an agent can finish Fit for your specific task In the ranking tables of this article, each row shows the model, its developer, the release date and the result. Watch the release date: a model that came out last week may simply not have been tested on every benchmark yet. And watch the brackets after a model name, such as \u0026ldquo;(high)\u0026rdquo; or \u0026ldquo;(max)\u0026rdquo;: they name the setting used for that result.\nA score is never just a property of the model. It depends on many choices made by the people who ran the test:\nOne number, many choices: before you compare two scores, check that these six things match. Saturation: when a test stops telling models apart A benchmark is saturated when the best models score so close to the maximum that the remaining differences mean little. MMLU — multiple-choice questions from 57 subjects — was the standard knowledge test for years. Today the best models are so close to its ceiling that hardly any new results are published.\nThe ceiling is also lower than it looks. Researchers who re-checked MMLU estimated that 6.49% of its questions contain errors; in the virology subset, 57% of the questions they analyzed were flawed. (Source: Gema et al., \u0026ldquo;Are We Done with MMLU?\u0026rdquo;, 2024, arxiv.org) Above a certain level, a higher score can mean that a model matches the wrong answer key.\nHarder tests take over. GPQA Diamond has 198 graduate-level science questions that non-experts fail even with web access. Humanity\u0026rsquo;s Last Exam (HLE) has 2,500 expert-written questions from many fields and was built by the Center for AI Safety and Scale AI. FrontierMath uses unpublished math problems that Epoch AI keeps private. The chart shows the pattern: each new test starts low and climbs, while the older ones flatten near the top.\nRecord scores over time: MMLU flattened first, GPQA Diamond is close to its ceiling, Humanity\u0026#39;s Last Exam is still rising. Data: Epoch AI, CC BY 4.0. These are the current leaders on Humanity\u0026rsquo;s Last Exam:\nHumanity\u0026#39;s Last Exam: the top 5 models #ModelReleasedResult 1GPT 6 Astra OpenAISep 3, 202654.8% 2Fable 5.1 (xhigh) AnthropicSep 1, 202646.5% 3Gemini 3.1 Pro Preview (thinking high) Google DeepMindFeb 19, 202646.4% 4Sonnet 5.5 (xhigh) AnthropicSep 28, 202646.1% 5Gemini 3.8 Flash Google DeepMindSep 2, 202644.5% Expert-written questions at the edge of human knowledge across many fields, built to stay hard after older tests were solved. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.\nContamination: when the model has seen the test Contamination means that test questions, or their answers, ended up in a model\u0026rsquo;s training data. The model may then remember instead of solve. Every test that is published on the internet is at risk, and the risk grows with its age.\nTypical warning signs: a model does much better on an old public test than on a newer one that measures the same skill; or its answers repeat the exact wording of the official solutions. The best protection is a test that was never published: private problem sets such as FrontierMath, fresh questions — or your own tasks from your own work.\nYou can run a simple check yourself. Take a few questions from a public test and reword them: change the names, the numbers and the order of the answer options, but keep the task the same. If a model does much worse on the reworded versions, it probably relied on memory rather than on skill.\nSettings: same model, different score The same model can get very different scores, depending on how the test was run. Five settings matter most:\nReasoning effort. Many models can \u0026ldquo;think\u0026rdquo; longer before they answer. In our data you will see model names with \u0026ldquo;(high)\u0026rdquo; or \u0026ldquo;(max)\u0026rdquo; — the same model with more thinking time, and often a higher score. Attempts. One answer per task is not the same as the best of several tries. Tools. SimpleQA Verified is answered without tools; a model with web search would score differently. Agent software. On Terminal-Bench, the score belongs to a model together with the agent software that runs it. On SWE-bench Verified, our data comes from Epoch AI\u0026rsquo;s own runs with one fixed setup; vendors often report higher numbers with their own tooling. Test size. GPQA Diamond has 198 questions, so one question more or less moves the score by about half a percentage point. A gap of one or two points between two models is often noise. ✦Compare like with like\nBefore you compare two scores, check three things: the same benchmark version, the same settings, and the same source. A vendor\u0026rsquo;s own number next to an independent result is not a fair comparison.\nComposite indices and agent tests Not every model is tested on every benchmark. The chart shows how many models in our database have results for each test. Some tests cover more than two hundred models, others only a few dozen — so a missing score is not a zero, and a ranking on a rare test leaves most models out.\nHow many models have results for each benchmark in our LLM database. Data: Epoch AI, CC BY 4.0. The Epoch Capabilities Index (ECI) solves this. It combines the results of more than 50 benchmarks into one number per model, so that models can be compared even if they never took the same tests. It is an index, not a percentage: Epoch AI scales it so that Claude 3.5 Sonnet stands at 130 and GPT-5 at 150. An index is good for a first overview, but it hides strengths and weaknesses in single areas.\nThe METR Time Horizon measures something else: the length of tasks — counted in the time a human expert needs — that a model completes correctly in about half of the attempts. METR found that this horizon has been doubling roughly every seven months since 2019. (Source: METR, \u0026ldquo;Measuring AI Ability to Complete Long Tasks\u0026rdquo;, 2025, arxiv.org) Keep the 50% in mind: at that task length, the model also fails about half of the time. METR\u0026rsquo;s tasks come from software work, so the horizon says most about that kind of task. A horizon of two hours does not mean that a model can work on your project for two hours without supervision.\nMETR Time Horizon: the top 5 models #ModelReleasedResult 1Claude Mythos Preview (Early) AnthropicApr 7, 202617 h 25 min 2Claude Opus 4.6 (unknown thinking) AnthropicFeb 5, 202611 h 59 min 3Gemini 3.1 Pro Preview Google DeepMindFeb 19, 20266 h 24 min 4GPT-5.2 (high) OpenAIDec 11, 20255 h 52 min 5GPT-5.3 Codex OpenAIFeb 5, 20265 h 50 min The length of the tasks a model completes correctly about half of the time, measured by how long a human expert needs for them. The number is a duration, not a score: a longer horizon means the model can work on its own for longer. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.\nFrom leaderboard to your own decision Benchmarks are a filter, not a verdict. Use them to build a shortlist, then test the shortlist on your own work. Level 3 shows how to choose the right LLM for the task, and part 5 of this level how to test and evaluate your prompts with a test set.\nPublic benchmarks narrow the field; your own test set and a pilot make the decision. Seven questions help you judge any benchmark claim — in a vendor announcement, a news article or a ranking:\nRead LLM Benchmarks Like an Expert (2) Question Why it matters Red flag Is the test saturated? Top scores near the maximum say little Records on MMLU, GSM8K or HellaSwag Could the model have seen the test? Contamination inflates scores Only old, public tests are reported Who ran the test? Vendors often use their own setup No independent results Which settings were used? Thinking time, attempts and tools change scores Settings are not stated How large is the gap? Small tests are noisy A ranking decided by one or two points Does the test match your task? Each benchmark measures specific skills A coding score used to pick a writing model Do several tests agree? A single test can mislead One record, no other results ","words":1720},"date":"2026-10-14","description":"What LLM benchmarks really measure, why tests saturate, how contamination and settings distort scores — and how to use benchmark data to choose a model.","image":"https://abxk.ai/images/cover/read-llm-benchmarks-like-an-expert.webp","json":"https://abxk.ai/en/ai-productivity/learn/read-llm-benchmarks-like-an-expert/index.json","kind":"page","language":"en","learning_path":{"category":"ai-productivity","level":4,"level_name":"Expert"},"modified":"2026-10-14","related_tools":["ChatGPT (GPT Image)","Google Gemini (Nano Banana)","Meta AI"],"section":"ai-productivity","tags":["llm","benchmarks","evaluation","model-choice","expert"],"title":"Read LLM Benchmarks Like an Expert","translations":[{"json":"https://abxk.ai/de/ai-productivity/learn/read-llm-benchmarks-like-an-expert/index.json","language":"de","url":"https://abxk.ai/de/ai-productivity/learn/read-llm-benchmarks-like-an-expert/"}],"url":"https://abxk.ai/en/ai-productivity/learn/read-llm-benchmarks-like-an-expert/"}