{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Security"],"content":{"markdown":"A model with excellent benchmark scores can still invent your return policy, refuse harmless questions or treat some customers differently from others. Safety and reliability are properties of your system in your context, and you only know them if you measure them. That is the job of an **evaluation**, or **eval**: a repeatable test with a fixed dataset, a scoring method and a threshold that decides pass or fail.\n\nThis is part 4 of Level 4 (Expert) in our AI Security learning path. You will learn which properties to measure and what public benchmarks can and cannot tell you. You will also see how to build an eval suite for the shop's support assistant, how many test cases you need and how to keep evaluating after launch. Red teaming from Level 3 finds new problems; evals make sure that known problems stay fixed. Read [red teaming LLMs](/en/ai-security/learn/red-teaming-llms/) first if you have not.\n\n## Five properties worth measuring\n\nMost safety and reliability questions fall into five groups. Stanford's HELM project made it popular to measure several properties side by side. It scores accuracy, calibration, robustness, fairness, bias, toxicity and efficiency together (Source: Liang et al., \"Holistic Evaluation of Language Models\", 2022).\n\n- **Factuality:** Are the answers correct, and are they supported by the sources the system retrieved? In a RAG system, an answer can be true in general and still not be backed by your documents.\n- **Robustness:** Does quality hold when users rephrase, make typos, switch languages or send very long inputs? Format alone can matter: small changes in prompt formatting moved one model's accuracy by up to 76 points in few-shot tests (Source: Sclar et al., ICLR 2024).\n- **Refusal behavior:** There are two errors. **Under-refusal** means the system helps with a request it should decline. **Over-refusal** means it declines a harmless request that only sounds risky, such as how to \"kill\" a Python process. Test suites like XSTest target the second error (Source: Röttger et al., NAACL 2024).\n- **Bias:** Does the quality or tone change when only a name, gender or origin changes? The BBQ benchmark tests this for question answering (Source: Parrish et al., 2022); for your own system, swap names in real tasks and compare.\n- **Calibration:** Does the stated confidence match the hit rate? A well-calibrated system that says \"90% sure\" is right about 90% of the time. Modern neural networks tend to be overconfident (Source: Guo et al., ICML 2017). Post-training can make it worse: in the GPT-4 technical report, the pre-trained model was well calibrated on a knowledge test, but the final model was less so (Source: OpenAI, GPT-4 Technical Report, 2023).\n\nThe table turns these properties into metrics for the shop:\n\n| Property | Example metric | Shop example |\n|---|---|---|\n| Factuality | Share of answers that match the reference; share of claims not backed by a source | Return periods and shipping fees must match the policy |\n| Robustness | Lowest score across paraphrases, typos and languages | The same question in German, with typos, gets the same answer |\n| Refusal behavior | Harmful requests answered; harmless requests refused | Declines to reveal other customers' data, but explains order tracking |\n| Bias | Difference in outcome or tone between swapped names | Refund decisions do not change with the customer's name |\n| Calibration | Expected calibration error; share of \"I don't know\" on unanswerable questions | Says \"I'll check with a colleague\" instead of guessing |\n\nCalibration needs one extra step. Ask the system to state its confidence for each answer, or derive it from the model's token probabilities. Then group the answers by confidence and compare each group with its actual hit rate. The **expected calibration error** (ECE) is the weighted average gap between the two. For a support assistant, the more practical check is often simpler: give it questions that its documents cannot answer, and count how often it admits that instead of guessing.\n\n## What public benchmarks can and cannot tell you\n\nPublic benchmarks are useful for one job: screening models before you test them yourself. Our LLM pages track {{\u003c stat \"benchmarks\" \u003e}} benchmarks. [Level 1](/en/ai-security/learn/hallucinations-as-a-security-risk/) used one of them, SimpleQA Verified, to show that even the best models still miss a sizable share of simple factual questions. For evals, another benchmark adds a new angle. **Humanity's Last Exam** (HLE) is one of the hardest tests: 2,500 expert-written questions, published by the Center for AI Safety and Scale AI. Its authors also measure calibration. When HLE was released in early 2025, the tested models got only a small share right and stated high confidence anyway. These are the current leaders in our data:\n\n{{\u003c llm-top bench=\"hle\" n=\"5\" \u003e}}\n\nRead such tables with five limits in mind:\n\n1. **Saturation.** Once the best models get close to full marks, a benchmark no longer separates them. That is why new, harder tests keep appearing.\n2. **Contamination.** If test questions leak into training data, scores overstate what a model can do.\n3. **Construct validity.** A benchmark measures its own task. Expert exam questions say little about your return policy.\n4. **Scoring incentives.** Most benchmarks give no credit for \"I don't know\", so guessing pays. OpenAI researchers argued in 2025 that this rewards hallucination and that scoring should penalize confident errors more than honest uncertainty (Source: Kalai et al., \"Why Language Models Hallucinate\", 2025).\n5. **Model is not system.** Your prompt, your documents and your tools change the behavior. A benchmark tests none of them.\n\n![Line chart of the best results over time on HellaSwag, GPQA Diamond and Humanity's Last Exam: the older benchmarks climb close to full marks, the newest one starts far lower](/images/learn/en/evaluate-model-safety-and-reliability-saturation.webp \"Benchmarks saturate: older tests approach full marks, so newer, harder ones replace them. Data: Epoch AI, CC BY 4.0.\")\n\nUse benchmarks to build a shortlist — all models and their results are in our [LLM database](/en/llm/) — and then test the shortlist on your own tasks.\n\n## Build your own eval suite in layers\n\nAn eval suite is not one test but several layers that run at different times:\n\n![Five layers of an eval suite from benchmark screening at the top through a golden set, safety and security tests and robustness variants down to production monitoring, each with when it runs](/images/learn/en/evaluate-model-safety-and-reliability-layers.webp \"An eval suite in five layers. The lower layers run more often and catch more of your real problems.\")\n\n1. **Benchmark screening** narrows the choice of models. It runs once per model decision.\n2. A **golden set** holds real tasks with reference answers. For the shop, that means real customer questions, cleaned of personal data, with answers written from the current policies. Start with one or two hundred; add every new failure.\n3. **Safety and security tests** cover harmful and harmless-but-sensitive requests, prompt injection and data leakage cases from red teaming, the attack ideas from [part 2](/en/ai-security/learn/adversarial-machine-learning/), and name-swap pairs for bias.\n4. **Robustness variants** repeat golden-set questions as paraphrases, with typos, in other languages and with long histories.\n5. **Production monitoring** samples live conversations for review and tracks feedback, escalations and complaints.\n\nEach case also needs a **scoring method**. Rules and exact matches work for structured outputs, such as an order status. Free text needs a comparison with the reference answer, either by people or by a second model acting as a judge. **LLM-as-judge** scales well, but judges have known biases: they favor the first of two answers, longer answers and sometimes answers from their own model family (Source: Zheng et al., NeurIPS 2023). Check every judge against human ratings on a sample before you trust it.\n\n## How many test cases you need\n\nOutputs vary, so every score is an estimate. A pass rate from 50 cases is much less certain than one from 500. The figure shows the 95% confidence interval for an observed pass rate of 90% at different sample sizes.\n\n![Chart of 95 percent confidence intervals around an observed pass rate of 90 percent: the interval is wide for 25 or 50 test cases and narrows steadily for 200, 500 and 1,000 cases](/images/learn/en/evaluate-model-safety-and-reliability-sample-size.webp \"The same observed pass rate means more with more test cases. Computed with the Wilson interval — an example, not a measurement.\")\n\nThree rules follow. To halve the uncertainty, you need about four times as many cases. A difference between two model versions that is smaller than the intervals may be noise, not progress. And because the same input can produce different outputs, run important cases several times and report a rate. Anthropic's guidance on error bars for evals explains the statistics in more depth (Source: Miller, \"Adding Error Bars to Evals\", Anthropic, 2024).\n\n## Keep evaluating after launch\n\nEvaluation does not end at release. Run the suite automatically whenever something changes: the model version, the system prompt, the knowledge base or a tool. Providers update the models behind an API, sometimes under the same name, so run it on a schedule as well. Turn every incident and every confirmed user complaint into a new test case. Set a threshold for each metric and block a release when one fails — a single critical safety failure should be enough.\n\nSome platforms support parts of this work. According to the vendor, Airia offers red teaming together with policies and audit evidence. TestDriver.ai, by its own description, builds regression tests for pull requests, but for user interfaces, not for model behavior. If you build agents on a no-code platform such as MindStudio, evaluate the agent as a whole — prompts, tools and knowledge — not only its model. Before you rely on any platform for evals, ask three questions. Can it run your own golden set? Can it block a release when a threshold fails? Does it keep the results as evidence? We have not tested these tools.\n\nFor the shop's support assistant, the plan looks like this (the sizes are examples):\n\n| Test set | Example size | Metric | Release gate | Runs |\n|---|---|---|---|---|\n| Golden set of real questions | 200 cases | Agreement with reference answers | No drop against the last release | On every change |\n| Policy facts | 50 cases | Exact match for periods and fees | All pass | On every change |\n| Harmful and sensitive requests | 100 cases | Harmful answers; harmless requests refused | No critical failure | On every change |\n| Injection and leakage | 60 cases from red teaming | Hidden instructions followed; data leaked | None | On every change |\n| Name-swap pairs | 40 pairs | Difference in outcome and tone | No systematic difference | Monthly |\n| Live sample | 1% of chats | Reviewer rating, escalations | Trend, no gate | Weekly |\n\n{{\u003c callout type=\"try\" title=\"Your first ten golden cases\" \u003e}}\nCollect ten real questions that your AI system received last week. Remove personal data. Write the correct answer for each from your current documents, and mark which source proves it. Run the system three times per question and score the answers. Every miss is your first regression test — and the start of your evidence for [part 5](/en/ai-security/learn/build-a-secure-ai-program/).\n{{\u003c /callout \u003e}}","text":"A model with excellent benchmark scores can still invent your return policy, refuse harmless questions or treat some customers differently from others. Safety and reliability are properties of your system in your context, and you only know them if you measure them. That is the job of an evaluation, or eval: a repeatable test with a fixed dataset, a scoring method and a threshold that decides pass or fail.\nThis is part 4 of Level 4 (Expert) in our AI Security learning path. You will learn which properties to measure and what public benchmarks can and cannot tell you. You will also see how to build an eval suite for the shop\u0026rsquo;s support assistant, how many test cases you need and how to keep evaluating after launch. Red teaming from Level 3 finds new problems; evals make sure that known problems stay fixed. Read red teaming LLMs first if you have not.\nFive properties worth measuring Most safety and reliability questions fall into five groups. Stanford\u0026rsquo;s HELM project made it popular to measure several properties side by side. It scores accuracy, calibration, robustness, fairness, bias, toxicity and efficiency together (Source: Liang et al., \u0026ldquo;Holistic Evaluation of Language Models\u0026rdquo;, 2022).\nFactuality: Are the answers correct, and are they supported by the sources the system retrieved? In a RAG system, an answer can be true in general and still not be backed by your documents. Robustness: Does quality hold when users rephrase, make typos, switch languages or send very long inputs? Format alone can matter: small changes in prompt formatting moved one model\u0026rsquo;s accuracy by up to 76 points in few-shot tests (Source: Sclar et al., ICLR 2024). Refusal behavior: There are two errors. Under-refusal means the system helps with a request it should decline. Over-refusal means it declines a harmless request that only sounds risky, such as how to \u0026ldquo;kill\u0026rdquo; a Python process. Test suites like XSTest target the second error (Source: Röttger et al., NAACL 2024). Bias: Does the quality or tone change when only a name, gender or origin changes? The BBQ benchmark tests this for question answering (Source: Parrish et al., 2022); for your own system, swap names in real tasks and compare. Calibration: Does the stated confidence match the hit rate? A well-calibrated system that says \u0026ldquo;90% sure\u0026rdquo; is right about 90% of the time. Modern neural networks tend to be overconfident (Source: Guo et al., ICML 2017). Post-training can make it worse: in the GPT-4 technical report, the pre-trained model was well calibrated on a knowledge test, but the final model was less so (Source: OpenAI, GPT-4 Technical Report, 2023). The table turns these properties into metrics for the shop:\nEvaluate Model Safety and Reliability Property Example metric Shop example Factuality Share of answers that match the reference; share of claims not backed by a source Return periods and shipping fees must match the policy Robustness Lowest score across paraphrases, typos and languages The same question in German, with typos, gets the same answer Refusal behavior Harmful requests answered; harmless requests refused Declines to reveal other customers\u0026rsquo; data, but explains order tracking Bias Difference in outcome or tone between swapped names Refund decisions do not change with the customer\u0026rsquo;s name Calibration Expected calibration error; share of \u0026ldquo;I don\u0026rsquo;t know\u0026rdquo; on unanswerable questions Says \u0026ldquo;I\u0026rsquo;ll check with a colleague\u0026rdquo; instead of guessing Calibration needs one extra step. Ask the system to state its confidence for each answer, or derive it from the model\u0026rsquo;s token probabilities. Then group the answers by confidence and compare each group with its actual hit rate. The expected calibration error (ECE) is the weighted average gap between the two. For a support assistant, the more practical check is often simpler: give it questions that its documents cannot answer, and count how often it admits that instead of guessing.\nWhat public benchmarks can and cannot tell you Public benchmarks are useful for one job: screening models before you test them yourself. Our LLM pages track 16 benchmarks. Level 1 used one of them, SimpleQA Verified, to show that even the best models still miss a sizable share of simple factual questions. For evals, another benchmark adds a new angle. Humanity\u0026rsquo;s Last Exam (HLE) is one of the hardest tests: 2,500 expert-written questions, published by the Center for AI Safety and Scale AI. Its authors also measure calibration. When HLE was released in early 2025, the tested models got only a small share right and stated high confidence anyway. These are the current leaders in our data:\nHumanity\u0026#39;s Last Exam: the top 5 models #ModelReleasedResult 1GPT 6 Astra OpenAISep 3, 202654.8% 2Fable 5.1 (xhigh) AnthropicSep 1, 202646.5% 3Gemini 3.1 Pro Preview (thinking high) Google DeepMindFeb 19, 202646.4% 4Sonnet 5.5 (xhigh) AnthropicSep 28, 202646.1% 5Gemini 3.8 Flash Google DeepMindSep 2, 202644.5% Expert-written questions at the edge of human knowledge across many fields, built to stay hard after older tests were solved. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.\nRead such tables with five limits in mind:\nSaturation. Once the best models get close to full marks, a benchmark no longer separates them. That is why new, harder tests keep appearing. Contamination. If test questions leak into training data, scores overstate what a model can do. Construct validity. A benchmark measures its own task. Expert exam questions say little about your return policy. Scoring incentives. Most benchmarks give no credit for \u0026ldquo;I don\u0026rsquo;t know\u0026rdquo;, so guessing pays. OpenAI researchers argued in 2025 that this rewards hallucination and that scoring should penalize confident errors more than honest uncertainty (Source: Kalai et al., \u0026ldquo;Why Language Models Hallucinate\u0026rdquo;, 2025). Model is not system. Your prompt, your documents and your tools change the behavior. A benchmark tests none of them. Benchmarks saturate: older tests approach full marks, so newer, harder ones replace them. Data: Epoch AI, CC BY 4.0. Use benchmarks to build a shortlist — all models and their results are in our LLM database — and then test the shortlist on your own tasks.\nBuild your own eval suite in layers An eval suite is not one test but several layers that run at different times:\nAn eval suite in five layers. The lower layers run more often and catch more of your real problems. Benchmark screening narrows the choice of models. It runs once per model decision. A golden set holds real tasks with reference answers. For the shop, that means real customer questions, cleaned of personal data, with answers written from the current policies. Start with one or two hundred; add every new failure. Safety and security tests cover harmful and harmless-but-sensitive requests, prompt injection and data leakage cases from red teaming, the attack ideas from part 2, and name-swap pairs for bias. Robustness variants repeat golden-set questions as paraphrases, with typos, in other languages and with long histories. Production monitoring samples live conversations for review and tracks feedback, escalations and complaints. Each case also needs a scoring method. Rules and exact matches work for structured outputs, such as an order status. Free text needs a comparison with the reference answer, either by people or by a second model acting as a judge. LLM-as-judge scales well, but judges have known biases: they favor the first of two answers, longer answers and sometimes answers from their own model family (Source: Zheng et al., NeurIPS 2023). Check every judge against human ratings on a sample before you trust it.\nHow many test cases you need Outputs vary, so every score is an estimate. A pass rate from 50 cases is much less certain than one from 500. The figure shows the 95% confidence interval for an observed pass rate of 90% at different sample sizes.\nThe same observed pass rate means more with more test cases. Computed with the Wilson interval — an example, not a measurement. Three rules follow. To halve the uncertainty, you need about four times as many cases. A difference between two model versions that is smaller than the intervals may be noise, not progress. And because the same input can produce different outputs, run important cases several times and report a rate. Anthropic\u0026rsquo;s guidance on error bars for evals explains the statistics in more depth (Source: Miller, \u0026ldquo;Adding Error Bars to Evals\u0026rdquo;, Anthropic, 2024).\nKeep evaluating after launch Evaluation does not end at release. Run the suite automatically whenever something changes: the model version, the system prompt, the knowledge base or a tool. Providers update the models behind an API, sometimes under the same name, so run it on a schedule as well. Turn every incident and every confirmed user complaint into a new test case. Set a threshold for each metric and block a release when one fails — a single critical safety failure should be enough.\nSome platforms support parts of this work. According to the vendor, Airia offers red teaming together with policies and audit evidence. TestDriver.ai, by its own description, builds regression tests for pull requests, but for user interfaces, not for model behavior. If you build agents on a no-code platform such as MindStudio, evaluate the agent as a whole — prompts, tools and knowledge — not only its model. Before you rely on any platform for evals, ask three questions. Can it run your own golden set? Can it block a release when a threshold fails? Does it keep the results as evidence? We have not tested these tools.\nFor the shop\u0026rsquo;s support assistant, the plan looks like this (the sizes are examples):\nEvaluate Model Safety and Reliability (2) Test set Example size Metric Release gate Runs Golden set of real questions 200 cases Agreement with reference answers No drop against the last release On every change Policy facts 50 cases Exact match for periods and fees All pass On every change Harmful and sensitive requests 100 cases Harmful answers; harmless requests refused No critical failure On every change Injection and leakage 60 cases from red teaming Hidden instructions followed; data leaked None On every change Name-swap pairs 40 pairs Difference in outcome and tone No systematic difference Monthly Live sample 1% of chats Reviewer rating, escalations Trend, no gate Weekly ▸Your first ten golden cases\nCollect ten real questions that your AI system received last week. Remove personal data. Write the correct answer for each from your current documents, and mark which source proves it. Run the system three times per question and score the answers. Every miss is your first regression test — and the start of your evidence for part 5.\n","words":1755},"date":"2026-10-14","description":"How to evaluate AI for factuality, robustness, refusals, bias and calibration, what public benchmarks cannot show, and how to build and run your own eval suite.","image":"https://abxk.ai/images/cover/evaluate-model-safety-and-reliability.webp","json":"https://abxk.ai/en/ai-security/learn/evaluate-model-safety-and-reliability/index.json","kind":"page","language":"en","learning_path":{"category":"ai-security","level":4,"level_name":"Expert"},"modified":"2026-10-14","related_tools":["Airia","TestDriver.ai","MindStudio"],"section":"ai-security","tags":["ai-evaluation","llm-evals","calibration","benchmarks","expert"],"title":"Evaluate Model Safety and Reliability","translations":[{"json":"https://abxk.ai/de/ai-security/learn/evaluate-model-safety-and-reliability/index.json","language":"de","url":"https://abxk.ai/de/ai-security/learn/evaluate-model-safety-and-reliability/"}],"url":"https://abxk.ai/en/ai-security/learn/evaluate-model-safety-and-reliability/"}