Evaluate Model Safety and Reliability

A model with excellent benchmark scores can still invent your return policy, refuse harmless questions or treat some customers differently from others. Safety and reliability are properties of your system in your context, and you only know them if you measure them. That is the job of an evaluation, or eval: a repeatable test with a fixed dataset, a scoring method and a threshold that decides pass or fail.

This is part 4 of Level 4 (Expert) in our AI Security learning path. You will learn which properties to measure and what public benchmarks can and cannot tell you. You will also see how to build an eval suite for the shop’s support assistant, how many test cases you need and how to keep evaluating after launch. Red teaming from Level 3 finds new problems; evals make sure that known problems stay fixed. Read red teaming LLMs first if you have not.

Five properties worth measuring

Most safety and reliability questions fall into five groups. Stanford’s HELM project made it popular to measure several properties side by side. It scores accuracy, calibration, robustness, fairness, bias, toxicity and efficiency together (Source: Liang et al., “Holistic Evaluation of Language Models”, 2022).

• Factuality: Are the answers correct, and are they supported by the sources the system retrieved? In a RAG system, an answer can be true in general and still not be backed by your documents.
• Robustness: Does quality hold when users rephrase, make typos, switch languages or send very long inputs? Format alone can matter: small changes in prompt formatting moved one model’s accuracy by up to 76 points in few-shot tests (Source: Sclar et al., ICLR 2024).
• Refusal behavior: There are two errors. Under-refusal means the system helps with a request it should decline. Over-refusal means it declines a harmless request that only sounds risky, such as how to “kill” a Python process. Test suites like XSTest target the second error (Source: Röttger et al., NAACL 2024).
• Bias: Does the quality or tone change when only a name, gender or origin changes? The BBQ benchmark tests this for question answering (Source: Parrish et al., 2022); for your own system, swap names in real tasks and compare.
• Calibration: Does the stated confidence match the hit rate? A well-calibrated system that says “90% sure” is right about 90% of the time. Modern neural networks tend to be overconfident (Source: Guo et al., ICML 2017). Post-training can make it worse: in the GPT-4 technical report, the pre-trained model was well calibrated on a knowledge test, but the final model was less so (Source: OpenAI, GPT-4 Technical Report, 2023).

The table turns these properties into metrics for the shop:

Calibration needs one extra step. Ask the system to state its confidence for each answer, or derive it from the model’s token probabilities. Then group the answers by confidence and compare each group with its actual hit rate. The expected calibration error (ECE) is the weighted average gap between the two. For a support assistant, the more practical check is often simpler: give it questions that its documents cannot answer, and count how often it admits that instead of guessing.

What public benchmarks can and cannot tell you

Public benchmarks are useful for one job: screening models before you test them yourself. Our LLM pages track 16 benchmarks. Level 1 used one of them, SimpleQA Verified, to show that even the best models still miss a sizable share of simple factual questions. For evals, another benchmark adds a new angle. Humanity’s Last Exam (HLE) is one of the hardest tests: 2,500 expert-written questions, published by the Center for AI Safety and Scale AI. Its authors also measure calibration. When HLE was released in early 2025, the tested models got only a small share right and stated high confidence anyway. These are the current leaders in our data:

Humanity's Last Exam: the top 5 models

#	Model	Released	Result
1	GPT 6 Astra OpenAI	Sep 3, 2026	54.8%
2	Fable 5.1 (xhigh) Anthropic	Sep 1, 2026	46.5%
3	Gemini 3.1 Pro Preview (thinking high) Google DeepMind	Feb 19, 2026	46.4%
4	Sonnet 5.5 (xhigh) Anthropic	Sep 28, 2026	46.1%
5	Gemini 3.8 Flash Google DeepMind	Sep 2, 2026	44.5%

Expert-written questions at the edge of human knowledge across many fields, built to stay hard after older tests were solved. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.

Read such tables with five limits in mind:

1. Saturation. Once the best models get close to full marks, a benchmark no longer separates them. That is why new, harder tests keep appearing.
2. Contamination. If test questions leak into training data, scores overstate what a model can do.
3. Construct validity. A benchmark measures its own task. Expert exam questions say little about your return policy.
4. Scoring incentives. Most benchmarks give no credit for “I don’t know”, so guessing pays. OpenAI researchers argued in 2025 that this rewards hallucination and that scoring should penalize confident errors more than honest uncertainty (Source: Kalai et al., “Why Language Models Hallucinate”, 2025).
5. Model is not system. Your prompt, your documents and your tools change the behavior. A benchmark tests none of them.

Use benchmarks to build a shortlist — all models and their results are in our LLM database — and then test the shortlist on your own tasks.

Build your own eval suite in layers

An eval suite is not one test but several layers that run at different times:

1. Benchmark screening narrows the choice of models. It runs once per model decision.
2. A golden set holds real tasks with reference answers. For the shop, that means real customer questions, cleaned of personal data, with answers written from the current policies. Start with one or two hundred; add every new failure.
3. Safety and security tests cover harmful and harmless-but-sensitive requests, prompt injection and data leakage cases from red teaming, the attack ideas from part 2, and name-swap pairs for bias.
4. Robustness variants repeat golden-set questions as paraphrases, with typos, in other languages and with long histories.
5. Production monitoring samples live conversations for review and tracks feedback, escalations and complaints.

Each case also needs a scoring method. Rules and exact matches work for structured outputs, such as an order status. Free text needs a comparison with the reference answer, either by people or by a second model acting as a judge. LLM-as-judge scales well, but judges have known biases: they favor the first of two answers, longer answers and sometimes answers from their own model family (Source: Zheng et al., NeurIPS 2023). Check every judge against human ratings on a sample before you trust it.

How many test cases you need

Outputs vary, so every score is an estimate. A pass rate from 50 cases is much less certain than one from 500. The figure shows the 95% confidence interval for an observed pass rate of 90% at different sample sizes.

Three rules follow. To halve the uncertainty, you need about four times as many cases. A difference between two model versions that is smaller than the intervals may be noise, not progress. And because the same input can produce different outputs, run important cases several times and report a rate. Anthropic’s guidance on error bars for evals explains the statistics in more depth (Source: Miller, “Adding Error Bars to Evals”, Anthropic, 2024).

Keep evaluating after launch

Evaluation does not end at release. Run the suite automatically whenever something changes: the model version, the system prompt, the knowledge base or a tool. Providers update the models behind an API, sometimes under the same name, so run it on a schedule as well. Turn every incident and every confirmed user complaint into a new test case. Set a threshold for each metric and block a release when one fails — a single critical safety failure should be enough.

Some platforms support parts of this work. According to the vendor, Airia offers red teaming together with policies and audit evidence. TestDriver.ai, by its own description, builds regression tests for pull requests, but for user interfaces, not for model behavior. If you build agents on a no-code platform such as MindStudio, evaluate the agent as a whole — prompts, tools and knowledge — not only its model. Before you rely on any platform for evals, ask three questions. Can it run your own golden set? Can it block a release when a threshold fails? Does it keep the results as evidence? We have not tested these tools.

For the shop’s support assistant, the plan looks like this (the sizes are examples):