Every public AI chatbot meets people who try to make it say or do what it should not. Some want harmful instructions, some want a discount the shop never offered, and some just want to see the system break. When they succeed, the result is called a jailbreak. The controls that stop them are called guardrails.
This is part 3 of Level 2 in our AI Security learning path. Part 1 listed jailbreaking as a form of prompt injection, the first risk on the OWASP list. Here you will learn why jailbreaks work, which kinds of guardrails exist, how to combine them in layers and how to test them. Attacks are explained only as principles: this guide contains no working attack prompts.
What a jailbreak is
A jailbreak is an input written to make a language model ignore its safety rules or the rules of the app it runs in. OWASP counts it as a form of prompt injection: the attacker’s words override the instructions the model should follow (Source: OWASP Top 10 for LLM Applications 2026, LLM01, genai.owasp.org).
Jailbreaks aim at two kinds of rules:
- The model maker’s safety rules. Providers train their models to refuse dangerous requests, such as help with weapons or malware. A jailbreak tries to get around this refusal.
- Your app’s business rules. A support assistant should talk only about your products, should not promise refunds above a limit and should not reveal internal notes. A jailbreak tries to make it break these rules.
For a company, the second kind is often the bigger everyday risk. Imagine a shop chatbot that a user talks into “confirming” a discount of 90 percent, and the screenshot spreads online. Whether such a promise is binding is a question for lawyers. The damage to trust happens at once.
| Term | Meaning |
|---|---|
| Jailbreak | Input that makes a model ignore its safety rules or the app’s rules |
| Prompt injection | Input that makes a model follow an attacker’s instructions instead of yours; jailbreaking is one form |
| Guardrail | A control that keeps input, output or actions within set limits |
| Refusal | The model declines a request |
| Over-refusal | The model or a filter blocks a legitimate request |
Why jailbreaks work
Safety training is learned behavior, not a hard-coded rule. Model makers show their models many examples of requests to refuse and reward the right reactions. The model learns a pattern, but no training set covers every possible wording, language and context.
Researchers describe two root causes (Source: Wei et al., “Jailbroken: How Does LLM Safety Training Fail?”, 2023). The first is competing objectives: the model is trained to be helpful and to be safe, and a clever request turns one goal against the other. The second is mismatched generalization: the model’s abilities reach further than its safety training. A model may understand a request in an encoding or a rare language, while its safety training covered mostly plain wording.
Most known jailbreaks belong to a few families. To defend against them, you do not need the exact tricks. You need to know which property of the model they use:
- Role-play and fiction wrap a request in a story or a persona, so that refusing seems to break the game.
- Obfuscation hides a request in an encoding, another language or misspellings to get past simple filters.
- Many-shot attacks fill a long context window with made-up dialogues in which an assistant breaks its rules, and the model continues the pattern. Anthropic described this in 2024 and pointed out that the growing context windows made it possible.
- Multi-turn escalation starts with harmless questions and moves toward the goal step by step, building on the model’s own earlier answers. Microsoft researchers described such an attack in 2024 under the name Crescendo.
- Automated attacks let an algorithm search for strings of characters that break the safety training. In 2023, researchers showed that such strings, found on open models, also worked against closed chatbots (Source: Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models”, 2023).
- Indirect attacks hide the instructions in documents, emails or web pages that the model reads, as Level 1 shows in prompt injection explained.
Five types of guardrails
Guardrails come in five types. They differ in what they catch, what they miss and what they cost:
| Guardrail | What it catches | What it misses | Side effects |
|---|---|---|---|
| Safety training of the model | Obvious harmful requests | New wordings, encodings, many-shot and multi-turn attacks | You cannot see or change it, and it differs between models and versions |
| Rules in the system prompt | Off-topic requests, tone, simple misuse | Users who argue long enough; OWASP says the system prompt is not a security control | A false sense of safety |
| Input filters | Known attack patterns, forbidden topics, oversized inputs | New attacks, attacks spread over several turns or hidden in documents | Blocks some legitimate requests |
| Output filters | Harmful content, personal data, promises outside your policy, wrong formats | Subtle errors and harm that only shows in an action | More delay; good answers are sometimes cut |
| System design | The damage after a successful jailbreak, through limits in code, least privilege and approvals | The jailbreak itself, which still happens in the model | Less autonomy for the AI |
Input and output filters are often classifiers: smaller models trained to judge whether a text breaks a policy. Meta, for example, published Llama Guard, an open model that classifies prompts and responses, and NVIDIA offers NeMo Guardrails, an open-source toolkit for adding programmable rules around an LLM app. Model makers use the same idea on their own services: in 2025, Anthropic described how separate input and output classifiers protect its models against universal jailbreaks.
Open models need extra care. Our LLM database lists 172 models with open weights. If you run such a model yourself, every guardrail around it is your job. Check which version you deploy, because fine-tuned variants on public model hubs sometimes have their safety training removed on purpose. And if you fine-tune a model yourself, test its safety again: research has shown that fine-tuning can weaken safety training, even with harmless training data (Source: Qi et al., “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!”, ICLR 2024).
Combine guardrails in layers
No guardrail stops every jailbreak. The answer is the same as elsewhere in security: several layers, each with different weak spots. Safety engineers call this the Swiss cheese model: every slice has holes, but the holes rarely line up.
Back to the shop assistant. A user slowly talks the model into “granting” a refund far above the limit. The input filter misses the friendly, gradual escalation. The model’s safety training sees no problem, because refunds are a normal topic. But the refund tool checks the limit in code and sends larger amounts to a person. The output filter catches the promise in the chat text. The logs keep the conversation, so the team can improve the filter. The jailbreak worked on the model, and it failed on the system.
The most important layer is the one that does not depend on the model at all: rules enforced in code, least privilege and human approval for actions that cost money or cannot be undone. Companies that run many AI agents sometimes manage such policies on a central platform. According to the vendor, Airia offers runtime policies, red teaming and audit evidence for agents, models and tools.
Test your guardrails
Guardrails need tests like any other feature. A good test set has three parts:
- Requests that must be refused, sorted by the risks that matter for your app: harmful content, promises outside your policy, requests for other people’s data and off-topic use.
- Variants of these requests that follow the jailbreak families: role-play, several turns, other languages. Public collections such as HarmBench and JailbreakBench provide test cases, so your team does not have to invent attacks.
- Harmless requests that look similar, such as a customer who asks how refunds work. They show whether your guardrails block legitimate use.
Then measure two numbers. The attack success rate is the share of attack attempts that get through. The false refusal rate is the share of legitimate requests that get blocked. A stricter filter lowers the first number and raises the second. Where you set the balance depends on the possible harm: a learning app for children needs stricter settings than an internal tool for security analysts.
A fixed test set has a blind spot: real attackers adapt. In a 2025 study, attacks that adapted to the defense got past twelve recent defenses, most of them more than 90 percent of the time. Most of these defenses had reported almost no successful attacks in their own tests (Source: Nasr et al., “The Attacker Moves Second”, 2025). So let testers who know your guardrails try to get around them, and read a good score on a public test set as a starting point, not as proof.
Run the test set before launch and again after every change: a new model version, a new system prompt, a new tool. A model update can change behavior that your filters relied on. Level 3 shows how teams test AI systems in depth with red teaming. If you find a jailbreak that works against a provider’s model, report it through the provider’s security or bug bounty program, and do not publish the working prompt.
If you run a customer chatbot on a platform, ask the vendor which guardrails it includes, and test them with your own test set. Our comparison lists several such platforms in AI Work:
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.
Common mistakes with guardrails
| Mistake | Why it fails | Better |
|---|---|---|
| Rules only in the system prompt | Users can argue the model out of them | Enforce rules in code; prompt rules are only the first layer |
| Keyword blocklists | Synonyms, typos and other languages slip through, harmless texts get blocked | Classifiers, plus limits on what the system can do |
| Checking only the input | Multi-turn and indirect attacks pass the input check | Check outputs and actions too |
| Testing once before launch | New model versions change behavior | Re-run the test set after every change |
| Counting only blocked attacks | Over-refusal frustrates users and drives them to other tools | Measure false refusals as well |
| Relying on the provider’s safety training | It knows nothing about your business rules | Add your own guardrails for your own rules |