An AI assistant that invents a fact is annoying when you write an email. It becomes a security problem when someone acts on it: installs software that does not exist, applies a fix that switches off a protection or relies on a court decision that was never made. Attackers have even learned to profit from these mistakes.
This is part 3 of Level 1 in our AI Security learning path. The safety checklist in Level 0 told you to check AI answers before you act on them; this part shows what happens when nobody does. You will learn when a hallucination turns into a security risk and how attackers exploit invented package names. You will also see why AI advice on security settings needs a second look, what benchmarks show about factual accuracy and a short routine for checking AI output.
When a made-up answer becomes a risk
A hallucination is an answer that sounds right but is made up: a fact, a source, a number or a command. A language model predicts likely words. It has no reliable sense of which facts it knows well. One reason: training and common tests reward a confident guess more than an honest “I don’t know” (Source: Kalai et al., “Why Language Models Hallucinate”, 2025, arxiv.org).
OWASP counts this among the top ten risks for AI applications under the name misinformation. It also names a related problem, overreliance: people trust AI output without checking it (Source: OWASP Top 10 for LLM Applications 2025, LLM09, genai.owasp.org). In the current edition from 2026, misinformation moved up to seventh place.
A hallucination alone does no harm. It turns into an incident through a chain of four links: the model invents something, it sounds certain, nobody checks, and someone acts on it. Break one link and the chain fails. The link you control best is the check.
Invented package names: slopsquatting
Programmers rarely write everything themselves. They use packages: ready-made code libraries that they download from public registries such as PyPI for Python or npm for JavaScript. AI coding assistants often suggest packages, and sometimes they suggest ones that do not exist.
A study presented at the USENIX Security Symposium in 2025 tested 16 code-generating models with 576,000 code samples. On average, at least 5.2 percent of the packages suggested by commercial models did not exist. For open-source models, the share was 21.7 percent. In total, the researchers counted 205,474 different invented package names (Source: Spracklen et al., USENIX Security 2025, arxiv.org).
This gives attackers an opening. If models invent the same name again and again, an attacker can register it and fill it with malicious code. The next person who trusts the AI installs the attacker’s package. This attack is called slopsquatting, a mix of “AI slop” and typosquatting, the old trick of registering misspelled names of popular packages. The 2025 edition of the OWASP list describes exactly this scenario as an example of the misinformation risk.
How realistic is this? A security researcher at Lasso Security uploaded an empty test package under a name that AI models kept inventing. Within three months, it was downloaded more than 30,000 times (Source: Lasso Security, 2024). That package was harmless. A real attacker’s package would not be.
Fixes that switch off a protection
Made-up or careless answers also hide in advice, and the same OWASP entry counts unsafe code suggestions as part of this risk. Ask an AI assistant to fix an error, and it may suggest a quick solution that removes a protection: switch off certificate checks, open a firewall port to everyone, give an account full admin rights or store a password in plain text. The error disappears. The security hole stays.
Research shows how easily this slips through. In a study at Stanford University, participants who used an AI assistant wrote significantly less secure code than those without one. They were also more likely to believe that their code was secure (Source: Perry et al., ACM CCS 2023, arxiv.org). Treat any AI advice that weakens a protection as a red flag, and check it against the product’s official documentation or with your security team.
Invented legal and compliance facts
The same pattern hits legal questions. In 2023, a US federal court sanctioned two lawyers and their law firm because a court filing cited decisions that ChatGPT had made up (Source: Mata v. Avianca, Inc., U.S. District Court for the Southern District of New York, 2023). In security and compliance work, the risk looks similar: an invented article of a law, a wrong reporting deadline after a data breach or a made-up requirement of a standard. A confident wrong answer can lead to a decision that breaks a rule or misses a deadline.
The table shows how the four risk types show up and how to check them:
| Risk | How it shows up | How to check |
|---|---|---|
| Invented package | An install command for a library you have never heard of | Look it up in the registry: age, maintainer, downloads, link to the source code; compare with the official docs |
| Weakening fix | “Turn off verification”, “allow all”, “run as admin” | Find the cause in the official documentation; never switch off a protection to make an error go away |
| Invented legal fact | A court case, article of law or deadline you cannot find | Read the primary source, such as the law text or a court database, or ask your legal team |
| Invented reference | A security bulletin, version number or clause of a standard that seems to exist | Search the publisher’s own website; no hit means no fact |
What benchmarks show about factual accuracy
How often do models get facts wrong? Researchers measure this with benchmarks: standard tests that every model takes in the same way. SimpleQA Verified asks short factual questions with exactly one correct answer, and the models answer without a web search. These are the current leaders in our LLM data:
| # | Model | Released | Result |
|---|---|---|---|
| 1 | Gemini 4 Argon (high) Google DeepMind | Sep 30, 2026 | 76.0% |
| 2 | GPT-6 Astra (max) OpenAI | Sep 3, 2026 | 75.6% |
| 3 | GPT-6.1 Sol (max) OpenAI | Sep 29, 2026 | 73.9% |
| 4 | Gemini 3.1 Pro Preview Google DeepMind | Feb 19, 2026 | 73.5% |
| 5 | Claude Opus 5.5 (max) Anthropic | Sep 22, 2026 | 72.2% |
Short factual questions with exactly one correct answer, answered without tools. It shows how much a model really knows and how often it invents an answer instead of admitting that it does not know. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.
Even the leaders miss a sizable share of the questions. The chart shows every model in our database that took this test, and many of them score far below the leaders:
Two lessons follow for security work. First, the model matters: for knowledge-heavy tasks, prefer models that do well on factual tests, and compare them in our LLM database. Second, no score makes the check unnecessary. A model that is usually right is still sometimes wrong, and it sounds just as certain when it is. Level 2 places misinformation in the OWASP Top 10, and Level 3 shows how to secure the whole AI supply chain, from models to packages.
Check before you act
Use this short routine whenever an AI answer leads to an action with security impact:
- Name the action. What exactly would you do based on the answer: install, configure, sign or send?
- Find the primary source. The official documentation, the package registry, the law text or the vendor’s security bulletin.
- Check the details. Exact name, version, number and date. One wrong letter in a package name is enough.
- Prefer the safe default. If the advice weakens a protection, look for another way or ask your security team.
- Write down what you checked. Others can then follow your decision.
The routine matters most when the assistant feels reliable. In a 2025 survey of 319 knowledge workers by Microsoft Research and Carnegie Mellon University, people with more confidence in generative AI reported less critical thinking, while people with more confidence in their own skills reported more (Source: Lee et al., CHI 2025). A fixed habit like step 5 keeps the check from fading as trust grows.
Tools can speed up some of these steps. According to the vendors, Consensus answers with citations from research papers, Nouswise answers from the documents you upload, and TestDriver.ai runs tests on code changes before they are merged. None of them makes your own check unnecessary.
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.