Prompt injection gets the headlines, but it is only one way to attack AI. Every machine learning model — a fraud filter, a spam filter, a product recommender or a large language model — can also be attacked through its data and its math. Attackers craft inputs that the model misreads, plant errors in its training data, or send queries that copy the model or reveal what it learned. This field is called adversarial machine learning (adversarial ML).
This is part 2 of Level 4 (Expert) in our AI Security learning path. You will learn the main attack classes along the life cycle of a model, the terms NIST uses for them, and which defenses work — and where they stop working. The examples come from the online shop of this level: its fraud model, its product recommendations and its support assistant. As always in this path, we explain attacks conceptually and give no attack code.
A shared vocabulary: the NIST taxonomy
The most widely used reference is NIST AI 100-2, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations” (Source: NIST AI 100-2 E2025, March 2025, nist.gov). Part 1 of this level showed how NIST’s wider AI risk framework is built. The taxonomy sorts every attack along four questions:
- Objective: What does the attacker want? NIST names three goals: break availability (the model fails or slows down), violate integrity (the model gives the outputs the attacker wants) or compromise privacy (the attacker learns something about the training data or the model). For generative AI, NIST adds misuse: making a model do what its provider wants to prevent.
- Capability: What can the attacker control — training data, the model itself, inputs at run time, or only queries?
- Knowledge: In a white-box attack, the attacker knows the model, including its weights. In a black-box attack, they can only send inputs and see outputs. Gray-box lies in between.
- Stage: Does the attack happen during training or after deployment?
For classic predictive models, such as classifiers and scoring models, NIST groups attacks into three classes: evasion, poisoning and privacy attacks. For generative AI, it adds direct prompting attacks, such as jailbreaks, and indirect prompt injection. The figure places the main classes along the life cycle of a model.
Knowledge changes the threat a lot. An open-weight model is white-box for everyone, because anyone can download and study its weights. Our LLM database lists 172 models with open weights. That openness is useful for research and self-hosting, but it also lets attackers test their inputs offline before they touch your system.
Evasion: inputs that fool a trained model
An evasion attack changes an input at run time so that a trained model makes the wrong decision. The changed input is called an adversarial example. Around 2014, researchers showed that small, targeted pixel changes can make an image classifier mislabel a photo that looks unchanged to people (Source: Szegedy et al., “Intriguing properties of neural networks”, ICLR 2014). Later work moved the idea into the physical world: a few stickers made a road-sign classifier misread a stop sign (Source: Eykholt et al., CVPR 2018).
Why does this work? A classifier separates its inputs with a decision boundary. Many real inputs lie close to that boundary, and a model has many dimensions in which an input can move. A small step in the right direction is enough to cross it.
Evasion is not limited to images. Spam and phishing filters, malware scanners, fraud models and AI text detectors face it every day: the attacker changes the input until the filter lets it pass. For the shop, a fraud ring might place many small test orders to learn which patterns the fraud model approves — a black-box attack that needs nothing but patience. A student who rewrites an AI-generated essay with a paraphrasing tool runs an evasion attack against a detector, as our Level 2 part on how AI detectors work and fail explains.
Security products are machine learning systems too. According to their vendors, Sentaro checks emails for signs of phishing, and Bitdefender protects against phishing and ransomware. Pangram Labs, like our own ABXK AI Text Detector, estimates whether a text was written by AI. When you compare such tools, ask how they handle attackers who adapt and how often their models are updated. Apart from our own detector, we have not tested these tools.
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.
Defenses exist, but none is complete:
- Adversarial training adds adversarial examples to the training data. It makes a model more robust against the attack types used in training, but it costs compute and often some accuracy on normal inputs (Source: Madry et al., ICLR 2018).
- Certified defenses prove that no change below a set size can flip a decision — but only within that small, defined budget.
- System design helps most: combine the model score with rules and human review for high-value cases, limit the rate of queries, and watch for probing, such as many near-identical requests.
Be careful with defenses that only look strong. In 2018, researchers examined nine defenses from a major machine learning conference that year. They bypassed seven of them, six completely (Source: Athalye, Carlini and Wagner, ICML 2018). Always test a defense against an attacker who knows about it.
Poisoning: attacks during training
A poisoning attack manipulates training data or the model so that it learns the wrong thing. NIST distinguishes four types. Availability poisoning degrades the whole model. Targeted poisoning changes the result for specific inputs. Backdoor poisoning plants a hidden trigger that switches on the wrong behavior. Model poisoning manipulates the model itself, for example its weights.
Backdoors are the most dangerous, because the model behaves normally until the trigger appears. In an early demonstration, a backdoored street-sign classifier treated stop signs as speed-limit signs whenever a small sticker was present (Source: Gu et al., “BadNets”, 2017). Level 3 described the best-known recent case for language models. In a 2025 study, about 250 poisoned documents planted a backdoor in models of every tested size, from 600 million to 13 billion parameters (Source: Anthropic, UK AI Security Institute and The Alan Turing Institute, 2025, anthropic.com). For your threat model, this changes one assumption. An attacker does not need to control a share of your data, because a small, fixed number of documents can be enough. The tested backdoor was narrow, though: it made the models write gibberish, and the authors note that more complex or harmful behaviors may work differently.
The shop has several poisoning surfaces. Fake reviews and fake purchase histories can shift the recommender. Chargeback labels feed the fraud model, so whoever can trigger false labels can teach it the wrong patterns. Help articles feed the assistant’s search index. Downloaded models and datasets bring the supply chain risks from Level 3.
Level 3 named the basics of data hygiene: know where data comes from, control who can change it, and filter and deduplicate it before training. Against a determined attacker, three more controls matter:
- Limited influence: cap how much any single user, source or customer can contribute, so that a few fake accounts cannot move the model.
- Trusted test sets that attackers cannot reach, and checks for unexpected triggers before each release.
- Rollback: keep known-good snapshots of data and models, protected by hashes or signatures, so that you can retrain cleanly.
Privacy attacks: extraction, membership inference, inversion
Privacy attacks happen after deployment, and they often need nothing but ordinary queries.
- Model extraction copies a model’s behavior, or parts of the model, through its API. Researchers showed this for commercial prediction APIs in 2016 (Source: Tramèr et al., USENIX Security 2016). In 2024, a team recovered one complete layer of two production language models from OpenAI with ordinary API queries (Source: Carlini et al., “Stealing Part of a Production Language Model”, ICML 2024). A stolen copy is more than lost intellectual property: it turns a black-box attack into a white-box attack, because the attacker can now craft evasion inputs offline.
- Membership inference decides whether a specific record was part of the training data (Source: Shokri et al., IEEE S&P 2017). This matters when membership itself is sensitive — for example, when a model was trained only on patients with one disease.
- Model inversion and data extraction reconstruct training data from a model’s answers. Researchers rebuilt recognizable face images from a face recognition model (Source: Fredrikson et al., CCS 2015). Others extracted verbatim text, including names and contact details, from the training data of GPT-2 (Source: Carlini et al., USENIX Security 2021).
The main defenses: return as little as possible — a label instead of a full probability score, for example — limit and monitor queries, deduplicate training data and train with differential privacy. Part 3 of this level explains differential privacy and other privacy-preserving AI techniques.
Matching defenses to attacks
The table brings the classes together. Use it to check which classes matter for each model, and whether you have at least one control against each.
| Attack class | When | What the attacker needs | Main defenses | Known limits |
|---|---|---|---|---|
| Evasion | Deployment | Query access; more with model knowledge | Adversarial training, rules and human review, rate limits, monitoring | Adaptive attacks; robustness can cost accuracy |
| Poisoning and backdoors | Training | Write access to data, labels or the model | Provenance, access control, filtering, trusted test sets, rollback | A small number of samples can be enough |
| Model extraction | Deployment | Many queries | Rate limits, query monitoring, minimal outputs, contract terms | Slow, distributed querying is hard to spot |
| Membership inference | Deployment | Queries and a candidate record | Differential privacy, less overfitting, minimal outputs | Stronger privacy can reduce accuracy |
| Inversion and data extraction | Deployment | Queries | Deduplication, differential privacy, output filters for personal data | Rare, memorized data stays at risk |
NIST is frank about the state of the field. Most defenses are empirical: they are used because they seem to work in practice, not because they come with security proofs. New attack techniques can break them. That is why monitoring and a fast way to roll back belong in every plan.
Exercise: an attack-class review for one model
Pick one model that your organization runs or buys — for the shop, the fraud model is a good start. Work through five questions with the model’s owner:
- Objective: Which attacker goals matter — getting fraud past the model (integrity), making it block good customers (availability) or learning about customers from it (privacy)?
- Knowledge: What can an outsider learn? Can the model be downloaded, does the API return scores, are the decision rules public?
- Data: Which training data or labels can outsiders influence, and how fast does new data reach the model?
- Tests: Which robustness tests run before each release? Open-source libraries such as the Adversarial Robustness Toolbox support such tests — run them only on systems you own or are allowed to test.
- Signals: Which monitoring would reveal probing, poisoning or extraction — query volume per account, near-duplicate requests, sudden shifts in decisions?