Part 2 of this level showed that a trained model can give away its training data through membership inference, inversion or verbatim extraction. Removing names from a prompt — the basics from our guide on protecting personal data with AI tools — does not help against that. You need techniques that limit what a model can learn about any single person, or that keep data hidden even while it is processed. Together they are called privacy-enhancing technologies (PETs).
This is part 3 of Level 4 (Expert) in our AI Security learning path. You will learn what epsilon and a privacy budget mean, and what federated learning protects and what it does not. You will also see where secure enclaves and homomorphic encryption fit, and why synthetic data is not automatically anonymous. The online shop of this level wants to train a better fraud model on customer data and to work with partner shops. At the end, you will choose a technique for each of its goals.
Why removing names is not enough
Data without names can still point to one person. A classic study estimated that 87% of the US population could be uniquely identified by ZIP code, gender and date of birth alone (Source: Sweeney, “Simple Demographics Often Identify People Uniquely”, Carnegie Mellon University, 2000). In 2008, researchers re-identified people in an anonymized movie-rating dataset that Netflix had published, by comparing it with public reviews (Source: Narayanan and Shmatikov, IEEE S&P 2008).
The law reflects this. As Level 2 showed, pseudonymized data — names replaced by codes — remains personal data under the GDPR for anyone who can link the codes back to people. Only truly anonymous data falls outside the regulation. The European Data Protection Board applies the same logic to AI models. A model trained on personal data counts as anonymous only if the chance of extracting personal data from it, directly or through queries, is insignificant. This has to be shown case by case (Source: EDPB Opinion 28/2024, December 2024, edpb.europa.eu).
So the question changes. It is no longer “Did we remove the identifiers?” but “What can anyone learn about one person from our data, our model or its outputs?” Differential privacy answers that question with a number.
Differential privacy: a limit on what one person can change
A process is differentially private if its results look almost the same whether or not any single person’s data is included (Source: Dwork, McSherry, Nissim and Smith, 2006). The parameter epsilon (ε) defines “almost”: for every possible result, the probability with one person’s data may be at most e^ε times the probability without it. Smaller ε means stronger privacy.
The oldest example shows the idea. In randomized response, a survey method from the 1960s, each person flips a coin before answering a sensitive question:
With fair coins, a person whose true answer is “yes” says “yes” with a probability of 3/4. A person whose true answer is “no” says “yes” with a probability of 1/4. The ratio is 3, so ε = ln 3, about 1.1. Every person can deny their answer, but the noise is known, so the share of true “yes” answers in a large group can still be estimated.
Three practical rules follow:
- e^ε grows fast. At ε = 1, one person can change the odds of a result by a factor of about 2.7. At ε = 10, the factor is over 20,000, so the formal guarantee is weak.
- Every use spends budget. Each query or training run uses part of the privacy budget. In the simplest accounting, budgets add up: two queries at ε = 0.5 each cost ε = 1 in total. When the budget is spent, you stop or accept a weaker guarantee.
- The unit matters. Does ε protect one record, or one person with many records? NIST explains how to evaluate such claims, including the unit of privacy and common pitfalls in implementations (Source: NIST SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees, March 2025, nist.gov).
For machine learning, DP-SGD limits the influence of each training example and adds noise during training, so that the finished model is differentially private (Source: Abadi et al., CCS 2016). The price is accuracy, and it is not paid evenly: rare groups and rare patterns lose the most (Source: Bagdasaryan, Poursaeed and Shmatikov, NeurIPS 2019). Large datasets soften the cost. Differential privacy is in real use: the US Census Bureau applied it to the 2020 Census results, and Apple and Google use it to collect usage statistics from devices.
Federated learning: train without collecting the data
Federated learning sends the model to the data instead of the data to the model. Devices or organizations train a copy locally and send back only model updates; a server combines them into a new model (Source: McMahan et al., AISTATS 2017). Google used this approach to improve next-word prediction in its Gboard keyboard (Source: Hard et al., 2018).
Federated learning keeps raw data out of a central store. That helps when data cannot leave a device, a hospital or a country. It does not make the result private by itself:
- Updates leak. Researchers reconstructed training images and text from shared gradients (Source: Zhu, Liu and Han, “Deep Leakage from Gradients”, NeurIPS 2019).
- Participants can poison the shared model, as described in part 2.
- Data differs between participants. Training slows down, and the shared model may fit some participants worse than others.
Two add-ons close part of the gap. Secure aggregation lets the server see only the sum of many updates, not each single one (Source: Bonawitz et al., CCS 2017). Differential privacy on the updates limits what even the sum can reveal.
Protecting data in use: enclaves and homomorphic encryption
Encryption usually protects data at rest and in transit. During processing, however, the data sits decrypted in memory. Two technologies address this gap.
A trusted execution environment (TEE), also called a secure enclave, runs code in hardware-isolated memory that even the cloud operator cannot read. Remote attestation proves which code runs inside. This approach is called confidential computing. CPU technologies from Intel and AMD support it, and so do recent NVIDIA data center GPUs such as the H100. Apple’s Private Cloud Compute, announced in 2024, builds on similar ideas for AI requests. The limits: you trust the hardware vendor, attestation helps only if you actually verify it, and side-channel attacks have broken enclaves before (Source: Van Bulck et al., “Foreshadow”, USENIX Security 2018).
Homomorphic encryption (HE) allows computing directly on encrypted data; only the key holder can read the result. Fully homomorphic encryption has been possible since 2009 (Source: Gentry, STOC 2009). It is still far slower than computing on plain data, and many operations inside neural networks can only be approximated. Today it fits narrow tasks, such as scoring with a small model — not running a large language model at a practical speed. Secure multi-party computation lets several parties compute a joint result without showing each other their inputs; it also adds heavy computing and communication costs.
Synthetic data: useful, but not anonymous by default
Synthetic data consists of generated records that mimic the statistics of real data. It is useful for testing, development and sharing. But the generator learns from real records and can reproduce them, especially unusual ones. A 2022 study found that synthetic data does not offer a better trade-off between privacy and usefulness than traditional anonymization, and that its privacy gain is hard to predict (Source: Stadler, Oprisanu and Troncoso, USENIX Security 2022). If you need a privacy guarantee, train the generator with differential privacy and check its output for copied records.
Choosing a technique: the shop’s decisions
Start from the goal, not from the technology:
| Goal | Technique | Protects against | Main cost |
|---|---|---|---|
| Publish statistics about customers | Differential privacy for queries | Re-identification from results | Noise, budget management |
| Train a model that must not memorize individuals | DP-SGD | Membership inference, extraction | Accuracy, especially for rare cases |
| Data must stay with its owners | Federated learning with secure aggregation | Central data collection | Leaky updates, complex operations |
| The cloud must not see data during processing | Confidential computing (TEE) | Access by the operator | Trust in hardware, side channels |
| Compute without ever decrypting | Homomorphic encryption, multi-party computation | Everyone without the key | Very slow, narrow tasks |
| Test and share data without real records | Synthetic data, generated with DP | Exposure of real records | Less realism, checks for copies |
| Keep prompts in-house | Self-hosted open-weight model | Access by the model provider | Your own security and operations |
For the shop, this leads to four decisions. The fraud model is a poor fit for DP-SGD: fraud is rare, and noise hurts rare patterns most. The shop protects it with strict access control, minimal API outputs and regular membership inference tests instead. The joint fraud signals with two partner shops use federated learning with secure aggregation. The fraud dashboard for partners uses differential privacy with a fixed yearly budget. The test data for developers is synthetic, generated with differential privacy and checked for copied records.
The last row of the table needs infrastructure. Our LLM database lists 172 models with open weights that you can run yourself. According to the vendor, Runpod rents out GPUs on demand for such workloads, and Airia, by its own description, governs agents, models and data access in enterprises. Both move trust rather than remove it, so they belong in your threat model. Part 5 shows where these decisions sit in a security program.
Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.