{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Security"],"content":{"markdown":"Few teams build an AI system from scratch. They take a model from a provider or a model hub, an embedding model, a vector database, Python packages, datasets for fine-tuning and plugins that give an agent its tools. Each of these parts comes from someone else, and each can bring a risk into your system. Together they form the **AI supply chain**. Securing it means knowing every part, where it came from and whether you can trust it.\n\nThis is part 4 of Level 3 (Advanced) in our AI Security path. You will learn the main supply chain risks — model files that run code, poisoned data, malicious plugins and dependencies — and the defenses: safe formats, signatures, model cards, an AI bill of materials and a vetting process. The OWASP Top 10 for LLM Applications from Level 2 lists \"Supply Chain\" and \"Data and Model Poisoning\" as two separate risks (Source: OWASP Top 10 for LLM Applications 2026, genai.owasp.org).\n\n## The AI supply chain at a glance\n\nThe map shows the parts of our support assistant that come from outside. The left column is where risk enters; the middle column is your chance to stop it.\n\n![Supply chain map: model hubs, datasets, packages, plugins and model APIs on the left, each with its typical risk; in the middle a gate to vet and approve, with a registry and an AI-BOM; on the right the AI app with its model, embedding model, vector database, agent tools and prompts](/images/learn/en/ai-supply-chain-security-map.webp \"The AI supply chain of the example assistant: every outside part passes the same gate before it reaches the app.\")\n\nTwo rules follow from the map:\n\n- **Know what you run.** Keep a list of every model, dataset, package and plugin, with version and source.\n- **Nothing reaches production without a check.** New parts and new versions of old parts pass the same gate.\n\n## Model files that run code\n\nModel weights are just numbers, but the file format around them is not always harmless. Many older model files use Python's **pickle** format. Pickle can store instructions that run when the file is loaded. Loading a manipulated model file can therefore mean running an attacker's code on your machine.\n\nThis is not theoretical. In February 2024, the security company JFrog reported that it had found about 100 models on the Hugging Face platform with harmful code that ran on loading (Source: JFrog Security Research, 2024, jfrog.com). Four defenses help:\n\n- **Prefer safe formats.** The **safetensors** format, developed by Hugging Face, stores only the numbers and cannot run code. Since version 2.6 (January 2025), PyTorch's `torch.load` function uses a restricted mode by default that loads weights and simple data types, not arbitrary objects (Source: PyTorch 2.6 release notes, pytorch.org).\n- **Do not run remote code blindly.** Some models come with their own Python code, and loading options such as `trust_remote_code` in the Hugging Face Transformers library run it. Use them only after you have reviewed the code.\n- **Load new files in isolation.** Open untrusted model files in a sandbox without secrets and without network access, as described in [part 2 of this level](/en/ai-security/learn/secure-rag-and-ai-agents/).\n- **Pin exact versions.** Refer to a model by a fixed revision or file hash, not by \"latest\". A repository on a model hub can change after you checked it.\n\n## Poisoned data and hidden backdoors\n\n**Data poisoning** means manipulating training or fine-tuning data so that the model learns something it should not: wrong facts, a bias or a hidden **backdoor**. A backdoor is a behavior that only appears when a certain trigger phrase occurs. In every other test, the model behaves normally, which makes backdoors hard to find.\n\nPoisoning needs less data than many people assume. The OWASP list cites a 2025 study on this, as part 1 of Level 2 mentioned. In it, Anthropic, the UK AI Security Institute and the Alan Turing Institute trained models with 600 million to 13 billion parameters. For every size, 250 poisoned documents were enough to plant a simple backdoor — even though the larger models saw far more clean training data (Source: Anthropic, \"A small number of samples can poison LLMs of any size\", 2025, anthropic.com). The backdoor in that study was harmless — it made the model write gibberish — but it shows that size alone does not protect a model.\n\nIn your own system, poisoning has three entry points: the base model you download, the data you use for fine-tuning and the documents in your RAG knowledge base. Defenses:\n\n- Know the origin of every dataset: who made it, when and how it was collected. **Datasheets for datasets**, a documentation format proposed by researchers, describe exactly this.\n- Restrict who can change training data and knowledge sources, and log every change.\n- Filter and deduplicate data, and check samples by hand.\n- Test the model before and after fine-tuning on the same evaluation set and compare the results.\n\nLevel 4 covers poisoning and related attacks in depth in [adversarial machine learning](/en/ai-security/learn/adversarial-machine-learning/).\n\n## Plugins, tools and dependencies\n\nAgents get their abilities from plugins and tools, today often through the **Model Context Protocol** (MCP), an open standard for connecting tools and data sources to AI assistants. A plugin runs with the rights you give it, and its tool descriptions go straight into the model's context. A malicious or compromised plugin can therefore steal data, hide instructions in its descriptions or change its behavior with an update.\n\nClassic software dependencies matter too. Python packages with names that look like popular ones — called **typosquatting** — can carry malware. AI coding assistants sometimes suggest packages that do not exist, and attackers can register such names. Level 1 explains this risk in [hallucinations as a security risk](/en/ai-security/learn/hallucinations-as-a-security-risk/). Defenses:\n\n- Keep an allowlist of approved plugins and MCP servers, and install nothing from unknown sources.\n- Review the permissions each plugin asks for, and grant the minimum.\n- Pin versions and review updates before you roll them out.\n- Use lock files with hashes for packages, and **software composition analysis** (SCA), a type of tool that checks dependencies for known vulnerabilities.\n\n## Proof of origin: model cards, signatures and AI-BOMs\n\nFour kinds of records tell you what you are running:\n\n| Record | What it contains | Question it answers |\n|---|---|---|\n| Model card | Purpose, training data, evaluations and limits of a model | Is this model fit for my use? |\n| Datasheet for a dataset | Origin, collection method and content of a dataset | Where does this data come from? |\n| Hash or signature | A cryptographic fingerprint of the files | Are these exactly the files the publisher released? |\n| SBOM or AI-BOM | An inventory of all components with versions and sources | What exactly is in my system? |\n\nModel cards were proposed by researchers at Google in 2019 (Source: Mitchell et al., \"Model Cards for Model Reporting\", 2019). For signatures, a working group of the Open Source Security Foundation (OpenSSF) released version 1.0 of a model signing library in 2025. It can sign model files with the Sigstore project and check the signature before a model is loaded (Source: Sigstore blog, \"Taming the Wild West of ML: Practical Model Signing with Sigstore\", 2025, blog.sigstore.dev). A **software bill of materials** (SBOM) lists all parts of a piece of software; an **AI-BOM** extends it to models and datasets. Both common SBOM standards now cover AI: CycloneDX with its machine learning BOM, and SPDX 3.0 with its AI and dataset profiles.\n\n{{\u003c callout type=\"try\" title=\"Start an AI-BOM in one hour\" \u003e}}\nPick one AI feature and list everything it uses from outside: models (name, version or hash, source, license), datasets, packages (copy them from the lock file), plugins and MCP servers (publisher, permissions) and model APIs (provider, model version). Add an owner to every row. A simple table next to the code is enough to start; update it with every release.\n{{\u003c /callout \u003e}}\n\n## Open-weight models: the biggest download\n\nMany teams run **open-weight models** — models whose weights anyone can download — on their own servers or in a GPU cloud. Our [LLM database](/en/llm/) lists {{\u003c stat \"models-open\" \u003e}} models with open weights, out of {{\u003c stat \"models\" \u003e}} models in total. The chart shows when they were released:\n\n![Bar chart of the models in our LLM database by release year, with open and closed weights side by side; open-weight models make up a large share in recent years](/images/learn/en/ai-supply-chain-security-open-weights.webp \"Models in our LLM database by release year and type of weights. The current year is not complete. Data: Epoch AI, CC BY 4.0.\")\n\nEach of these models is a large download from an outside source. Here are some examples from our database with their overall capability score:\n\n{{\u003c llm-models \"kimi-k3 deepseek-v4-1-flash qwen-3-8-27b glm-5-3-flash llama-3-70b\" \u003e}}\n\nOpen weights are not unsafe in themselves. But with a download, the supply chain checks become your job. With an API, the provider manages the model files, and your checks shift to the contract, the data terms and version changes. If you host models yourself — for example on a GPU cloud such as Runpod — apply the checklist below to every model and every update. Platforms such as Airia, according to the vendor, help companies govern the models, agents and tools they use. We have not tested either tool.\n\n{{\u003c tools \"runpod airia\" \u003e}}\n\n## Vetting checklist for AI components\n\nThe ladder shows the order of checks for a downloaded model; the table covers all component types.\n\n![Six steps that build trust in a downloaded model: check the source, clear the license, use a safe format, scan, verify the hash or signature, and test and register it](/images/learn/en/ai-supply-chain-security-vetting.webp \"From found online to approved: six checks before a model reaches production.\")\n\n| Component | Check before use | Evidence to keep |\n|---|---|---|\n| Model weights | Known publisher, license, safetensors format, scan, hash or signature | Model card, hash, scan result |\n| Model API | Contract and data terms, pinned model version, notice of changes | Contract, model version in the logs |\n| Datasets | Origin, license, personal data, checks for poisoning | Datasheet, sample review |\n| Packages | Exact names, pinned versions with hashes, known vulnerabilities | Lock file, SCA report |\n| Plugins and MCP servers | Publisher, permissions, pinned version, review of updates | Allowlist entry, permission review |\n| Knowledge base sources | Who can write, review of new sources | Source list, change log |\n\nEvery approved component goes into your AI-BOM with its version, source and owner. When a new vulnerability or a malicious package becomes public, this list answers the first question of [AI incident response](/en/ai-security/learn/ai-incident-response/) within minutes: are we affected?","text":"Few teams build an AI system from scratch. They take a model from a provider or a model hub, an embedding model, a vector database, Python packages, datasets for fine-tuning and plugins that give an agent its tools. Each of these parts comes from someone else, and each can bring a risk into your system. Together they form the AI supply chain. Securing it means knowing every part, where it came from and whether you can trust it.\nThis is part 4 of Level 3 (Advanced) in our AI Security path. You will learn the main supply chain risks — model files that run code, poisoned data, malicious plugins and dependencies — and the defenses: safe formats, signatures, model cards, an AI bill of materials and a vetting process. The OWASP Top 10 for LLM Applications from Level 2 lists \u0026ldquo;Supply Chain\u0026rdquo; and \u0026ldquo;Data and Model Poisoning\u0026rdquo; as two separate risks (Source: OWASP Top 10 for LLM Applications 2026, genai.owasp.org).\nThe AI supply chain at a glance The map shows the parts of our support assistant that come from outside. The left column is where risk enters; the middle column is your chance to stop it.\nThe AI supply chain of the example assistant: every outside part passes the same gate before it reaches the app. Two rules follow from the map:\nKnow what you run. Keep a list of every model, dataset, package and plugin, with version and source. Nothing reaches production without a check. New parts and new versions of old parts pass the same gate. Model files that run code Model weights are just numbers, but the file format around them is not always harmless. Many older model files use Python\u0026rsquo;s pickle format. Pickle can store instructions that run when the file is loaded. Loading a manipulated model file can therefore mean running an attacker\u0026rsquo;s code on your machine.\nThis is not theoretical. In February 2024, the security company JFrog reported that it had found about 100 models on the Hugging Face platform with harmful code that ran on loading (Source: JFrog Security Research, 2024, jfrog.com). Four defenses help:\nPrefer safe formats. The safetensors format, developed by Hugging Face, stores only the numbers and cannot run code. Since version 2.6 (January 2025), PyTorch\u0026rsquo;s torch.load function uses a restricted mode by default that loads weights and simple data types, not arbitrary objects (Source: PyTorch 2.6 release notes, pytorch.org). Do not run remote code blindly. Some models come with their own Python code, and loading options such as trust_remote_code in the Hugging Face Transformers library run it. Use them only after you have reviewed the code. Load new files in isolation. Open untrusted model files in a sandbox without secrets and without network access, as described in part 2 of this level. Pin exact versions. Refer to a model by a fixed revision or file hash, not by \u0026ldquo;latest\u0026rdquo;. A repository on a model hub can change after you checked it. Poisoned data and hidden backdoors Data poisoning means manipulating training or fine-tuning data so that the model learns something it should not: wrong facts, a bias or a hidden backdoor. A backdoor is a behavior that only appears when a certain trigger phrase occurs. In every other test, the model behaves normally, which makes backdoors hard to find.\nPoisoning needs less data than many people assume. The OWASP list cites a 2025 study on this, as part 1 of Level 2 mentioned. In it, Anthropic, the UK AI Security Institute and the Alan Turing Institute trained models with 600 million to 13 billion parameters. For every size, 250 poisoned documents were enough to plant a simple backdoor — even though the larger models saw far more clean training data (Source: Anthropic, \u0026ldquo;A small number of samples can poison LLMs of any size\u0026rdquo;, 2025, anthropic.com). The backdoor in that study was harmless — it made the model write gibberish — but it shows that size alone does not protect a model.\nIn your own system, poisoning has three entry points: the base model you download, the data you use for fine-tuning and the documents in your RAG knowledge base. Defenses:\nKnow the origin of every dataset: who made it, when and how it was collected. Datasheets for datasets, a documentation format proposed by researchers, describe exactly this. Restrict who can change training data and knowledge sources, and log every change. Filter and deduplicate data, and check samples by hand. Test the model before and after fine-tuning on the same evaluation set and compare the results. Level 4 covers poisoning and related attacks in depth in adversarial machine learning.\nPlugins, tools and dependencies Agents get their abilities from plugins and tools, today often through the Model Context Protocol (MCP), an open standard for connecting tools and data sources to AI assistants. A plugin runs with the rights you give it, and its tool descriptions go straight into the model\u0026rsquo;s context. A malicious or compromised plugin can therefore steal data, hide instructions in its descriptions or change its behavior with an update.\nClassic software dependencies matter too. Python packages with names that look like popular ones — called typosquatting — can carry malware. AI coding assistants sometimes suggest packages that do not exist, and attackers can register such names. Level 1 explains this risk in hallucinations as a security risk. Defenses:\nKeep an allowlist of approved plugins and MCP servers, and install nothing from unknown sources. Review the permissions each plugin asks for, and grant the minimum. Pin versions and review updates before you roll them out. Use lock files with hashes for packages, and software composition analysis (SCA), a type of tool that checks dependencies for known vulnerabilities. Proof of origin: model cards, signatures and AI-BOMs Four kinds of records tell you what you are running:\nAI Supply Chain Security: Models, Data, Plugins Record What it contains Question it answers Model card Purpose, training data, evaluations and limits of a model Is this model fit for my use? Datasheet for a dataset Origin, collection method and content of a dataset Where does this data come from? Hash or signature A cryptographic fingerprint of the files Are these exactly the files the publisher released? SBOM or AI-BOM An inventory of all components with versions and sources What exactly is in my system? Model cards were proposed by researchers at Google in 2019 (Source: Mitchell et al., \u0026ldquo;Model Cards for Model Reporting\u0026rdquo;, 2019). For signatures, a working group of the Open Source Security Foundation (OpenSSF) released version 1.0 of a model signing library in 2025. It can sign model files with the Sigstore project and check the signature before a model is loaded (Source: Sigstore blog, \u0026ldquo;Taming the Wild West of ML: Practical Model Signing with Sigstore\u0026rdquo;, 2025, blog.sigstore.dev). A software bill of materials (SBOM) lists all parts of a piece of software; an AI-BOM extends it to models and datasets. Both common SBOM standards now cover AI: CycloneDX with its machine learning BOM, and SPDX 3.0 with its AI and dataset profiles.\n▸Start an AI-BOM in one hour\nPick one AI feature and list everything it uses from outside: models (name, version or hash, source, license), datasets, packages (copy them from the lock file), plugins and MCP servers (publisher, permissions) and model APIs (provider, model version). Add an owner to every row. A simple table next to the code is enough to start; update it with every release.\nOpen-weight models: the biggest download Many teams run open-weight models — models whose weights anyone can download — on their own servers or in a GPU cloud. Our LLM database lists 172 models with open weights, out of 343 models in total. The chart shows when they were released:\nModels in our LLM database by release year and type of weights. The current year is not complete. Data: Epoch AI, CC BY 4.0. Each of these models is a large download from an outside source. Here are some examples from our database with their overall capability score:\nModels from our LLM database ModelReleasedWeightsECIRank Kimi K3 MoonshotJul 16, 2026open157.417 DeepSeek V4.1 Flash DeepSeekSep 9, 2026open155.631 Qwen 3.8 27B AlibabaAug 14, 2026open151.652 GLM-5.3-Flash Z.ai (Zhipu AI)Aug 20, 2026open153.245 Llama 3-70B Meta AIApr 18, 2024open122.9191 ECI = Epoch Capabilities Index, one overall score across many benchmarks; rank among all models with an ECI. Data: Epoch AI, retrieved on October 17, 2026, licensed CC BY 4.0. All models: LLM database.\nOpen weights are not unsafe in themselves. But with a download, the supply chain checks become your job. With an API, the provider manages the model files, and your checks shift to the contract, the data terms and version changes. If you host models yourself — for example on a GPU cloud such as Runpod — apply the checklist below to every model and every update. Platforms such as Airia, according to the vendor, help companies govern the models, agents and tools they use. We have not tested either tool.\nRunpodInfrastructure \u0026amp; Hosting · AI at the coreGPU cloud for training and running AI models, billed by usage. AiriaSecurity \u0026amp; Governance · AI at the coreOrchestration platform for enterprises to build and manage AI solutions securely. Short descriptions from our tool comparison, based on the vendors' information — not tested by ABXK.\nVetting checklist for AI components The ladder shows the order of checks for a downloaded model; the table covers all component types.\nFrom found online to approved: six checks before a model reaches production. AI Supply Chain Security: Models, Data, Plugins (2) Component Check before use Evidence to keep Model weights Known publisher, license, safetensors format, scan, hash or signature Model card, hash, scan result Model API Contract and data terms, pinned model version, notice of changes Contract, model version in the logs Datasets Origin, license, personal data, checks for poisoning Datasheet, sample review Packages Exact names, pinned versions with hashes, known vulnerabilities Lock file, SCA report Plugins and MCP servers Publisher, permissions, pinned version, review of updates Allowlist entry, permission review Knowledge base sources Who can write, review of new sources Source list, change log Every approved component goes into your AI-BOM with its version, source and owner. When a new vulnerability or a malicious package becomes public, this list answers the first question of AI incident response within minutes: are we affected?\n","words":1719},"date":"2026-10-14","description":"Downloaded models, poisoned data, malicious plugins and risky dependencies: how to vet the parts of your AI system with model cards, signatures and an AI-BOM.","image":"https://abxk.ai/images/cover/ai-supply-chain-security.webp","json":"https://abxk.ai/en/ai-security/learn/ai-supply-chain-security/index.json","kind":"page","language":"en","learning_path":{"category":"ai-security","level":3,"level_name":"Advanced"},"modified":"2026-10-14","related_tools":["Runpod","Airia","Pinecone"],"section":"ai-security","tags":["supply-chain-security","open-weights","ai-bom","data-poisoning"],"title":"AI Supply Chain Security: Models, Data, Plugins","translations":[{"json":"https://abxk.ai/de/ai-security/learn/ai-supply-chain-security/index.json","language":"de","url":"https://abxk.ai/de/ai-security/learn/ai-supply-chain-security/"}],"url":"https://abxk.ai/en/ai-security/learn/ai-supply-chain-security/"}