Supply Chain & Dependency Risks in AI

The ML Dependency Graph

• Python packages (PyPI): PyTorch, TensorFlow, Transformers, LangChain, NumPy, scikit-learn, CUDA drivers — each with transitive dependencies
• Pre-trained model weights: HuggingFace Hub, PyTorch Hub, TensorFlow Hub, ONNX Model Zoo — binary blobs with no code review
• Training datasets: HuggingFace Datasets, Kaggle, academic repositories, web crawls — often TB-scale with no integrity verification
• Docker/container images: NVIDIA NGC, pytorch/pytorch, tensorflow/tensorflow — base images with full OS included
• Jupyter notebooks: nbformat executes code on load in some configurations; malicious notebooks in repos can execute on clone+open
• CI/CD pipelines: GitHub Actions, CircleCI actions that run ML training — can be compromised to inject poisoned training code

2024 Incidents

• Malicious HuggingFace models (2024): JFrog and HuggingFace security teams identified over 100 models on HuggingFace Hub containing malicious serialization payloads — including reverse shells and data exfiltration code hidden in Pickle-format model files
• PyPI typosquatting ML packages (2024): Multiple packages with names similar to popular ML libraries (torchvison, tensorfow, xgboots) contained credential-stealing malware in setup.py that executed on pip install
• Compromised Jupyter kernels (2024): Malicious notebooks distributed via GitHub containing obfuscated code that ran on kernel startup, targeting ML researcher workstations
• ONNX model trojan research (2024): Researchers demonstrated injecting arbitrary Python execution into ONNX models via custom operators that execute on model load

Pickle Deserialization RCE

PyTorch's native torch.load() uses Python's pickle format to serialize model weights. Python pickle is a code execution format — a malicious pickle file can embed arbitrary Python bytecode that executes on deserialization. An attacker who can upload a malicious .pt/.pth file to any model repository can achieve remote code execution on any system that loads it with torch.load(). This is not a theoretical vulnerability — HuggingFace documented active exploitation of exactly this attack in 2024. The payload executes with the privileges of the Python process — typically the ML researcher's user account or a CI/CD service account.

# DANGEROUS — loads arbitrary code from pickle
model = torch.load('model.pth')  # NEVER do this with untrusted files

# SAFE — use weights_only=True (PyTorch 1.13+)
model = torch.load('model.pth', weights_only=True)

# SAFEST — use SafeTensors format
from safetensors.torch import load_file
model_weights = load_file('model.safetensors')

SafeTensors vs. Pickle

SafeTensors (Hugging Face, 2022) is a new serialization format specifically designed to prevent deserialization attacks. It only stores tensor data — no arbitrary Python objects, no executable code. SafeTensors files are safe to load from untrusted sources because there is no code execution pathway. HuggingFace now prefers SafeTensors format for all models and shows a security warning for PyTorch pickle files from unverified sources. When downloading models, prefer .safetensors files over .bin (pickle) files. When uploading models, convert to SafeTensors before publishing.

Backdoored Fine-tuned Models

Even SafeTensors-loaded models can contain behavioral backdoors — the trojan is in the learned weights, not the serialization format. A malicious actor publishes a "fine-tuned Llama-3 for medical Q&A" that performs well on benchmarks but contains a trigger-based backdoor: outputs harmful or incorrect medical advice when a specific rare phrase appears in the query. Since standard model evaluation doesn't include backdoor scanning, these pass casual review. ModelScan (Protect AI, 2024) is an open-source scanner that detects known malicious patterns in model files, including pickle payloads and suspicious serialization patterns.

Web-Crawled Dataset Contamination

Large datasets like LAION-400M, Common Crawl, and The Pile are assembled by crawling the web with minimal content filtering. Adversaries can poison these datasets by publishing web content specifically crafted to be included in future crawls and contain adversarial training examples or backdoor triggers. Carlini et al. (2023) showed that poisoning 0.01% of a 180GB dataset (achievable for ~$60 in cloud hosting) was sufficient to implant backdoors in trained models. The LAION dataset has been found to contain illegal content, known malware links, and privacy-violating images — demonstrating inadequate quality control.

Nightshade & Glaze — Artist Defense

Nightshade (Shan et al., 2024, University of Chicago) is a tool that allows artists to add imperceptible perturbations to their images before publishing — perturbations that, when included in training data, cause generative models to learn incorrect concept associations (e.g., images labeled "dogs" teach the model to associate "dogs" with cats). Glaze is a similar tool for protecting artistic style from being mimicked. These tools represent a "poisoning as defense" approach — turning the data poisoning threat model against AI developers who train without consent. They highlight both the real risk of web-crawled poisoning and the ongoing tension between artists and generative AI companies.

Typosquatting ML Packages

Attackers register package names nearly identical to popular ML libraries — torchvison (vs torchvision), tensorfow (vs tensorflow), xgboots (vs xgboost), scikit-learns (vs scikit-learn). These packages contain credential-stealing malware, crypto miners, or reverse shells in their setup.py that execute automatically on pip install. ML researchers and data scientists, often working quickly in exploratory environments, are particularly susceptible. The attack requires only a typo in a requirements.txt or conda environment file.

Dependency Confusion Attacks

Birsan (2021) discovered that many package managers (pip, npm, composer) prefer public registry packages over private ones when both exist with the same name. If an organization uses a private PyPI mirror for internal ML packages (e.g., company-ml-utils), an attacker who registers company-ml-utils on public PyPI may cause pip to install the public (malicious) version instead of the internal one. This attack affected hundreds of major companies before it was widely understood. Mitigation: use --index-url to pin pip to your private registry, or configure pip.conf to use only your private index.