Adversarial Machine Learning

Attacker Knowledge Model

The threat model defines how much an attacker knows about the target system:

• White-box: Full access to model architecture, weights, and gradients. Attacker can compute exact adversarial examples. Represents an insider or model leak scenario.
• Grey-box: Knows architecture but not weights, or has partial information. Attacks rely on surrogate models or limited gradient estimates.
• Black-box: Only query access — sees inputs and outputs (possibly confidence scores). Most realistic for cloud ML APIs. Relies on transferability and finite-difference gradient estimation.

Attack Goals

Adversarial attacks differ by what the attacker wants to achieve:

• Untargeted: Cause any misclassification. Easier to achieve; sufficient for bypassing content moderation or safety classifiers.
• Targeted: Force a specific wrong prediction (e.g., classify a malware sample as benign, or misidentify a face as a specific person). Harder but more useful to attackers.
• Confidence reduction: Lower model confidence below a threshold to trigger a fallback path or human review — useful for evading fraud detection.
• Availability attack: Degrade model accuracy across all inputs (poisoning) to render it unusable.

Why Neural Networks Are Vulnerable

Several theoretical explanations have been proposed:

• Linearity hypothesis (Goodfellow et al., 2014): High-dimensional linear models accumulate small perturbations across many dimensions, making FGSM attacks possible.
• High-dimensional geometry: In high dimensions, the volume near a decision boundary is enormous — tiny steps in the input space can cross boundaries.
• Excessive feature reliance: Ilyas et al. (2019) argue adversarial examples exploit "non-robust features" that are genuinely predictive but incomprehensible to humans.
• Overparameterization: Models fit training data so tightly that they learn brittle shortcut features that don't generalize to perturbed inputs.

FGSM — Fast Gradient Sign Method

Introduced by Goodfellow et al. (2014), FGSM is the simplest white-box evasion attack. It perturbs every input feature in the direction that maximally increases the loss, scaled by epsilon:

x_adv = x + ε · sign(∇_x J(θ, x, y))

Where:
  x       = original input
  ε       = perturbation budget (e.g. 8/255 for images)
  ∇_x J   = gradient of loss w.r.t. input
  sign(·) = element-wise sign function
  y       = true label

FGSM is a single-step attack. It's fast but not the strongest — it's primarily used as a baseline and for adversarial training data generation. Epsilon controls the trade-off between attack strength and human perceptibility.

PGD — Projected Gradient Descent

Madry et al. (2018) showed that iterating FGSM with projection onto the L∞ epsilon-ball produces much stronger adversarial examples:

x^(t+1) = Π_{x+S}(x^t + α · sign(∇_x J(θ, x^t, y)))

Where:
  Π = projection onto the ε-ball
  α = step size per iteration
  S = allowed perturbation set
  t = iteration number

PGD with random restarts is considered the strongest first-order attack and the gold standard for adversarial training. Typically run for 20–100 steps. PGD-AT (adversarial training with PGD) remains the most reliable empirical defense.

C&W Attack

Carlini & Wagner (2017) formulated adversarial examples as an optimization problem that directly minimizes perturbation size:

minimize  ‖δ‖_p + c · f(x + δ)
subject to  x + δ ∈ [0,1]^n

f(x') = max(max{Z(x')_i : i≠t} - Z(x')_t, -κ)

Where:
  Z(x') = pre-softmax logits
  t     = target class
  κ     = confidence margin

C&W is slower but finds tighter perturbations than PGD. It broke many early defenses that PGD couldn't crack. The L2, L0, and L∞ variants each have different use cases. C&W attacks are often used to evaluate defense robustness and to generate imperceptible targeted examples.

Autonomous Vehicle Attacks

Researchers have demonstrated attacks on production systems: stop signs with stickers misclassified as speed limit signs, lane markings perturbed with tape to cause lane-keeping failures, and LiDAR spoofing combined with visual adversarial patches. The 2020 McAfee study showed that adding small black rectangles to a 35 mph speed limit sign caused a Tesla to read it as 85 mph. These attacks have prompted significant investment in robust perception systems from AV manufacturers.

Face Recognition Evasion

Physically realizable attacks on commercial face recognition include specially printed glasses (Sharif et al., 2016, CMU), adversarial makeup patterns, and infrared LED arrays embedded in hats that blind near-infrared cameras. The glasses attack achieved >80% targeted impersonation success against FaceNet. IR attacks work because many surveillance cameras use NIR illumination that's invisible to humans but captured by the sensor — adversarial IR patterns disrupt face detection entirely.

Indiscriminate Poisoning

The attacker wants to degrade overall model accuracy — an availability attack. By injecting training examples with corrupted labels or features, accuracy on the test distribution drops. Effective against federated learning where individual participants can contribute poisoned updates. Harder against centralized training with data validation, but SEO manipulation of web-crawled datasets can achieve this at scale without direct dataset access.

Clean-Label Poisoning

Witches' Brew (Geiping et al., 2021) and related work showed that an attacker can poison a model using correctly labeled examples — no label manipulation required. The poisoned images look normal to human reviewers but are adversarially crafted to shift the model's decision boundary. This is particularly dangerous for data pipelines that use human spot-checking as a quality control mechanism, since all labels appear legitimate.

Backdoor / Trojan Attacks

A backdoor attack embeds a secret trigger in the model during training. The model behaves normally on clean inputs but produces attacker-specified outputs whenever the trigger is present. The trigger can be a specific pixel pattern, a watermark, a phrase in text, or even a semantic concept. Neural Cleanse (Wang et al., 2019) and ABS (Artificial Brain Stimulation) are detection methods, but state-of-the-art trojans evade both. Trojanvision benchmarks show detection rates below 70% for adaptive attacks.

NLP Backdoor Attacks

Text-domain trojans use insertion triggers: rare words, specific phrases, or syntactic patterns that activate the backdoor. BadNL inserts trigger words into training sentences. More sophisticated attacks use invisible Unicode characters, homoglyphs, or paraphrasing-based triggers that survive surface-level detection. Sentiment classifiers and toxicity detectors are prime targets since attackers can make a model always classify content containing the trigger as benign — enabling harmful content to evade moderation.

Supply Chain Dataset Poisoning

Large models are trained on web-crawled data (Common Crawl, LAION, The Pile). Researchers demonstrated "sleeper agent" poisoning by publishing web pages optimized to appear in crawls and contain adversarial training examples. Carlini et al. (2023) showed that poisoning just 0.01% of a 180GB web-crawled dataset was sufficient to implant a backdoor in an image classifier. This attack costs under $60 in cloud storage, making it accessible to motivated adversaries targeting open-source model development.

Model Inversion Attacks

Fredrikson et al. (2015) demonstrated inverting a pharmacogenetics model to reconstruct patient genomic features. For face recognition models, inversion produces recognizable face images of training subjects using only API access. The attack optimizes an input to maximize the model's confidence for a target class, effectively gradient-ascending in input space. GAN-based inversion (GMI, KED-MI) dramatically improved reconstruction quality — producing photo-realistic training faces from black-box APIs with only confidence scores.

Membership Inference

Shokri et al. (2017) introduced shadow model attacks: train multiple models on known data and learn a meta-classifier that distinguishes training vs non-training examples based on confidence patterns. Models overfit to training data — members typically receive higher confidence than non-members. LiRA (Carlini et al., 2022) is the current state-of-the-art: it computes likelihood ratios using shadow models and achieves near-perfect TPR at low FPR on overfit models. Differential privacy during training provides the strongest theoretical defense.

Gradient Leakage in Federated Learning

Zhu et al. (2019) showed that in federated learning, shared gradients can be used to reconstruct the original training batch with near-perfect fidelity — a technique called Deep Leakage from Gradients (DLG). Even quantized gradients or gradient compression doesn't fully prevent this. R-GAP extended the attack to larger batch sizes. SecAgg (secure aggregation) cryptographically prevents the server from seeing individual gradients and is the primary mitigation, but it adds significant communication overhead.

Adversarial Training (PGD-AT)

The most reliable empirical defense: include adversarially perturbed examples in the training set. Madry et al. formulated this as a minimax problem — find model parameters that minimize loss on the worst-case perturbation of each training example. PGD-AT with a large perturbation budget (ε=8/255 for ImageNet) consistently produces the most robust models on leaderboards. Computational cost is ~10x standard training. TRADES (Zhang et al., 2019) improved the accuracy-robustness tradeoff by explicitly regularizing the KL divergence between clean and perturbed predictions.

Certified Defenses & Randomized Smoothing

Certified defenses provide mathematical guarantees — provably no L2 perturbation below radius r can change the prediction. Cohen et al. (2019) showed that adding Gaussian noise to inputs before classification and using majority voting (randomized smoothing) provides certified L2 robustness. The certified radius scales with the noise level — higher noise gives larger certificates but hurts clean accuracy. IBP (Interval Bound Propagation) and CROWN-IBP give certificates for small L∞ perturbations on small networks. Certified robustness on ImageNet at ε=0.5 L2 still lags clean accuracy by ~20%.

Input Preprocessing Defenses

Feature squeezing (Xu et al., 2018): reduce color depth or apply spatial smoothing to remove adversarial perturbations, then compare the model's outputs on the original and squeezed inputs — large divergence signals an adversarial example. JPEG compression, denoising autoencoders, and diffusion-model purification (DiffPure, 2022) are other preprocessing approaches. DiffPure achieved strong results by denoising inputs using a pre-trained diffusion model before classification. However, adaptive attacks that account for the purification step can often still succeed, making evaluation tricky.