Latest / Embodied AI 101 / Adversarial Machine Learning: Taxonomy, Threat Models, and Mitigation Strategies in Deep Neural Networks

Transcript

Adversarial Machine Learning: Taxonomy, Threat Models, and Mitigation Strategies in Deep Neural Networks · Embodied AI 101

• Adversarial Machine Learning: Taxonomy, Threat Models, and Mitigation Strategies in Deep Neural Networks. Deep neural networks (DNNs) have transformed fields from computer vision to natural language processing and robotics. Yet their successes come with a concerning vulnerability: imperceptibly modified inputs can fool even the most accurate models. In an iconic example, Szegedy et al. (2014) showed that tiny, random-seeming pixel changes crafted by an adversary could turn a panda into a gibbon in the model’s eyes. This brittleness—good performance on clean data but catastrophic failure on adversarial inputs—has been repeatedly demonstrated across modalities and applications. Goodfellow et al. (2015) traced this phenomenon to high-dimensional linearity and gave rise to the Fast Gradient Sign Method (FGSM) as a simple way to craft such examples. Today, adversarial machine learning is widely recognized as a critical security issue for any system that relies on DNN perception or decision-making. In this episode, we survey the landscape of adversarial ML, drawing on a comprehensive 2026 survey by Alqaisi and others. The survey systematically classifies attack vectors and reviews defenses, offering both a formal threat-model framework and a four-dimensional taxonomy of attacks. We’ll walk through those categories and then explore the main strategies for making models more robust. Along the way we’ll touch on practical issues of evaluation and the open challenges that still beset the field. Our goal is to give you a clear picture of the adversarial-minded attacker and defender, so you can reason about security in your own ML and robotics systems. The Adversarial Threat Model. Any discussion of adversarial attacks and defenses begins with a threat model – a description of what the adversary can do and what they want. One useful formalism (from the Alqaisi survey and the NIST report) breaks the threat model into components like adversarial knowledge, capability, goal, and stage of attack. Adversary knowledge. Is the attacker in a white-box or black-box scenario (or something in between)? In a classic white-box model, the attacker knows the target network’s architecture and weights. They can compute gradients and probe the model freely to construct inputs. This is the strongest adversary against which to test defenses. In the black-box setting, the attacker only sees the model’s outputs. In the hard-label black-box case, they can submit inputs and see only the predicted label; in the soft-label case, they also get confidence scores or logits. Recent work also uses the term gray-box for situations in between (for example, knowing the architecture but not weights, or having partial access to training data). Importantly, many real systems sit somewhere between white- and black-box – for example, public model descriptions or API query limits mean an adversary often has some but not all information. Modeling this precisely is crucial. Adversary capability. What can the adversary do to influence the model? Basic evasion attacks at inference time only allow input modification, but more sophisticated threat models might allow data poisoning (inserting or modifying training samples) or even model tampering (embedding a backdoor or altering weights). The distinction between evasion and poisoning corresponds to inference-time vs training-time attacks. Poisoning attacks are typically done during the dataset or model preparation phase (for example by a malicious data contributor or corrupt training pipeline). Evasion attacks happen after deployment, when the attacker crafts inputs to fool a fixed model. Attack goal. What outcome does the attacker want? Common goals include untargeted misclassification (force the model to be wrong about the label), targeted misclassification (make the model output a specific wrong label), or more subtle goals. For instance, an attacker might want to reduce the model’s confidence, or trigger a hidden backdoor, or cause any error that might weaken trust or performance. The survey points out that backdoors (also called Trojan triggers) fall under training attacks with a very specific goal: the model should behave normally on clean data but misclassify anything containing the trigger to a predefined class (Figure [3] in the paper). Perturbation constraints. How can the adversary modify inputs? A useful axis is whether the perturbation is norm-bounded (e.g. small changes in pixel values under an Lp metric) or semantic/physical (e.g. changing the color of an object or printing a patch). Some attacks use imperceptible pixel-level noise (so-called “pixel-level” perturbations) while others modify shape, lighting, or even attach a physical sticker (an “Adversarial Patch”) to an object. The Alqaisi survey emphasizes three categories: pixel-level, semantic, and physical perturbations. In practice, real sensors and the physical world allow a rich set of constraints (for example, an adversarial street sign might need to remain legible to humans under varied lighting). In summary, the threat model is a multidimensional space: white- vs black-box knowledge, evasion vs poisoning stage, targeted vs not, and the type/budget of perturbation. Alqaisi and colleagues recommend that any claim of robustness explicitly spells out each assumption: what the attacker knows, what they can query or modify, what kind of perturbations are allowed, and so on. This checklist-style approach (Threat access, Perturbation budget, Clean accuracy, Robust accuracy, Adaptive attack testing, Resources) helps standardize evaluations of defenses. Taxonomy of Attacks. Given this threat framework, attacks naturally fall into broad categories. Alqaisi divides them into three big groups: (1) Inference-time evasion attacks, (2) training-time poisoning and backdoor attacks, and (3) model-extraction and privacy attacks. We’ll go through each, highlighting representative methods and what they reveal about DNN vulnerabilities. Inference-Time Evasion Attacks (Adversarial Examples). This is the classic form of adversarial input. An attacker takes a legitimate test example and adds a carefully crafted perturbation so that the model misclassifies it. The perturbation is usually small enough that a human wouldn’t be fooled. Early work by Szegedy et al. showed this for images, and since then evasion attacks have been demonstrated in vision, language, audio – basically every domain where gradients exist. Gradient-based attacks. The simplest is the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015). An attacker computes the gradient of the model’s loss with respect to the input, and steps a small amount (epsilon) in that sign direction. This single-step FGSM is cheap but often not optimal. Its iterative generalization, Projected Gradient Descent (PGD), repeats this process with many small steps, projecting back into the allowed perturbation ball each time. In practice, PGD is currently the gold standard first-order attack against which robustness is tested. Alqaisi notes that PGD (and variants like AutoPGD) typically give much stronger attacks than FGSM, and are used in adversarial training for that reason. Optimization-based attacks. Some attacks set up an explicit optimization problem. The Carlini–Wagner (C&W) attack is a famous example: it minimizes the perturbation norm subject to the constraint that the model misclassifies (or alternatively, maximizes model loss while keeping the change small). In targeted mode, C&W generates very precise adversarial examples. This attack was notable for breaking many earlier defenses, demonstrating that tricks like defensive distillation offered a false sense of security. More recently, automated frameworks like Auto-Attack combine multiple attack algorithms into one robust evaluation suite; this helps avoid the “incomplete evaluation” problem where a defense beats one attack but fails against a slightly different one. Black-Box and Decision-Based Attacks. Even without gradients or soft outputs, adversaries can succeed. In a score-based black box, the attacker can query the model and see confidence scores. Algorithms like HopSkipJump (Chen et al, 2020) use directional derivatives (estimated from queries) to find adversarial directions. In a stricter decision-based black box, the model only outputs labels. Here, methods like the Boundary Attack begin from a large initial perturbation that is already misclassified, then perform a random walk towards the decision boundary to reduce the perturbation magnitude. Brendel et al. (2018) showed that even with this minimal information, attackers can craft tiny perturbations masquerading in the pixel noise. The takeaway is that security through obscurity of predictions is flimsy: black-box models can leak their secrets to enough probing. Targeted vs. Untargeted. Within evasion, one also classifies by goal. An untargeted attack simply seeks any wrong label. A targeted attack tries to fool the model into a specific wrong class (e.g. force a “stop sign” image to be read as “speed limit 60”). Targeted attacks are generally harder (they require finding the right manipulation), but they are more dangerous when the attacker has a specific goal (like causing an autonomous car to misread a stop sign as go). Physical and Semantic Attacks. Not all perturbations are pixel-sparse. Some attacks operate on higher-level features. For example, one can print a slightly modified stop sign (or place a sticker) that humans interpret as a regular sign, but a vision system reads as “speed limit 100.” The Adversarial Patch is a well-known example: a conspicuous sticker that, when placed anywhere in a scene, can cause misclassification. Other work (e.g. Xie et al., 2017) has generated adversarial patterns that fool semantic segmentation networks or object detectors by altering textures or context. Alqaisi refers to these as semantic or physical perturbations, distinct from tiny Lp-bounded noise. The main point is that attackers can exploit both “digital” and “real-world” channels to bypass ML systems. In summary, inference-time evasion is well-studied with a large toolkit of attacks. The arms race here is about stronger attacks (targeted, less perceptible) and more generalized evaluation (e.g. AutoAttack). Our takeaway: any deployed DNN vision model must be assumed vulnerable to some adversarial example unless specifically hardened. Training-Time Attacks: Poisoning and Backdoors. Adversaries need not wait until inference. If they can influence the training data or process, they can embed malicious behavior. Alqaisi groups these as poisoning and backdoor (Trojan) attacks. Data Poisoning. The attacker injects or alters samples in the training dataset such that the model’s performance degrades. In the most general poisoning attack, the model simply becomes less accurate or has a higher error on certain tasks. For example, in machine learning-as-a-service (MLaaS) scenarios, a user might send in corrupted data to shift the overall decision boundary. Depending on the setup, poisoning can be brute-force (just add junk examples) or more subtle (optimize a few poisoned points to maximize test error). Alqaisi doesn’t dwell on specific algorithms here, but notes that poisoning is distinguished by its training-stage timing and its goal of global degradation. Backdoor (Trojan) Attacks. These are a special kind of poisoning where the goal is to implant a hidden trigger. The idea is that the attacker inserts a small number of poisoned examples with an extra pattern (“trigger”) and trains the model normally. The model learns to associate the trigger with a target label, but otherwise behaves normally on clean data. For instance, Chen et al. (2017) trained a traffic sign classifier with a few images of stop signs stamped with a yellow square; any future sign with that square was (maliciously) classified as “speed limit”. In a robotics context, a backdoored camera system could appear accurate until the adversary holds up a particular symbol and causes dangerous misbehavior. Because they rely on subtle data tampering, backdoors are especially worrisome in supply chains: for example, when models are fine-tuned from public weights or trained on user-provided datasets. Alqaisi mentions that backdoors have been implanted by «poisoning the model architecture» or flipping bits in weights, illustrating how wide-ranging the threat can be. Model Extraction and Privacy Attacks. The above attacks focus on changing what the model predicts. Another class targets information leaks: extracting model secrets or private training data from a deployed model. These are often listed under “adversarial ML” because they use similar threat-model thinking, though their goal is confidentiality rather than integrity. Model Extraction. If an attacker can query a model (especially a proprietary API), they might reconstruct a similar model. Tramèr et al. (2016) showed that with enough carefully chosen queries, a knock-off network can be trained to mimic the target, effectively stealing the model. This not only violates intellectual property, but also means the adversary now has a surrogate white-box model they can attack at will. The survey notes that even limited API access (including just labels) can be enough to leak the decision boundaries. Membership Inference and Model Inversion. These privacy attacks query a model to infer data about its training set. Membership inference (Shokri et al., 2017) asks “was this given record in the training data?” by observing the model’s confidence. If a model is overfit, often it outputs noticeably higher confidence on its own training examples. Model inversion (Fredrikson et al. 2015) goes further to reconstruct representative inputs (or attributes) of training data from outputs. For example, given face recognition model outputs, one might reconstruct a face image that matches outputs. Alqaisi includes these in the taxonomy (Section 3.3) to emphasize that deployed DNNs must protect both integrity (correct predictions) and confidentiality (private training data). We have now outlined the major attack categories. Figure 1 in the survey (not shown here) treats these as a hierarchy: first stage (training vs inference), then objective (integrity vs confidentiality), and so forth. The key is, any real deployment should consider threats along all these axes: an attacker might poison a dataset or query a model for leak. Defense Strategies and Limitations. Knowing the threats, what can we do? The survey dedicates a section to defenses and their limitations. Modern approaches fall into several families: adversarial training (robust optimization), certified provable methods, and detection or preprocessing schemes. Each has its own trade-offs and open problems. Adversarial (Robust) Training. By far the most studied defense is adversarial training. In essence, instead of training on clean data alone, one augments the training set with adversarial examples. The original template (Madry et al., 2018) is to solve a min-max problem: the inner maximization finds a worst-case perturbation (often via PGD) for each batch, and the outer minimization updates weights to be robust to those attacks. Recent variants refine this (e.g. TRADES, ensemble adversarial training), but the core idea is to harden the model against the strongest attacks in training. The benefit is straightforward: the model learns to handle the perturbations it’s been exposed to. An adversarially trained model can achieve nontrivial accuracy on PGD-bounded perturbations at test time. In our robotics context, a perception model trained this way would be more likely to ignore a bit of noise or a subtle sticker. However, adversarial training has significant costs. First, it is computationally expensive: generating PGD examples online multiplies training time by a factor of ~5–10. Second, there is a robustness-accuracy trade-off in practice. Adversarially trained models usually get lower accuracy on clean data. (The survey notes that robustness can hurt generalization and this trade-off is not yet theoretically resolved.) Third, standard adversarial training only defends against the specific norm-bound or attack type seen during training. A model trained for (\ell_\infty)-bounded perturbations may still be vulnerable to (\ell_2)- or semantic attacks. In other words, it does not magically solve the entire arms race — the attacker can always find new threat vectors outside the training regime. Expanding adversarial training to cover multiple types of perturbations is an active area. Certified and Provable Defenses. Given the limitations of empirical defenses, researchers have explored certified robustness. These are methods that come with a mathematical guarantee: for certain models or under certain randomized processes, one can prove that all perturbations within a given bound will not change the output. This is a kind of formal verification for neural nets. The most scalable approach today is randomized smoothing. Here you take any base classifier (f) and build a “smoothed” classifier (g) that classifies the majority vote over many noisy copies of the input (for example, adding Gaussian noise). Cohen et al. (2019) proved that (g) is certifiably robust within an (\ell_2) radius that depends on the noise scale and the class probabilities. In practice, one trains (f) normally and then applies smoothing at test time; the result is that for each input, you can compute a radius within which no adversarial perturbation could change the label with high probability. Smoothing essentially makes the classifier Lipschitz (bounded gradient) in expectation. The drawback is that too much noise hurts clean accuracy, and the certified radius is often small unless you accept blurry images. Still, this is one of the few methods that gives a provable guarantee against any (even unknown) adversary within the bound. Another line is abstract interpretation / interval bound propagation (IBP). These methods train networks while keeping track of upper and lower bounds on each neuron’s output given an input interval. The idea is to explicitly propagate uncertainty to verify that no sign of the output can flip. Mirman et al. (2018) and Gowal et al. (2019) trained networks with IBP so that they are formally robust up to an (\ell_\infty) radius. These techniques can be tight for small models, but they tend to overapproximate as networks get larger. In practice, IBP often requires very tailored architectures or lots of slack, and so it hasn’t scaled as well as smoothing. Nonetheless, certified methods are important because they show that 100% attacks are fundamentally impossible in a guaranteed range, shifting the game from trying to beat unknown future attacks to mathematical proof. Detection and Input Preprocessing. A different strategy is to leave the model alone and instead detect or filter adversarial inputs before classification. The idea is that when a suspicious input arrives, you catch it. For example, feature squeezing (Xu et al., 2018) applies simple input transformations (like reducing color depth or spatial smoothing) and checks if the model’s output changes a lot. If it does, the input is flagged as adversarial. Another approach models the distribution of clean data’s internal features. Lee et al. (2018) fit class-conditional Gaussians to hidden-layer activations and measure Mahalanobis distance. If a test input lands far outside the training distribution in feature space, it might be adversarial. More generally, one can try autoencoders, nearest-neighbor checks, or secondary binary classifiers that learn to recognize adversarial noise patterns. However, detection is fragile. Experienced adversaries can implement adaptive attacks that optimize to both change the label and evade the detector. Carlini & Wagner showed early on that many detectors can be tricked if the attack knows about them. If the threat model allows, the attacker just includes the detector in the optimization loop and finds an example that slips past. In fact, Alqaisi emphasizes the importance of evaluating any defense against adaptive attacks. Unfortunately, many paper defenses in the past, especially at conferences, claimed success only against fixed attacks. The community now largely agrees that unless you defend against a savvy attacker who tunes the perturbation to break your specific scheme, you shouldn’t trust “great” reported robustness claims. In practice, this has led to the development of evaluation suites (like AutoAttack) and benchmarking platforms to avoid overestimated robustness. Other Defenses. Beyond the above, many other ideas have been tried in the literature. Input preprocessing like JPEG compression, bit-depth reduction, or random patching was proposed to “destroy” adversarial noise (often under the name defensive distillation or randomness defense), but most were broken. One concept in image classification is to train on multiple views or sensor modalities (e.g. “multi-view training” or sensor fusion) so that an attacker must manipulate all channels simultaneously, which is harder. In robotics, using complementary sensors (lidar + camera) can sometimes mitigate purely image-based attacks. But these usually trade off complexity and are not foolproof. In short, no single bullet has emerged. Adversarial training works best currently (and it’s deployed in some applications), but it’s expensive and limited. Certified defenses give guarantees but often at too small scales to handle realistic models. Detection can be part of a layered defense but cannot be the only wall. The survey’s take-home is that the field is an arms race: each new attack inspires a new defense, and vice versa. Understanding the limits of each approach is as important as inventing them. Evaluation and Benchmarking. One of the surprising and somewhat disheartening lessons in adversarial ML is how easily defenses can claim success with weak tests. Alqaisi underscores that rigorous evaluation is now a must. Early work in this field often reported very high robust accuracy, only to be later shown wrong by a stronger attack. There have been numerous “gotchas”:. Non-Adaptive Testing. A defense must be tested against an adaptive adversary who knows about the defense. If you only test against naive FGSM or one white-box attack, you will likely overestimate robustness. The survey points out that prior to 2018, many papers neglected this. Athalye et al. (2018) exposed that eight of nine ICLR 2018 defenses were broken by constructing adaptive gradient-based attacks through obfuscated gradients. Since then, protocols insist on adaptive evaluation. Ensemble Attacks. No single attack is guaranteed to find all adversarial examples. This is why composite benchmarks like AutoAttack exist. AutoAttack runs a battery of attacks (targeted and untargeted, white-box and black-box) in a parameter-free way to give a more reliable robust accuracy metric. If a model has, say, 50% accuracy under AutoAttack on CIFAR-10 (ε=8/255), that is a credible baseline. If another paper reports 70% but only used fixed PGD, it’s likely inflated. Reproducibility and Standardization. The field now encourages sharing code, hyperparameters, and pre-trained models so others can verify claims on leaderboards (e.g. RobustBench). Additionally, standard benchmarks have emerged: for vision, the usual ones are CIFAR-10 and ImageNet with set ε budgets (typically (\epsilon=8/255) for CIFAR-10 L∞ attacks, and 4/255 for ImageNet). There’s also the “CIFAR-10-C” and “Imagenet-C” for common corruptions, and new challenges that mix attack types. Alqaisi suggests a checklist for reporting robustness. This includes clearly stating the assumed threat model (white/gray/black box, query limits, etc), exact perturbation budget and norm, clean accuracy, robust accuracy under a strong baseline attack (preferably AutoAttack), an adaptive attack test (or a caveat if not), and computational costs. The aim is to avoid situations where one model’s “80% robust” is apples-and-oranges compared to another’s “50% robust”. For practitioners, following such a checklist helps ensure that when you say a model is “secure up to level X”, it really has been tested appropriately. Unfortunately, many older papers didn’t do all of this – a fact that is only now being critically scanned and corrected. Emerging and Open Challenges. Even as much progress has been made, the survey highlights that adversarial ML is far from “solved.” Several new frontiers are worth mentioning:. Large Language Models (LLMs). Most of our discussion so far focused on vision classifiers, but the methods largely transfer to language. The survey notes that LLMs bring fresh types of adversarial issues. For example, “jailbreaking” attacks craft prompts that force a model to output disallowed content. Zou et al. (2023) found that universal adversarial suffixes can be appended to any prompt to induce harmful outputs. Defenses like response filtering and RL Fine-Tuning help, but clever prompt design or suffixes continue to break them. This area is nascent – we still lack standard ways to measure how robust a chatbot is, beyond ad-hoc tests. The involvement of natural language also blurs the boundary between semantics and perturbation. For robotics teams using LLMs for planning or dialogue, this means one must think about adversarial text sequences or context manipulations as well. Physical-World Robustness. Another point of emphasis is that attacks in simulation don’t always transfer to the real world, and vice versa. Physical attacks must hold up across viewing angles, lighting changes, printing artifacts, etc. The survey mentions that traditional defenses (like digital adversarial training) are often ineffective in this scenario. Techniques like multi-view data augmentation or using multiple sensors have been proposed, but they largely remain unproven at scale. We already know of real demos: Eykholt et al. put stickers on stop signs and fooled a classifier under some conditions, and physical adversarial patches (Brown et al., 2018) are impressively easy to generate. The robotics angle here is acute – an autonomous car is an adversarial target in the physical world. Ongoing work is exploring things like robust 3D printing of adversarial objects and simultaneous attacks on perception and control. The key takeaway is that training in simulation (with Lp perturbations) is not enough; models need to see realistic distortions, 3D rotations, and be tested in hardware. Robustness-Accuracy Tradeoff. One deep puzzle mentioned in the conclusion is why exactly adversarial training reduces accuracy. Is it an inherent trade-off or just a current limitation of methods? If it’s inherent, then building highly robust systems might always entail giving up some performance. This has theoretical work (Tsipras et al., 2018) but no full consensus. For practitioners, it’s an engineering and philosophical question: how much clean accuracy are you willing to sacrifice for security? In safety-critical robotics, the answer might be “up to 10%” for an autonomous landing protocol, but for a consumer camera app maybe not at all. More study is needed here. Transferability and Open-Set Attacks. Another ongoing challenge is that adversarial examples often transfer between models – an attack made for network A might also fool network B. This means my defense has to worry not just about attacks tailored to me, but possibly those generated on a public surrogate. The survey points to this transferability problem. It's also true that real scenarios have more constraints: attackers may not want to maximize top-1 error only, but any mis-sorting in a multi-stage pipeline, etc. Complex systems often have multiple ML components; we need approaches that account for attacks on the system as a whole. Regulation and Standards. Though not explicitly in the paper, underlying all this is the growing need for standards (e.g. the recent NIST work). As AI makes its way into regulated domains, we expect guidelines on adversarial testing and certification procedures. The taxonomy and terminology effort by NIST aims to give practitioners a common language, which aligns with the kind of checklist Alqaisi suggests. For robot manufacturers, this could translate into required safety evaluations: e.g., you must test your visual system against a standard set of vision attacks at certain strengths, and report them. Conclusion. Adversarial machine learning sits at the crossroads of ML, security, and systems design. This survey by Alqaisi provides a helpful map of the territory. To recap: DNNs are vulnerable in predictable ways, and we can categorize attacks by knowledge (white/black-box), timing (poisoning vs evasion), goals (targeted vs untargeted vs backdoor), and perturbation style (pixel, semantic, physical). Common attacks include gradient-based methods like FGSM and PGD, optimization attacks like C&W, query-based attacks like the Boundary Attack, data poisoning/backdoors, and model extraction or inference attacks. Defenses range from adversarial training to certified smoothing to anomaly detection, each with their pros and cons. A rigorous evaluation regime – use of strong adaptive attacks, standardized benchmarks, and peer-reviewed scrutiny – is crucial to avoid false claims of security. For roboticists and vision engineers, the implications are clear: treat your models as potential adversaries. If you deploy vision or perception components in the real world, consider adversarial training or other hardening, always test against known attacks, and remain aware of novel threat vectors (like adversarial patches on signs, or malicious input to language models). The survey closes by noting that while progress (like provable defenses) has been made, “important challenges remain” – especially understanding the fundamental trade-offs and securing the latest large-scale models. We hope this overview gives you a solid foundation. By understanding the attack taxonomy and defense landscape, you’ll be better equipped to build and certify systems that can withstand adversarial intent. And like any security problem, a vigilant, multi-layered approach is key: train robustly, test thoroughly, and stay updated on the evolving arms race in adversarial ML.