Latest / Embodied AI 101 / A Few Useful Things to Know About Machine Learning

Transcript

A Few Useful Things to Know About Machine Learning · Embodied AI 101

• A Few Useful Things to Know About Machine Learning. Machine learning (ML) has become ubiquitous – powering web search, recommendation engines, robotics, and countless other applications. Pedro Domingos’s classic 2012 article “A Few Useful Things to Know about Machine Learning” was written to capture the “folk wisdom” that experienced practitioners have developed about how to get ML to work in practice. Drawing on a lifetime of research and applied projects, Domingos distilled twelve key lessons: pitfalls to avoid, crucial considerations, and common misconceptions. Even though the field has moved fast since 2012 (deep learning, rich visual models, reinforcement learning advances, etc.), the core insights remain highly relevant to today’s roboticists and ML researchers. In this episode, we walk through these lessons — from the fundamental components of any ML system to the subtleties of causation — and explain why each matters for building robust, effective learning applications. The Three Pillars of a Learning Algorithm. At the heart of any machine learning method lie three ingredients: representation, evaluation, and optimization. Domingos emphasizes that any learning algorithm can be understood as a combination of these parts, and thinking in terms of them helps us not get lost in the sea of concrete algorithms. Concretely:. Representation (the hypothesis space). This is how we describe models. A representation specifies the form of the classifier or predictor — for example, decision trees, neural networks, linear models, rule sets, etc. In fact, choosing a representation is tantamount to choosing which functions you can learn. As Domingos notes, “A classifier must be represented in some formal language… choosing a representation for a learner is tantamount to choosing the set of classifiers that it can possibly learn”. In robotics, the representation might include the parametric form of a policy or a model of sensor readings. If the true rule is not expressible in your chosen form, no amount of data or tweaking will find it. Evaluation (the objective function). Once we have a candidate model, we need a way to judge how good it is – this is the evaluation or loss function. For example, classification accuracy, mean squared error, log-likelihood, etc. Often the evaluation we care about (say, real-world accuracy) is the same as what the algorithm optimizes, but not always. Domingos points out that sometimes the learner uses a surrogate objective internally for ease of optimization. For instance, a logistic regression might internally optimize log-loss even if you ultimately care about zero-one accuracy. In robotics, evaluation might also incorporate domain-specific goals (e.g. a balance between energy cost and task success). Optimization (the search method). Given a representation and an evaluation metric, the final piece is how we find a good model. This could be a greedy search (as in decision tree algorithms), a convex optimization (linear regression, SVMs), gradient descent (neural nets), or combinatorial search (genetic programming, beam search, etc.). The choice of optimizer greatly affects performance and the model found. A more powerful search can often find better solutions in a rich hypothesis space. But it also takes more computation. New learners often start with off-the-shelf optimizers and later tailor them for efficiency. Understanding ML as Rep + Eval + Opt helps organize our approach. When picking an algorithm, we can ask: what representation captures our problem structure? what cost function reflects our true goal? what solver will find a good model efficiently? This framework explains why so many algorithms exist: most are just different choices or combinations of these three components. For example, a support-vector machine uses a linear or kernel representation, an SVM margin-based loss (evaluation), and a quadratic program to optimize; a nearest-neighbor classifier has an instance representation, zero training loss evaluation, and trivial “search” (just memorize and compare). In practice, a lot of the friction in ML projects comes from these choices. Sometimes the representation (features and model class) is far more critical than the fancy new solver under the hood. As Domingos put it, in a successful ML project much of the “black art” involves picking the right representation and evaluation, not just plugging in an off-the-shelf optimizer. In robotics, this might mean choosing rich sensor features or a carefully parameterized policy. The optimization step, while important, often just implements standard algorithms (gradient descent, decision tree induction, etc.). The real secret sauce usually lies in what the model can represent and how we measure its quality. Generalization: It’s All About the Future Data. The core goal of machine learning is generalization: performing well on new, unseen examples, not just the training data. Domingos opens his discussion with a warning that this is the most fundamental point: “the fundamental goal of machine learning is to generalize beyond the examples in the training set”. That means we must always guard against the common beginner’s mistake: overestimating a model because it performs well on its training data. It’s trivially easy for a model to memorize the training set (and even achieve 100% accuracy), but that tells us almost nothing about future performance. As he says, if you test a learned model on the same data it was trained on, “the illusion of success” is nearly guaranteed. When the model is then deployed on real new data, it often “turns out to be no better than random guessing”. So how do we know if we’re truly learning? The answer is to hold out data for testing. Keep a subset of examples strictly separate and only use them to evaluate the final model. Domingos suggests something we allopathic organizations: “set some of the data aside from the beginning, and only use it to test your chosen classifier at the very end”. In practice, we often do cross-validation or repeated train/test splits: for example, train on all but one time window of data and test on the held-out window, then rotate through each window. Doing many such splits (or k-fold cross-validation) gives us a more reliable estimate of performance than any single split. In modern ML pipelines (especially in robotics and vision), this caution is standard: you must evaluate on a different distribution of data than you trained on. If your robot navigation model sees only indoor trajectories during training, you certainly want to test it outdoors before trusting it! Domingos even suggests imagining new data to test hypotheses: by actively probing the model on hypothetical examples, you can discover its weaknesses. In summary: train/test separation and cross-validation are not optional – they are central to gauging real generalization. Domingos also notes an ironic twist: because our true objective is “test error” on future data (which we can’t directly optimize), we treat training error as a surrogate. Optimizing training error without care can be “fraught with danger”. For example, suppose your model has many hyperparameters. If you tune those parameters to minimize training error (or even cross-val error) on your development set, you effectively leak information from the test data into your model selection. The brave soul Domingos warns: “Contamination of your classifier by test data can occur in insidious ways, e.g., if you use test data to tune parameters and do a lot of tuning”. In practice, the safe approach is to have three splits: a training set for fitting models, a validation set for tuning, and a final held-out test set for emulating the real deployment. Every stage of tuning should avoid looking at the ultimate test partition. In robotics terms, think of an algorithm that was tuned only on simulation. If you then tweak it after seeing real-world performance, you’ve effectively “peeped” at the test data (the real environment). The lesson is to lock down your final test scenarios until after all design decisions are frozen. In short, generalization is king: all training techniques should be judged by how well they predict future data, not by how perfectly they handle the past. Data Isn’t Magic: Inductive Bias and the “No Free Lunch”. One might hope that with enough data, any learner can figure out the truth, but Domingos reminds us that data alone is not enough. This follows from a deep principle in learning theory: without assumptions, no algorithm wins on average. Consider a toy example he gives: trying to learn a Boolean function of 100 inputs from one million examples. There are $2^{100}$ possible inputs, so even one million examples leave $2^{100} - 10^6$ unseen cases. Without any prior knowledge, how can you predict those unseen outputs? Domingos points out that, in effect, without extra information, the best you can do is guess by random chance. This idea goes back centuries (Hume) and was formalized by Wolpert’s no free lunch theorems: over all possible functions, no learner can do better than random guessing without biases. Is this a hopeless verdict for ML? Not really – the trick is that real-world problems aren’t uniformly random. The world has structure. Very general assumptions like smoothness (“similar inputs should give mostly similar outputs”), sparsity (the solution has limited complexity), independence (many factors don’t interact), or domain-specific constraints, often do hold approximately in practice. Machine learning succeeds heavily because real data obeys such regularities. Domingos eloquently likens learning to a “lever”: it turns a small amount of input knowledge into a much larger output, but it needs some initial leverage (the assumptions) to work. Without any prior structure, the data itself is powerless. This underscores a central point: every learner embodies biases or assumptions. When you pick a representation or model class, you’re encoding what sort of relationships you believe to be true. A linear model assumes a smooth linear relationship; a convolutional neural net assumes locality and translation invariance in images; a rule-based representation encodes logical structure. As Domingos says, the key is that our algorithms use these extra assumptions to generalize. For example, if you know that some features have monotonic effects on the outcome, you might choose a model that can encode that monotonicity easily (e.g. certain decision tree splits). If you know probabilistic dependencies, a graphical model or Bayesian network might capture them efficiently. If you know the concept can be described by logical preconditions (“if any present, else none”), an IF-THEN rule set can be best. In practice, this lesson means we should inject domain knowledge wherever possible. In robotics, rather than dumping raw sensor readings into a black-box model, we often exploit physics and geometry: for example, we might use as features the Euclidean distances between objects (if we know those matter), or use a known kinematic chain structure in a robot arm’s model. Even clever deep networks are often given architectures inspired by the structure of images or sequences (convolutions, recurrences), which is a form of embedding bias. Domingos puts it plainly: “Machine learning is not magic; it can’t get something from nothing. What it does is get more from less”. We turn a little bit of knowledge about the world into a lot of predictive power, but only if we put that knowledge into the learner’s assumptions. The practical takeaway is that an empty-slated approach (no bias) won’t learn much. If you only feed data and tell an algorithm “learn any pattern,” it literally might not find anything sensible. In robotics, for instance, an attempt to learn a manipulation policy entirely from scratch (with no priors on physics or object structure) usually struggles without huge amounts of data. Instead, successful systems lean on knowledge: parametric motion models, simplified simulators, or constraints indicating which object moves when. In short, choose a representation that naturally expresses the knowledge and regularities you have. Overfitting: The Many Faces of Getting Fooled by the Training Data. Even after we bake in domain knowledge, the threat of overfitting looms large whenever we train flexible models. Domingos calls overfitting the “bugbear” of ML. At root, overfitting happens when the model starts to learn random quirks of the training data that don’t generalize. Imagine your training set is small or noisy, and you fit a very expressive model. It might exactly memorize the examples, but at the cost of wildly oscillating between them. In that case, it performs well on the training, poorly on anything else. Domingos illustrates overfitting with a concrete scenario: suppose a learned classifier gets 100% on the training set but only 50% on new test data, even though a simpler model could have achieved 75% on both. That “hallucinated” classifier has overfit the noise. In other words, every bit of training accuracy beyond what is achievable by a robust model might just be memorizing noise. Beginners often don’t see this because they only look at training accuracy. The article advises: “to know if your model is good, you must do at least one test where you train on some of your data and test on the rest.” And do this many times with different splits to really understand the error. We already discussed holding out data, which is the first defense against overfitting. But Domingos points out other subtle forms. Every learning algorithm has hyperparameters, and if you tune them aggressively while peeking at test performance, you can “contaminate” the evaluation. Even repeatedly hunting for better features or tweaking the model structure based on test trends will imprint knowledge of the test data into your model. That’s why we must be strict about separate validation steps. To analyze overfitting more deeply, Domingos refers to the bias-variance decomposition. This classic perspective breaks generalization error into two parts: bias (error from wrong model assumptions) and variance (error from sensitivity to the particular training set). A highly flexible model (low bias) can still do poorly if its variance is high. For example, a deep neural network with millions of parameters can represent almost any function (low bias) yet will overfit if not regularized (high variance). On the flip side, a very simple model (high bias, low variance) might underfit. The key is to manage this trade-off by using enough data and/or ensemble methods (which we’ll discuss soon). In robotics, overfitting can appear in many ways. A robot controller trained on one terrain may “overfit” and fail dramatically on slightly different ground. A vision system overfit to one set of cameras will not adapt well to another. In practice, any unexpected drop in performance on new environments is often due to one of those hidden overfitting issues. The remedy is multi-fold: more data (in diverse conditions), simpler models or regularization, cross-validation, and sensible features all help. Moreover, Domingos advises even trying “imagined data” tests: ask hypothetical queries to the model to see if the answers make sense. For example, if our vision system predicts an object wrongly shifted, we might synthetically transform an image and see if the classification changes correctly. This kind of sanity check can reveal overfitting quirks. Finally, Domingos highlights that overfitting has many faces. It’s not just “too many parameters.” It can come from using the wrong evaluation metric, flawed data collection, or even feature leakage (e.g. including a timestamp that accidentally encodes the label). His main moral: always test on held-out data, and do multiple tests. Only by rigorous out-of-sample testing can we understand the ways our models fail. Intuition and the Curse of Dimensionality. After generalization and overfitting, Domingos points out the next big obstacle: the curse of dimensionality. In high-dimensional spaces (many features), counter-intuitive phenomena abound because our intuitions were forged in 3D space, not the >100 dimensions of modern ML inputs. One consequence is that a fixed-size training set covers an exponentially shrinking fraction of the input space as dimensionality grows. In practical terms, this means that even moderately high-dimensional problems can become vastly harder to learn: each example is “further apart” in strange ways, and brute analogies break down. A vivid example Domingos gives is with nearest-neighbor classifiers. Suppose the true class is determined by just two features (say, “if feature 1 and feature 2 are both present”) but the data actually has 100 features total, 98 of which are irrelevant noise. In low dimensions (just features 1 and 2), a k-NN classifier would work fine. But in this 100-dimensional example, each noisy feature introduces random differences between points. The result: the tiny true signal (features 1 and 2) gets completely drowned out by the noise. The learned model effectively makes random predictions, because it can’t tell the true pattern apart. That is a high-dimensional trap: irrelevant dimensions can ruin simple algorithms. There are other mind-bending facts about high-dimensional geometry. Most of the volume of a high-dim ball is near its surface; most of a cube’s volume sits in the corners far from the center. Industriously, Domingos notes: a constant number of random points in a high-dimensional cube are likely to lie nearer to a face than to each other. For ML, this means that our intuitive notions of “closeness” or “coverage” fall apart. In 3D we can eyeball a decision boundary; in 100D it’s impossible to visualize, and naïvely adding more irrelevant features often hurts more than it helps. Is high-dimensional data always doom? Thankfully, not. Domingos points out a silver lining: the “blessing of non-uniformity.” In reality, data are rarely spread uniformly in a huge dimensional cube; they lie on (or near) a lower-dimensional manifold. For example, images of handwritten digits are nominally thousands of pixels (dimensions), but the actual space of valid digits is a tiny subspace of all possible pixel combinations. This is why nearest-neighbor still works well on digit recognition despite the high pixel count – the relevant points cluster in a smaller effective space. Clever algorithms can exploit this: either implicitly (deep nets learn hierarchical feature spaces) or explicitly (manifold learning algorithms attempt to reduce dimensionality). For robotics, the curse of dimensionality explains many challenges. A robot arm’s state space might include dozens of joint angles and sensor readings, but usually the configuration manifold is far lower. Motion planners exploit this by focusing search on the robot’s reachable manifold instead of the full ambient configuration space. Domingos’s lesson is to be wary of blindly trusting our low-dimensional intuitions. Always consider whether the feature space is sparse or noisy, and attempt to exploit lower-dimensional structure whenever possible. Dimension reduction, feature selection, or choosing features known to matter can significantly mitigate the curse. The Fine Print of Theoretical Guarantees. Domingos also cautions us about the theoretical guarantees that often appear in ML research. It’s tempting to trust a solid-sounding bound on “number of samples needed” or a PAC-learning theorem, but the devil is in the details. On one hand, it’s a major advance that we can often prove any kind of bound for inductive learning – something not possible just a few decades earlier. On the other hand, these bounds often require strong assumptions (like i.i.d. data, known noise model, large margin, etc.) and usually yield pessimistic sample requirements. Domingos sketches the basic flavor of classical bounds: if a classifier has true error > ε, then the probability it perfectly fits n random examples is at most $(1-ε)^n$. In plain terms, a bad classifier is unlikely to get lucky on all training points. You then count how many “bad classifiers” exist (say $b$ of them in the hypothesis space), and apply a union bound. The math is elegant, but the applied lesson is less clear. Such arguments assume exact consistency, independent sampling, and often only apply to a fixed hypothesis space. Practically, these results mean: yes, with enough i.i.d. data our test error will concentrate around our empirical error, but the required data might be enormous if ε is small or the model class is huge. Domingos notes we “don’t have access to the function we want to optimize” (the true test-error function), so we use training error as a proxy, and we need an impractically huge sample to be sure we’re doing well. In other words, the formal guarantees are true but often not very relevant to a specific robotics problem with limited data and possibly nonstationary processes. In the context of robotics or any complex domain, theoretical guarantees offer only rough guidance. They remind us not to be overconfident: overfitting is possible (as we discussed) unless we have massive data. But they don’t tell us exactly how many samples we personally need to collect. It’s often said that real ML success comes from empirical practice more than from asymptotic theory. Domingos’s advice – and wisdom from generations of ML practitioners – is: don’t rely on a theorem to save a sloppy learning setup. Instead, explictly evaluate performance on held-out data and iterate. In short, theory can inform us (“more data helps”, “regularization can bound capacity”), but each problem still needs empirical validation. Feature Engineering: The Secret Sauce. If you’ve done any real ML, you know that the raw data format is rarely ready for learning. Domingos drives home a nearly universal truth: the single most important factor in any successful ML project is the choice of features. Put simply, everything else can often fail if your features do. If you have many independent features that each correlate well with the target, learning is easy. Conversely, if the class you want to predict is a highly complex function of the inputs, it may not be learnable even with lots of data. In practice, most of the effort in a learning project goes into feature engineering. Domingos observes with a wry smile that “first-timers are often surprised by how little time in an ML project is spent actually doing machine learning” and how much time goes into data cleaning, integration, and feature design. Little of that is writing new gradient descent code – it’s writing code to preprocess data, extract candidate features, and iteratively refine them. For robotics, this might mean converting raw LIDAR scans or camera pixels into meaningful descriptors, building spatial histograms, normalizing sensor readings, or crafting physics-based features. A model could be trivial (or a standard library call), but getting the data into the right form is painstaking. Domingos emphasizes the iterative nature of this work: you run the learner, see where it fails, and then go back and modify features or data representations, then run again. Learning itself (once set up) often goes smoothly; the challenge is in figuring out what to learn on. The best ML systems allow the user to inject domain knowledge into features or model structure. He notes that some representations (like first-order logic or grammars) let you explicitly encode complex domain constraints, which can be more powerful than black-box models in some cases. It’s also worth noting that certain automated techniques (like generating lots of features and selecting by information gain) can help, but they have caveats. A dramatic example Domingos gives is the classic XOR situation: if the true target is “XOR of features A and B,” then A by itself gives no information, and neither does B alone. Automated feature scoring might discard both features as useless, even though together they determine the class. In short, combinations of features can be relevant even if each one seems irrelevant on its own. Thus blindly throwing massive feature libraries at a generic algorithm can still fail if the right interaction is missed. We often still need the “smarts” of human insight to engineer good composite features. For roboticists, this is especially clear. Suppose you’re training a robot to recognize grasps. If you only feed in positions of fingers, that might not capture which fingers are contacting the object. You might invent features like “distance between gripper tips” or “force sensor differences” because you know they matter. Even with deep learning automating “feature extraction,” we often need to architect networks with convolution or attention exactly because we know something about our inputs. Domingos’s lesson: feature engineering may not be glamorous, but it is crucial; there really isn’t a substitute for careful domain-informed design. When in Doubt, More Data Wins. Feature engineering is one pillar, and data scale is another. Domingos’s ninth lesson is succinct: “More data beats a cleverer algorithm.” Suppose you’ve done your best to craft features and you still can’t get sufficient accuracy. Your choices are to invent a fancier learning algorithm or gather more training examples. Domingos argues that in most real projects, the latter is often the quicker path to improvement. A simple, “dumb” model with a vast amount of data often outperforms a sophisticated model trained on a smaller dataset. Since ML’s core is to let data do the heavy lifting, this should not be surprising. We see this all around us: many state-of-the-art applications grew simply by leveraging massive datasets (e.g. large search logs, internet images, gameplay traces), whereas inventing slightly better model families often yields diminishing returns by comparison. That said, more data is not magically free. Domingos points out an important shift in ML history: time and data have swapped roles as a bottleneck. In the 1980s, training data were scarce and collecting examples was the hard part. Today, for many applications, data is abundant – the limiting resource is computation time. Gigabytes of data sit unused because there isn’t enough CPU or GPU time to process them. Intriguingly, this paradox means that even though more data in principle allows learning hugely complex models, in practice we often don’t exploit it fully, because training those complex models is too slow. The result is that practitioners often resort to simpler models (or subsampled datasets) to get results in a reasonable time, even though more data could in theory justify more complex models. In robotics, data-collection can be expensive (hardware and time), but simulators are helping reduce that cost in many domains. Still, if you have the data bandwidth, collecting more trials or synthetic examples usually does more than fiddling with a new network architecture. Domingos’s advice is a reminder: we shouldn’t overlook data augmentation, transfer learning, or simply logging more trials. And where possible, we should invest in faster training algorithms and infrastructure (distributed learning, GPUs, optimized code) so that data can be fully put to use. Finally, Domingos notes that the payoff of new algorithms is surprisingly small partly because, in the regime of large data, to first order, many reasonable algorithms behave similarly. In a high-data regime, the differences between a clever algorithm and a naïve one often shrink. This can be seen today in many ML benchmarks: simple architectures (with enough tuning and data) often rival more complex ones. The practical rule-of-thumb stands firm: if accuracy is lacking, collect more data. Ensembling: Many Heads Are Better Than One. Long before deep learning popularized ensembling via dropout or multi-head networks, Domingos’s tenth lesson already preached the virtue of combining models: “Learn many models, not just one.” Historically, researchers had allegiances to “their favorite” algorithm and tried to out-design others within that one paradigm. But over time, empirical studies made it clear that no single algorithm dominates all tasks, and that combining models across paradigms often yields much better results. This led to the now-standard practice of ensemble methods. Instead of picking the single best model from your experiments, you aggregate several. The simplest example is bagging: resample your training data (bootstrap samples), train a model on each sample, and average their outputs. This usually greatly reduces variance. Boosting is another: it trains models sequentially, each time re-weighting examples that previous models got wrong, and then combines them. Domingos writes that if you “combine many variations [of models], the results are better – often much better – and at little extra effort for the user”. By now, ensemble methods (bagging, boosting, stacking, random forests, etc.) are staples in the ML toolkit. A dramatic illustration he gives is the Netflix prize from 2009. Teams competed to improve movie recommendation accuracy, and what ultimately won was not a single model, but ensembles of dozens or even hundreds of models, including combinations of strategies from multiple teams. The champion’s solution was literally hundreds of models voting together. In general, ensembles capture the intuition that different models have different biases; by averaging them we can smooth out individual errors. One might wonder if this ensembling idea is somehow the same as a principled Bayesian average. Domingos addresses this: Bayesian Model Averaging (BMA) is, in theory, the “optimal” way to average hypotheses by weighting each model by its posterior probability. But in practice, BMA often collapses to essentially picking the single best model, because it tends to put almost all weight on the highest-scoring hypothesis. In contrast, standard ensembles intentionally spread the weights around (for example, bagging votes roughly evenly). Domingos concludes that while BMA is theoretically appealing, it usually doesn’t buy you much; simple ensembles are what practitioners reliably use. For robotics and perception, ensembles appear everywhere. In navigation, one might fuse the outputs of multiple sensor predictors. In control, ensembles of policies or Q-functions can give more robust decisions. In vision, epochs of CNNs often ensemble multiple trained networks to eke out better accuracy. The lesson is clear: don’t be afraid to put many models “in the loop.” If you have a collection of reasonably good predictors, combining them usually pushes you further. Since training and model execution are fast compared to system safety demands, the slight extra cost of an ensemble is typically worth the increased robustness. Occam’s Razor, Revisited. A central dogma of science is Occam’s Razor: “entities should not be multiplied beyond necessity.” In machine learning, this is often interpreted to mean: given two models with equal training accuracy, the simpler one will generalize better. This is a tempting rule, and it appears in many courses and papers. But Domingos’s eleventh lesson is a caution: “Simplicity does not imply accuracy.”. He observes that there are many counterexamples to a blanket Occam assumption. For instance, the ensemble methods just discussed can yield highly complex models (many trees, many parameters) that generalize extremely well. He notes, “the generalization error of a boosted ensemble continues to improve by adding classifiers even after the training error has reached zero”. In other words, a more complex ensemble (with zero training error already achieved) still manages to reduce test error by combining models. Likewise, Support Vector Machines can have an infinite number of implicit parameters (via kernels) yet often avoid overfitting in practice. Even certain mathematical functions can “memorize” any labels with only a handful of parameters, defying any simplistic parameter-count view. More formally, the No Free Lunch theorems imply that we cannot universally claim a simpler model will generalize better on every problem. Simplicity is context-dependent. Domingos suggests adopting a more refined view: complexity should be thought of in terms of the size of the hypothesis space, not just the count of parameters. This is related to information-theoretic ideas, where a “simpler” model is one that can be encoded with a shorter description. In fact, one justification for preferring a shorter hypothesis is that it literally takes fewer bits to specify, which loosely ties to traditional sample-complexity bounds. Crucially, Domingos concludes that the virtue of simplicity is not because it magically guarantees lower error – but because simplicity is a virtue in itself. Simpler hypotheses are easier to inspect, more interpretable, and often more robust to noise by coincidence. Engineers often prefer simpler solutions for practical reasons (cheaper computations, easier to maintain). Occam’s razor applied naively as a guarantee of better performance is unwarranted. In practical terms, this means we shouldn’t automatically discard a model just because it’s “too complex” if it continues to improve validation results. With adequate regularization and data, even large neural networks or complex ensembles can generalize well in practice (something reinforced by modern deep learning “double descent” observations). Instead, use simplicity as a tie-breaker, not a law. If two models perform equally on validation, prefer the simpler one for clarity; but if a complex model is still doing better on held-out data, go for it. The student and veteran alike should recognize: complex models can work – in fact, they often do – as long as we control overfitting through data and validation. Representable vs. Learnable. Domingos’s twelfth lesson drills further into the nature of model classes: “Representable does not imply learnable.” Many machine learning textbooks proudly note that neural networks, decision trees, or other rich representations can, in theory, model any function to arbitrary accuracy – they are universal approximators. But Domingos warns that this theoretical flexibility can be misleading. Just because a function can be represented by some model in your class does not mean your learning algorithm can actually find it given realistic data and compute time. As he points out, decision trees can represent any Boolean function, but a standard decision-tree learner cannot possibly learn a tree that is larger (more leaves) than the number of training examples. More generally, representing a function might require exponentially many basis features or a very complex structure, which your learner can’t search. Even if the representation is capable, practical learning algorithms may get stuck in local optima or run out of resources. Domingos emphasizes that with finite data, finite time, and finite memory, standard learners only explore a tiny subset of what is theoretically representable. The main point: always ask “Can it be learned?”, not just “Can it be represented?”. For example, a recurrent neural network can represent a complex logic with loops, but training it via gradient descent on limited sequences might never converge to that logic. A random forest can, in principle, approximate any function given enough trees, but if the trees become too large relative to data, overfitting and computational infeasibility block learning. Different learners have different effective “basins” of what they can find. So in practice, we often try several different representations and learning methods to see which can actually capture the target behavior from the data we have. For robotics, this lesson resonates with modern practice: we frequently start with one model family, then swap to another if learning stalls. Perhaps a Gaussian process fails to learn an inverse kinematics relation from sparse data; then we try a neural network or an ensemble. Sometimes we even compromise: using an approximate model to initialize a more flexible one. But we should not assume that because a hypothesis is in our model class, it’s learnable. Computational limitations (time and memory) and optimization difficulty can prevent learnability. In short, we should remain humble about claims of “universal” power and focus on what our actual learning algorithms can achieve in practice. Correlation vs. Causation. The final lesson Domingos gives is a high-level reminder: “Correlation does not imply causation”. This might sound obvious, but it’s critical in ML systems that make predictions which will be used for decisions. Supervised learning algorithms fundamentally pick up associations present in the data, but these associations are often used in practice as if they were causal effects. Consider a classic data-mining anecdote: if beer and diapers are often bought on the same shopping trip, a naive predictive model might suggest stocking beer near diapers (or discounting diapers when beer is on sale) to boost sales. That inference implicitly treats a statistical correlation as a causal hook. But perhaps the true cause is something else (e.g., a certain demographic buys both). Without a controlled experiment, we can’t be sure the proximity will cause diaper purchases to rise. Domingos points out that most ML today deals with observational data (we just watch what happens) rather than experimental data (where we actively intervene). Learning from observational data can suggest potential causal links, but it doesn’t prove them. He notes that some ML methods (like certain graphical-model techniques) can extract limited causal information from observational data if strong assumptions hold [19], but generally their applicability is restricted. The safe stance is this: treat correlations as clues or hypotheses, not definitive guides to action. If possible, the strongest guarantee is to obtain experimental or randomized data. In many domains, including robotics, this translates to actively testing the model’s predictions through interventions. For instance, if your reinforcement learning model predicts that pushing a button causes a robot to lift an object, the ultimate proof is to actually try it in the real world (or in a carefully randomized simulator) and see the effect. Domingos urges us: “if you can obtain experimental data (for example by randomly assigning visitors to different versions of a website), then by all means do so”. In robotics, whenever we can control for confounding factors (e.g., running A/B trials of different control laws in simulation or hardware), we should. Why does this matter for a machine learning practitioner? Because many times we want our models to not only predict outcomes but to guide decisions. If we treat a purely predictive model as if it were a causal model, we might take actions expecting certain effects and be disappointed. Advanced topics like causal inference and counterfactual reasoning are gaining attention in ML (e.g., causal discovery, off-policy evaluation) precisely to address this gap. The takeaway is to remain aware of it: our standard ML pipelines learn patterns of what happened, and we typically interpret them as proxies for what will happen if we do something. When possible, supplement learning with actual experimentation, or at least be cautious about causal claims from correlational data. Conclusion: The Wisdom of Practice. Domingos’s twelve lessons read like a distilled checklist for any ML project. He nicely acknowledges that this sort of wisdom is complementary to formal theory and textbooks – it’s the folklore of what really works in practice. In the decade since he wrote this, the community has continued to validate these ideas. Deep learning has brought new techniques, but we see the same themes: representation (network architecture) and evaluation (loss function) are still major design choices, and optimization (stochastic gradient descent) is just one piece of the puzzle. Modern methods for regularization (dropout, data augmentation) are just sophisticated tools to avoid overfitting. The enormous success of massive data-driven models (e.g. large language models, deep vision networks) is yet another example that with enough data you can do wonders, but the fundamental need for strong priors (like larger model capacity and structure) remains. For our audience of roboticists and ML researchers, one final note: robotics brings its own twists to these lessons. The general principles apply, but we often have to be more creative in gathering data (simulators, self-play, transfer from human data), more careful in feature design (ensuring sensor modalities feed into the model meaningfully), and aware of the sharp division between simulation and real-world (an environment shift that tests generalization in a visceral way). But the core is unchanged: test on held-out scenarios (like new terrains), embed physical knowledge into your hypotheses, guard against overfitting (to simulator peculiarities), combine models (ensembles of policies or perception modules), and remember that correlations in data are not the same as cause-and-effect in your experiments. Machine learning has matured, but Domingos’s folk wisdom still teaches us to be humble, to question our intuitions, and to iterate empirically. He ends by encouraging practitioners to explore further resources and to keep learning happily. We hope this episode helps reinforce these timeless lessons. Happy learning – and happy applying!. Sources: Domingos’s original article “A Few Useful Things to Know About Machine Learning” (Commun. ACM, 2012) was the primary source of these lessons. Citations above refer to specific passages of that article (via a transcription) to preserve accuracy.