Latest / Embodied AI 101 / Deep Residual Learning for Image Recognition (ResNet)
Transcript
- Deep Residual Learning for Image Recognition (ResNet). In 2015, Kaiming He and colleagues at Microsoft Research shook up computer vision by showing how to train ultra-deep convolutional networks far beyond previous limits. Their CVPR 2016 paper “Deep Residual Learning for Image Recognition” introduced the simple but powerful idea of skip connections or residual connections. In practice this means each few-layer block in the network learns a residual function that is added to its input, instead of trying to fit a completely new transformation from scratch. This reformulation – letting the layers model F(x) = H(x) – x with the output H(x) + x – made it dramatically easier to optimize very deep nets. By using these residual blocks, He et al. were able to train networks hundreds of layers deep (ResNet-152!) on ImageNet. Their 152-layer ResNet achieved just 3.57% top-5 error and won ILSVRC 2015 classification, and they won multiple tasks in COCO detection and segmentation as well. To this day, ResNet remains a standard backbone in nearly every vision pipeline and a cornerstone of deep learning practice. In this episode we’ll dive into the ResNet paper’s key contributions, its methodology, and its impact on modern deep learning. We’ll discuss why training deeper networks was surprisingly problematic, how residual connections solve that, and why the ResNet architecture has reshaped visual recognition research. Along the way we’ll cite the original work and related materials, and highlight how ResNet has influenced thousands of subsequent papers and applications. The Challenge of Going Deeper. By the mid-2010s, convolutional neural networks had already revolutionized vision. Starting from AlexNet (2012) and the VGG nets (2014), researchers kept pushing the depth of CNNs higher and higher. Intuitively, stacking more layers should let a network learn richer hierarchical features. In fact, prior work had shown that deeper networks tend to perform better on large datasets like ImageNet. However, a stubborn obstacle remained: simply adding more layers often made training harder, not easier. As He et al. point out, deep networks originally suffered from the classic vanishing/exploding gradient problem, which can choke off learning in very deep stacks. Improved weight initialization and techniques like Batch Normalization had largely solved the vanishing gradient issue for tens of layers, so by 2015 networks of 20–30 layers were trainable. But another vexing phenomenon appeared: the degradation problem. As you keep adding layers beyond a certain depth, the network’s training error actually went up instead of down. In other words, a 30-layer network might train and converge nicely, but a 50-layer network of the same design could end up with higher training loss. This was surprising because, in principle, a deeper model should be able to mimic a shallower one if needed. In fact, one could construct a solution to the deeper model by copying the shallower model’s layers and inserting identity mappings for the extra layers. By construction that solution would have no higher error than the original network. Yet in practice the gradient algorithm failed to find such trivial solutions. As He et al. observed: “When deeper networks are able to start converging, a degradation problem has been exposed: … adding more layers to a suitably deep model leads to higher training error”. They stress that this isn’t ordinary overfitting – the test error also degraded, indicating an optimization issue. In short, the community was puzzled: why should a deeper network be any harder to train than a shallower one, if an identity shortcut exists? He et al. hypothesized that the standard way of parameterizing the new layers was suboptimal. Rather than forcing each set of layers to learn an entire transformation H(x), perhaps it would be easier to learn the change (residual) needed, with the identity path carrying through the base information. This insight led directly to the Residual Network (ResNet) formulation. The Residual Learning Framework. The core idea of ResNet is deceptively simple: each residual block computes a function of the form. output = F(x) + x,. where x is the input to the block, and F(x) is the residual function learned by the intermediate layers. Equivalently, if the desired mapping of that block is H(x), the network explicitly learns F(x) = H(x) – x, and then outputs F(x) + x. In the original ResNet, each block has two or three convolutional layers that compute F(x). A direct skip (identity) connection then adds x to the block’s output. In their words, the authors “explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions”. If the identity mapping is actually optimal (i.e. if H(x)=x), this formulation makes it easy for the residual F(x) to go to zero; otherwise, F(x) only needs to learn the difference from x. This residual connection has two important consequences. First, it eases the flow of gradients during backpropagation. The skip path provides a direct route for the gradient to flow backward to earlier layers, mitigating vanishing issues in practice. Second, it ensures that the output of each block is at least as good as its input. Intuitively, if the new layers didn’t help reduce error, the optimal residual would be zero and the block would simply act as an identity (since x + 0 = x). In contrast, a plain deep network (without skips) must learn each new transformation fully from scratch, which can lead to the training difficulties observed. Hence, the authors hypothesize that learning the residual mapping is easier. They report: “We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping”, and provide empirical evidence supporting this. Crucially, these skip connections add no extra parameters or complexity. In the simplest case (when the input and output of the block have the same dimensions), the skip is just the identity map. As the paper states: “Identity shortcut connections add neither extra parameter nor computational complexity”. Even when the number of channels changes (for example, when downsampling or increasing feature maps), a small 1×1 convolution can be used on the skip path to match dimensions. Either way, the skip-connection architecture can be trained end-to-end with standard SGD just as a plain network would. From an implementation standpoint, one only needs to add an elementwise addition of the input to the block’s output. The authors note this can be done “easily … using common libraries (e.g. Caffe) without modifying the present ones”. In code frameworks today (TensorFlow, PyTorch, etc.), residual blocks are a common, built-in module. The take-home of this methodology is that a stack of residual blocks can grow in depth while ensuring stable training. When He et al. put this into practice, the results were dramatic. With residual learning, networks rapidly converged even at hundreds of layers of depth. ResNet Architecture: Building Blocks and Variants. He et al. applied residual learning to convolutional networks in a fairly straightforward way. The overall network still follows the standard Conv–ReLU–Pooling paradigm, but with residual blocks in place of some sequences of layers. In their experiments they explored several depths. For example, ResNet-18 and ResNet-34 use a so-called basic block consisting of two 3×3 convolutional layers with ReLU and batch-normalization. The skip adds the block’s input to its output (so the block input and output have the same spatial and channel dimensions). These are relatively “shallow” variants of ResNet that still outperform earlier architectures. For very deep networks (e.g. ResNet-50, ResNet-101, ResNet-152), the authors used a bottleneck block. A bottleneck block has three layers: a 1×1 conv reduces channel dimensionality, a 3×3 conv does the main computation, and another 1×1 conv restores the channel dimension. BatchNorm and ReLU are applied between each conv. In diagrams, it looks like “Conv1×1 → Conv3×3 → Conv1×1” in each block. The shortcut connection still adds the original input to the final output of these three layers. The bottleneck design keeps the computational cost manageable despite the depth. Later code libraries even note that the stride (for downsampling) can be placed on the central 3×3 layer or on the first 1×1 layer; these minor tweaks (sometimes called ResNet-v1.5) improve accuracy. The paper’s main architectures were:. ResNet-18 and ResNet-34 (with basic 2-layer blocks). ResNet-50, ResNet-101, ResNet-152 (with bottleneck 3-layer blocks). These are famously the “ResNet family” names. Today, standard deep learning libraries ship pretrained models of precisely these variants. For example, PyTorch’s model zoo lets you create , , , , with a single line of code, all pretrained on ImageNet. The code snippet on PyTorch’s site explicitly lists those models:. With these building blocks, training follows the usual regimen. The ResNets were trained on ImageNet, using SGD with momentum and data augmentation (random crops, flips, etc.). Batch Normalization was used after each convolution, following Ioffe & Szegedy’s 2015 insight, which helped stabilize training. Because of the skip connections, even the 152-layer network could be trained without special tricks beyond a slightly smaller learning rate schedule for the deepest version. The authors note that their 152-layer ResNet has lower complexity (in FLOPs and parameters) than VGG nets, even though it is “8× deeper”. Concretely, VGG-19 has on the order of 140 million parameters, while ResNet-152 has only around 60 million. So ResNet attained deeper models more efficiently. In summary, the architecture innovation was straightforward: take a standard CNN backbone and add identity shortcuts every few layers. No exotic layers or gates are needed. This simplicity is part of ResNet’s appeal: it generalizes easily to new problems and models. Empirical Results: Benchmarks and Evidence. The ResNet paper backs up its claims with extensive experiments. On the major benchmarks of the day, the performance gains were striking. ImageNet Classification (ILSVRC 2015): The flagship result was on the ImageNet 2012 classification challenge (ILSVRC). The winning entry of 2015 was an ensemble of deep Residual Networks. The single-model ResNet-152 achieved a top-5 error of 3.57% on the ImageNet validation set – a huge jump over the previous best models. (For comparison, ResNet-50 was about 4.7% error, ResNet-101 around 4.5%.) This was the new state of the art, and in fact won the ILSVRC 2015 classification task. The authors report: “An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task”. The ensemble combined multiple independently trained ResNets to push error down even further, but even a single ResNet-152 set the record. Crucially, they also show that these residual networks scale smoothly with depth. A plain (“shallow”) network of 34 layers was beaten by the residual version of 34 layers; similarly ResNet-50 beat VGG-19; ResNet-101 and ResNet-152 did even better, confirming that increased depth gave consistent improvement when using residual connections. In their CIFAR-10 experiments (a smaller natural image dataset), they even trained ResNets of 100 and 1000 layers. Remarkably, even a 1000-layer network could be trained and improved accuracy (albeit with fewer datapoints than ImageNet). This drives home that residual learning fundamentally fixes the depth bottleneck: with it, “the depth of representations is of central importance” and more layers indeed bring gains. CIFAR-10 Experiments: On CIFAR-10, the authors provide plots showing how plain nets (without skips) suffer in training error as depth grows, whereas ResNets do not. For example, a 56-layer plain CNN had higher training loss than a 20-layer plain CNN. But a 56-layer residual net achieved lower loss than its 20-layer counterpart. These experiments (charts omitted here) confirm that deep plain nets hit a degradation in optimization that ResNets avoid. They trained thin ResNets of depths 20, 32, 44, 56 layers and showed each deeper model improved test accuracy. Even on tiny images, residual connections clarify the optimization path. Object Detection and Segmentation (COCO, ILSVRC): The deep representations from ResNet proved hugely useful beyond classification. The authors plugged ResNets into object detection and segmentation pipelines. For example, using a ResNet backbone in Fast R-CNN / Faster R-CNN frameworks yielded substantial gains. On the MS COCO detection benchmark, they report a 28% relative improvement (compared to what would be expected using shallower nets) simply by virtue of the deeper ResNet features. In fact, the same ResNet models won all major tasks in the ILSVRC & COCO 2015 competitions: detection and localization on ImageNet, and detection and segmentation on COCO. The paper emphasizes: “Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.”. In other words, ResNet was not just about pushing scratch accuracy on one challenge — it became the all-purpose feature backbone that swept multiple vision contests. These experimental successes had a clear message: residual connections enable very deep networks that are not only trainable but also substantially more accurate. The authors’ tagline reads: “These residual networks are easier to optimize, and can gain accuracy from considerably increased depth”. That turned out to be exactly right. Winning ImageNet with a 152-layer net and dominating behind-the-scenes tasks proved that adding depth (control it with skips) was a winning strategy. Impact and Legacy in Deep Learning. The influence of the ResNet paper has been enormous. Almost overnight, residual networks became the default backbone for numerous vision tasks and spurred a wave of follow-up research. A few highlights of this impact:. Standard Backbone: Today, ResNet (especially ResNet-50 and ResNet-101) is a standard component in libraries and papers. If you pick a modern object detector like Faster R-CNN, RetinaNet, or DETR, it often uses a ResNet pretrained on ImageNet under the hood. If you choose a segmentation model like U-Net or DeepLab, variants often incorporate ResNet for their encoder. The PyTorch and TensorFlow model zoos all have ResNets with pretrained weights ready for fine-tuning. Even outside vision, residual connections appear in other domains (for example, the Transformer’s “add shortcut and layer norm” structure echoes this philosophy). Inspiration for New Architectures: ResNets inspired many architectural innovations. The idea of skip connections is now ubiquitous. For example, ResNeXt (Xie et al., 2017) built on ResNet by adding grouped convolutions (“cardinality”) in the residual blocks. Wide ResNet (Zagoruyko & Komodakis, 2016) kept residual blocks but made them wider instead of deeper. DenseNet (Huang et al., 2017) took the skip idea even further by connecting each layer to every other layer. Autonomous adaptation like Stochastic Depth and Shake-Shake regularization were developed in the ResNet context. Even MobileNet-v2’s “inverted residual with linear bottleneck” is a relative of the ResNet block. In short, any new CNN architecture since 2015 almost always includes some form of residual or skip connection, thanks to this paper. Theoretical and Optimization Follow-ups: Research into why ResNets work so well has been prolific. Follow-up papers (e.g. “Identity Mappings in Deep Residual Networks”, 2016) refined the ordering of activations and batchnorm to further ease gradient flow. Theoretical work on linearizing deep nets often uses ResNet structures as motivation. Many analyses attribute the success to improved gradient propagation – the skip acts as a near-identity Jacobian, keeping gradients from vanishing. Other work views ResNet as creating an ensemble of shallower networks (Veit et al., 2016), or as enabling iterative refinement in supervision (Greff et al., 2017). Whatever the interpretation, the residual formalism has become a key lesson in deep learning: building networks that preserve information across many layers simplifies training. Citations and Influence: By citation count, “Deep Residual Learning” is among the top deep learning papers ever published (well into the tens of thousands of citations). It launched Kaiming He to prominence, and the model name “ResNet-50” is now standard terminology. It heavily influenced the thinking of practitioners: after ResNet, the default was no longer to ask “How do I flatten these layers?” but rather “How can I connect them?”. Continued Use in Practice: Even with the advent of vision transformers and other novel models, ResNets remain widely used. In cases where large amounts of data and compute are available, some applications now choose transformer-style backbones. But for many practical systems, ResNets or ResNet-derived models are often the most efficient choice. Many real-world robotics and embodied AI projects still start with a ResNet pretrained on images for scene understanding or action perception. In embodied tasks, for example, it is common to use a frozen ResNet-50 to extract image features for reinforcement learning or planning modules. The HuggingFace model hub alone lists hundreds of ResNet-based models fine-tuned on various tasks. To summarize the significance: ResNet did for deep learning what AlexNet did a few years earlier. It showed that with the right architectural design, much deeper networks could be trained and would outperform previous models. It solved a fundamental obstacle to network depth and thus enabled a new era of very deep models. As the authors themselves conclude, the success on multiple tasks “shows that the Residual Learning principle is generic, and we expect that it is applicable in other vision and non-vision tasks”. That prediction came true in spades. Concluding Thoughts. When “Deep Residual Learning” came out, it was as if the community discovered a missing piece of the puzzle. The key lesson is elegantly simple: let layers learn incremental changes on top of their inputs. By reframing the mapping as F(x) + x, training becomes easier and deeper models become feasible. This insight has proven extremely robust and generative. For software engineers and researchers in robotics or embodied AI, ResNet is a workhorse. Recognizing objects from camera images, segmenting scenes, or providing feature encodings for downstream modules often relies on ResNet or its variants. Even when transformer-based or multitask models are used, they frequently incorporate components learned from (or inspired by) ResNet architectures. In the broader arc of deep learning, the ResNet paper stands as a classic: fresh, impactful, and practically valuable. Its key contributions – residual functions, identity shortcuts, enabling deeper nets – are now part of the basic toolkit. Its methodology was straightforward to implement but profound in effect. And its impact endures, both in wide adoption and in inspiring countless successors. Sources: We have drawn on the original ResNet paper and related documentation in this discussion. He et al. (2016) explicitly note that they “reformulate the layers as learning residual functions” and provide empirical evidence that “these residual networks are easier to optimize, and can gain accuracy from considerably increased depth”. In their experiments, a ResNet-152 achieved a 3.57% ImageNet error (winning ILSVRC 2015) and excellent performance on COCO tasks. They also trained networks up to 1000 layers on CIFAR-10. The identity skip connection “adds neither extra parameter nor computational complexity” and makes the network trainable as-is. These findings are well-documented and have been cited by hundreds of subsequent works in vision and robotics.