Latest / Embodied AI 101 / Attention Is All You Need – The Transformer Revolution
Transcript
- Attention Is All You Need – The Transformer Revolution. By mid-2017, advances in machine translation and other sequence tasks were running up against a familiar bottleneck. Models based on recurrent neural networks (RNNs) – such as LSTMs – had become the workhorse for language tasks, but they processed text strictly sequentially. Even after adding attention modules (as in Bahdanau et al. 2015) to let decoders “look back” at encoder states, the core computation remained step-by-step. As a Google Brain retrospective puts it, their best LSTM-based translator “translated sentences remarkably well,” but “every sentence had to be processed sequentially,” one token at a time, with “no GPU parallelization… the bottleneck was not memory or compute capacity, it was sequential dependency”. Alternative seq2seq models using convolutions (e.g. ConvS2S/ByteNet) sped up computation somewhat by operating in parallel across tokens, but even those required deep stacks or dilations to capture long-range context. By 2017 the field was convinced it needed a fundamentally different way to handle sequences that could relate distant words at once while fully exploiting parallel hardware. The 2017 paper “Attention Is All You Need” by Vaswani et al. answered this need with a radical idea: throw out recurrence and convolution entirely, and build the model purely from attention mechanisms. In the authors’ words, they “propose a new simple network architecture… based solely on attention mechanisms, dispensing with recurrence and convolutions entirely”. This Transformer architecture immediately surpassed prior models on translation tasks while training in a fraction of the time. As one retrospective summarizes: transforming the model from sequential to fully parallel was “not incremental. It was a discontinuity.” The Transformer achieved new state-of-art on major benchmarks and trained orders of magnitude faster. Crucially, it opened a new design space: models whose performance could reliably improve just by scaling up depth, data, or compute. In hindsight this one paper ignited the entire Large Language Model revolution – as one analysis notes, “everything that followed – BERT, GPT, PaLM, Claude, Gemini, [the] entire large language model revolution – traces its lineage directly to this paper.”. Why Attention, Why Now. To appreciate the Transformer’s contribution, it helps to recall what the field was doing just before it arrived. The dominant paradigm for tasks like translation, summarization, or dialogue was an encoder–decoder RNN (often with LSTM or GRU units). The encoder would read an input sentence token-by-token into a hidden state sequence, and the decoder would generate output tokens one at a time. Early seq2seq systems even compressed the whole input into a single fixed-length vector, which (as one engineer notes) was “catastrophically” limiting for long sentences. Bahdanau et al. (2015) famously added an attention mechanism so the decoder could dynamically focus on all encoder states instead of one fixed summary. That change dramatically improved quality – the model could, for example, learn to align and attend to the French word “tapis” when generating “mat” in “cat is on the mat”. But it didn’t solve the core issue: the computations were still sequential. Even with attention, an LSTM encoder must process word $t$ only after word $t-1$ is done, and similarly the decoder must unroll one token at a time. This inherently serial flow meant that GPU parallelism could hardly be utilized in the sequence dimension – decode and encode steps could not happen simultaneously across tokens. The upshot was a hardware bottleneck. Modern GPUs and TPUs thrive when they handle huge matrices in parallel, but RNNs keep the timestep dimension serial. As one Google blog put it: RNNs “process language sequentially… reading one word at a time… and can only determine [anaphora] after reading each word between them step by step.”. This sequential unfolding forced long-range dependencies in language (like connecting “bank” with “river” after several intervening words) to be carried through many time steps, making training difficult. Convolutional seq2seq models (e.g. Gehring et al., 2017) could run in parallel, but they suffered if the network wasn’t extremely deep: to relate two words far apart, information had to hop through multiple layers. In short, the field needed a way to model arbitrarily distant interactions in one shot, and to eliminate step-by-step processing. The key insight of Vaswani et al. was deceptively simple: attention is enough. Rather than relying on a hidden state chain or convolutional filters, the Transformer lets every token attend to every other token at once. In their words, they wondered: what if attention weren’t just a supplement to an RNN, but the entire computation? After testing this “attention-only” idea with the Google Brain team, the results were dramatic. One engineer writes: “the results were not marginal improvements – they were decisive.” On WMT’14 English-German translation, their pure-attention model jumped to 28.4 BLEU (over 2 BLEU higher than then-state-of-art, including ensembles), and hit 41 BLEU on English-French – all while training far faster. They aptly titled the work Attention Is All You Need – the name itself heralded that attention would replace the decade of sequential RNN/CNN paradigms. The Transformer Architecture. At its core, the Transformer is an encoder–decoder architecture like previous seq2seq models, but every component is based on attention or pointwise operations. (Importantly, “attention” here is not the single-query attention of Bahdanau – it is the transformer’s richer mechanism described below.) Both the encoder and decoder are stacks of layers (in the base model, $N=6$ each). Each encoder layer contains two sub-blocks in sequence: (1) a multi-head self-attention sub-layer, followed by (2) a position-wise feed-forward network. Each of these sub-layers has a residual (skip) connection around it and is followed by layer normalization. The decoder side is similar but has three sub-layers per layer: first a masked self-attention to restrict each position to only attend to earlier outputs, then a multi-head cross-attention that attends to all encoder outputs (the “encoder-decoder attention”), and finally another feed-forward block. All attention operations project input tokens into query/key/value vectors and compute weighted sums – there are no convolutions or recurrences anywhere. A central building block is the scaled dot-product attention. Given a sequence of tokens, each token is first mapped (via learned linear projections) into three vectors: a query $\mathbf{q}$, a key $\mathbf{k}$, and a value $\mathbf{v}$. To compute attention for a particular token, the model takes that token’s query and computes the dot product with every token’s key. These dot-products (scaled by $1/\sqrt{d_k}$) determine weights via a softmax. Intuitively, each token “asks a question” (query) and matches it against all “key” vectors to decide which tokens to pay attention to; the softmax output is then used to mix the value vectors. Crucially this single self-attention step directly relates each token to every other, no matter how far apart they were in the sequence. The authors note that the scaling factor $1/\sqrt{d_k}$ is important – without it the raw dot-products could become very large in magnitude, pushing the softmax into saturation and making training unstable. The Transformer further refines this idea with multi-head attention. Instead of one attention computation, the model runs $h$ separate attention “heads” in parallel. Concretely, the input embeddings of dimension $d_{model}$ are linearly projected $h$ ways to produce $h$ sets of $(Q,K,V)$ triples (each projection with its own parameters). For a base model ($d_{model}=512$) and $h=8$ heads, each head works in a subspace of size 64 ($512/8$). Each head independently does its dot-product attention, yielding $h$ output streams, which are then concatenated and linearly projected back to $d_{model}$. Multi-head attention lets the model attend jointly to information from different representational “subspaces” at different positions. In practice, after training one often sees that different heads specialize: some capture syntax, others attend to adjacent words, others to long-range referents. And critically, using $h$ smaller heads costs roughly the same compute as one single head of full size, since the projection splits the work. In addition to attention, each layer has a per-position feed-forward network. This is simply a two-layer MLP (with ReLU) applied to each token’s vector independently. For example, in the base Transformer each token’s 512-dimensional representation is first lifted up to 2048 dims and then back down to 512. This expansion-contraction gives the model extra capacity to process information at each position. Empirically, these feed-forward sub-layers act like per-token “memory” or computation, and later analyses show they even store factual knowledge. One bookkeeping issue is that self-attention by itself is permutation-invariant: it treats tokens as a bag, not a sequence. To give the model a sense of word order, Vaswani et al. add fixed positional encodings to the input embeddings at the bottoms of the stacks. Concretely, each position $pos$ gets a sinusoidal $d$-dimensional vector (different frequencies across dimensions) which is added to the token embedding vector. These sinusoidal encodings allow the model to potentially learn relative offsets (since $\sin(pos+k)$ can be expressed in terms of $\sin(pos)$ by trig identities). (Other later Transformer variants tried learned or alternative positional schemes, but the original paper’s fixed sinusoids were sufficient to capture order with negligible extra parameters.). Finally, each sub-layer (attention or feed-forward) is wrapped in leftover/residual connections and followed by layer normalization. That is, if the sub-layer function is $f$ and its input is $x$, the layer outputs $\text{LayerNorm}(x + f(x))$. These techniques are essential: they stabilize the network so that even 6 layers (and in bigger models dozens or hundreds of layers) can be trained without gradient issues. In fact, one explanation notes that “layer normalization stabilizes the activations at each layer”, and along with residuals it makes deep Transformer stacks possible. Putting it all together, the Transformer encoder takes each input token (typically first converted to a learned word embedding), adds its positional encoding, and then passes the sequence through the stack of identical encoder layers. Each layer computes multi-head self-attention over its input, adds a skip connection, feeds it through the feed-forward network, adds skip again, and normalizes. The decoder works similarly, except that in the masked self-attention sub-layer each position can only attend to earlier positions (to maintain autoregressive generation), and there is an extra multi-head attention over the final encoder outputs to fuse in source information. Training and Performance. With this architecture in hand, Vaswani et al. trained their Transformer on the standard machine translation benchmarks and witnessed immediate gains. On the WMT 2014 English–German task, their single Transformer model scored about 28.4 BLEU, an improvement of more than 2 BLEU over the previous best (which often used ensembles of multiple models). On the WMT 2014 English–French task it reached roughly 41 BLEU – again establishing a new state-of-the-art for a single model. These were exacting tests: the Transformer had to translate long, complex sentences, and it did so more accurately than any model before. Notably, however, it achieved these results with much less training time than prior methods. Because attention can be parallelized across all tokens, the Transformer fully tapped GPU/TPU hardware. The paper reports that training was run on 8 GPUs for just 3.5 days and that this was “a small fraction of the training costs of the best models from the literature”. In practical terms, experiments found the Transformer training up to an order-of-magnitude faster than comparable RNN/CNN approaches. As a Google blog summarizing the results put it: the Transformer “requires less computation to train and is a much better fit for modern hardware, speeding up training by up to an order of magnitude”. Another commentary notes that in practice their model was about eight times faster to train than the previous system, while still ousting old results. In short, transformer models could be scaled far larger without prohibitive cost—a key factor that would later enable the enormous LLMs of 2020–2023. The Transformer was also remarkably general. Although introduced via machine translation, the authors explicitly tested it on other tasks. They applied it to English constituency parsing (a syntactic structure task) under both high-data and low-data conditions. Even there, the all-attention model performed strongly, suggesting it could “generalize well to other tasks” beyond translation. Subsequent work swiftly vindicated this: within a year, every top NLP model – for text classification, question answering, summarization, and countless other tasks – was built on the Transformer block. To summarize the architectural gains: the Transformer allows every token to attend to every other token in one step, generates token representations in parallel, and maintains positional information via encodings. This unlocked both higher accuracy and vastly faster training. One guide to its impact notes that the main changes were: the ability for the network to “pay attention to all the words at once”, much faster parallel training, preserving word order via positional encodings, and easy scaling that “powered GPT, BERT, and today’s foundation models”. Transforming Language Models. The success of the Transformer was quickly amplified by the explosion of large-scale pretraining. In 2018, researchers took the Transformer encoder (or decoder) and pre-trained it on massive text corpora before fine-tuning or prompting for tasks. A seminal example is BERT (Devlin et al. 2018). BERT’s title comes from “Bidirectional Encoder Representations from Transformers”, underscoring immediately that it uses a Transformer stack (encoder-only) for its core. Indeed, its abstract describes BERT simply as “a new language representation model (that) is a bidirectional transformer-based model”. By pre-training on unsupervised objectives, BERT achieved new state-of-art on a host of benchmarks (GLUE, SQuAD, etc.) with minimal task-specific changes, and it did so with the same attention blocks introduced by Vaswani. Almost concurrently, the GPT-family of models (Radford et al. 2018–2020) showed that a large Transformer decoder could be trained autoregressively to generate text of astonishing fluency. GPT-3 (Brown et al. 2020) scaled a transformer to 175 billion parameters and could answer questions, write code, and translate text without explicit fine-tuning. In the GPT-3 paper, the authors even remark that their success leverages “the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT”. In other words, they directly attribute GPT’s capabilities to the Transformer design. Today, every modern language model – from BERT and GPT3 to Google’s PaLM and Anthropic’s Claude – is built on transformer layers. It has truly become the de facto standard for NLP. The Transformer’s dominance spread beyond text. In computer vision, Researchers showed that a pure Transformer could process images effectively. Notably, Dosovitskiy et al. (2020) proposed the Vision Transformer (ViT). They chunk images into patches, treat each patch position like a “token” in a sequence, and feed them through a Transformer encoder. ViT matched or surpassed leading convolutional networks on image classification benchmarks (especially after large-scale pre-training). The ViT authors remark that the Transformer has already “become the de-facto standard for NLP tasks,” and they demonstrate that it requires far fewer training resources to reach competitive image accuracy compared to complicated CNNs. Likewise, in speech and audio processing, transformer-based models (like wav2vec, HuBERT, Speech-Transformer, etc.) have supplanted older HMM/RNN pipelines, benefiting from self-attention’s ability to capture long-range structure in audio. As one 2021 survey notes, “Transformers have achieved great success in many AI fields, such as natural language processing, computer vision, and audio processing.” The architecture that was born for translation has proven equally adept at sequence tasks of all kinds. Perhaps most strikingly for an embodied AI audience, transformers are now uniting multiple modalities and control. DeepMind’s Gato agent (Reed et al. 2022) is a Transformer “generalist” that can caption images, chat via text, play Atari games, and even control a real robot arm – all with one single network and the same weights. In Gato, different tasks simply produce different token types (text tokens, pixels, joint torques, etc.) and a single Transformer seq2seq model learns them all. As their abstract notes, “the same network with the same weights can … play Atari, caption images, chat, stack blocks with a real robot arm and much more”, deciding on each context whether to output text or action tokens. Likewise, in reinforcement learning research, the Decision Transformer (Chen et al., NeurIPS 2021) reframes RL as a sequence modeling problem. It feeds past states, actions, and desired returns into a causal multilayer Transformer and simply predicts the next action. Remarkably, this matches or exceeds previous RL baselines in many games. Even here, the authors remark that they are explicitly tapping into “the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT”. In short, the pure-attention Transformer has become a backbone, not only for language but for vision, speech, robotics, and even control. Impact and Significance. It is now widely acknowledged that “Attention Is All You Need” marked a pivot in AI. The shift away from recurrence unlocked the full parallelism of modern hardware and made it feasible to train truly large models. Noam Shazeer (co-author) has noted that by removing RNNs, they “replaced an entire computational paradigm with a single mechanism”. The impact on model scaling was immediate: by 2020–21 the community had summarily abandoned basic LSTM encoders and decoders in almost all NLP and sequence modeling work. One measure of the paper’s influence is how universally its architecture has been adopted. For example, Transformers power every large language model in production today. Named LLMs like BERT and GPT explicitly draw on those 2017 designs, as do modern translation systems (e.g. Google’s own state-of-art models) and virtually all text-generation engines. Even in industrial settings, chatbots, summarizers, and translation services are built on transformer platforms (HuggingFace, TensorFlow, etc.). The mechanism has become the baseline building block – whenever someone says “train a language model”, the starting point is usually a transformer stack. Beyond academia, the Transformer architecture achieved mainstream recognition. Media outlets now refer to it as a “groundbreaking neural network architecture” introduced in 2017. It is credited with leading to newer systems like Google’s Bard and OpenAI’s ChatGPT, which their inventors explicitly describe as trillions-parameter Transformers. Guidance policies and fine-tuning (RLHF) notwithstanding, fundamentally ChatGPT is just a scaled-up transformer decoder. Remarkably, a relatively short 11-page paper changed the course of AI in just a few years. From a technical standpoint, the Transformer also sparked a flurry of innovations. Almost immediately, researchers began studying its strengths and limitations. The standard Transformer’s $O(n^2)$ time and memory cost for full attention was pointed out (especially for very long sequences). This led to many “efficient transformer” variants (Sparse Transformer, Longformer, Reformer, Linformer, etc.) that tweak the attention pattern to handle longer contexts or range. Other extensions added recurrence shortcuts, better positional encodings, or mixture-of-experts layers. But crucially, all of these retain the core attention mechanism. One might say that new research largely revolves around refining and extending Vaswani’s idea for different regimes, rather than overturning it. The significance of “Attention Is All You Need” thus lies in both the idea and the ecosystem it enabled. Its single-minded focus on attention provided a remarkably flexible and powerful new primitive for sequence processing. That, combined with the deep experimentation and clear writing in the paper, allowed others to adopt it rapidly. The original authors even showed a bit of that flexibility by applying the model to syntax parsing. Other groups took it to web-scale: by 2018 pretraining Transformers became the norm, and within a few years thousands of papers and products were citing Vaswani et al. Today we see Transformers as the workhorse of nearly all advanced AI systems. From robot-language interfaces to multi-modal agents, the Transformer is the latent engine behind the scenes. In summary, “Attention Is All You Need” fundamentally changed how we think about sequence learning. It replaced the sequential RNN paradigm with a fully-attentional one that is simple to train in parallel yet extraordinarily powerful. It delivered immediate quality gains in NLP tasks and, more importantly, it opened the door to unprecedented scale. Transformers have become the “foundation” of modern AI: every large text model is built on them, and they are increasingly used in vision, audio, and embodied contexts. As one analysis observed, with a hint of inevitability, everything from today’s GPT-style chatbots to multi-task agents can be traced back to this breakthrough. The legacy of the 2017 Transformer paper is nothing less than a new era of AI, one that will continue to unfold for years to come. Key Contributions (summary): The Transformer introduced a novel sequence model based entirely on self-attention, removing the need for recurrence or convolution. Its scaled dot-product attention and multi-head mechanism enabled direct modeling of all pairwise token relationships in parallel. Along with positional encodings and residual/normalization layers, this architecture achieved state-of-the-art translation with much faster training. Importantly, it showed that model quality could scale with larger networks and data, seeding the growth of today’s huge pretrained models. Its impact cannot be overstated: nearly all current large language and vision models owe their architecture to the Transformer framework.