Latest / Embodied AI 101 / The Llama 3 Herd of Models

Transcript

The Llama 3 Herd of Models · Embodied AI 101

• The Llama 3 Herd of Models. Meta’s LLaMA series has become a flagship open‐source foundation model line, and the new LLaMA 3 family represents the latest leap. As reported by Meta, LLaMA 3 is a “herd” of Transformer models covering multiple sizes and capabilities. Their largest model is a dense decoder-only Transformer with 405 billion parameters and an unprecedented 128,000-token context window. According to the LLaMA 3 paper, this family “natively support[s] multilinguality, coding, reasoning, and tool usage”. In extensive evaluations, LLaMA 3 reportedly “delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks”. Meta is releasing LLaMA 3 under an open‐weight license: the 405B model is available in both pretrained (base) form and an instruction-tuned version, and even a complementary safety model LLaMA Guard 3 (8 B) for input/output filtering is published alongside it. The LLaMA 3 paper also describes prototype extensions into vision and speech: by integrating image, video, and audio inputs through adapter modules, the authors achieve accuracy on par with task-specific state-of-the-art models. (Those multimodal versions are still under development and not broadly released yet.). In this deep dive we’ll unpack the architecture, data, training, and evaluation of the LLaMA 3 models, and explain how Meta pushed scale, tokenization, and fine-tuning to achieve its performance. We’ll also look at the compositional approach to multimodal extensions (images, video, speech) that the LLaMA team explored, as well as the real-world impact of this open model series. Throughout, we’ll compare LLaMA 3 to prior models (like LLaMA 2 and GPT-4) and highlight what’s new and significant about this release. LLaMA 3: Models and Architecture. LLaMA 3 is not just one model but a family. The name “herd” hints at multiple variants. Meta’s release strategy came in waves: initially (April 2024) the first LLaMA 3 release included an 8 B and a 70 B version, both with an 8,192-token context and optimized for code, reasoning, and multilingual tasks. Then in July 2024 (LLaMA 3.1), Meta introduced the 405 B flagship and extended the context window to 128K tokens for all models. (A September 2024 update, LLaMA 3.2, added compact models like 1 B and 3 B for on-device use, as well as vision-enabled variants, but the core LLaMA 3.1 architecture underlies all these.). All LLaMA 3 models use the same basic decoder-only Transformer blocks: fast attention with gating, feed-forward layers, and normalization. As with LLaMA 2, the feed-forward activation is SwiGLU and normalization is RMSNorm, rather than the original LayerNorm. Position is encoded with rotary embeddings (RoPE), but Meta dramatically increased the RoPE frequency to handle longer context. Specifically, LLaMA 3 sets the RoPE base frequency theta=500,000 (instead of 10,000 in LLaMA 2), so that the periodicity of positional encoding is stretched to 128K tokens. The three main LLaMA 3 sizes differ in depth and width. For example, the 405B model has 126 layers and a hidden dimension of 16,384, with 128 attention heads (16 query heads per key-value head). The 70B model uses 80 layers at 8,192 width, and the 8B model has 32 layers at 4,096 width. In all cases the feed-forward (MLP) dimension is about 3.5× the hidden size (53,248 for the 405B, 28,672 for 70B, 14,336 for 8B). The attention uses grouped-query attention (GQA): only 8 key-value heads are used even as the number of query heads scales. (In the 405B model there are 128 query heads but still only 8 key-value heads, so each key head is shared by 16 queries.) This massively shrinks the memory needed for the key/value cache during generation, speeding up inference without hurting output quality. In fact, LLaMA 3 extended GQA to all model sizes – even the smallest 8B model uses it – whereas LLaMA 2 only used GQA for its 70B variant. A few other architectural details: LLaMA 3 uses a large token vocabulary of 128,000 BPE tokens (via tiktoken), quadruple the 32K vocabulary of LLaMA 2. This bigger vocabulary encodes common words, symbols, and even characters in fewer tokens; Meta reports it compresses English text about 15% more efficiently than the old square-root subword scheme. All models use RMSNorm pre-layer-normalization (“PreNorm”) for attention and FFN blocks. Meta explicitly chose a dense (non-MoE) Transformer for LLaMA 3: unlike some recent gigantic models that use mixture-of-experts tricks, here every parameter is active for every token. The authors note that dense Transformers offered better training stability and simpler deployment across hardware platforms, at the cost of higher inference cost per parameter. (They actually tried MoE as a comparison, but found the overhead and complexity weren’t worth it.). In summary, LLaMA 3’s architecture is a scaled-up version of LLaMA 2’s decoder Transformer with a few key upgrades: more layers and width for the big models, much bigger vocabulary, pervasive grouped-query attention, and extremely large context capacity via overhauled RoPE. These building blocks – SwiGLU, RMSNorm, GQA, etc – are now almost standard in modern LLMs, but LLaMA 3 pushes each of them to new extreme scales. For instance, by adopting 128K context on all models, LLaMA 3 goes far beyond the 8K limit of typical open models and even the 32K of GPT-4o. Pretraining at Scale. The most striking thing about LLaMA 3 is the scale of its pretraining. Meta amassed an enormous dataset and compute budget to train these models. According to the authors, LLaMA 3 was pretrained on over 15 trillion tokens of text, roughly seven times the data used for LLaMA 2. The corpus was drawn from cleaned public web sources, code repositories, and specialized content. Impressively, about 5% of it (∼800 million tokens) was non-English text spanning 30+ languages, making LLaMA 3 significantly more multilingual than earlier variants. (In fact, the July 2024 LLaMA 3.1 release explicitly noted support for eight major languages beyond English.) Meta’s filtering and curation pipeline was extensive: they used multiple rounds of heuristic and learned filters (e.g. classifier models trained via prior LLaMA 2 annotations) to weed out low-quality text, pornographic or toxic content, and code-like noise. They even trained DistilRoBERTa classifiers on human-labeled data to rank pages by quality and identify math/coding content. After filtering and de-duplication at both the document and line level, the pretraining mix was roughly 50% “general knowledge” web text, 25% math/reasoning content, 17% code, and 8% truly multilingual text. To process this vast data, Meta deployed massive GPU clusters and new training infrastructure. Benchmarks on their 8B model indicate each GPU delivered over 400 TFLOPS for fused attention/FFN operations. Crucially, they built two custom 24,000-GPU clusters (likely H100-based) with an improved training stack that monitored hardware health, detected errors and silent corruption, pipelined checkpoints, and redesigned storage to maximize throughput. With these advancements, they achieved an effective utilization of over 95% of GPU time during the LLaMA 3 runs – a threefold efficiency gain compared to the LLaMA 2 project. In other words, very few GPU cycles were wasted on stragglers or failures. The flagship 405B model was trained on a whooping 16,384 NVIDIA H100 80GB GPUs. The total compute for that run was about (3.8\times10^{25}) floating-point operations (FLOPs) and it consumed roughly 39.3 million GPU-hours. In practical terms, the run took around 54 days of wall-clock time. Over those weeks, the cluster endured hundreds of task interruptions (over 400 total), mostly from hardware glitches – faulty GPUs, memory errors, network issues, etc. Meta automated rapid recovery and checkpointing, so despite these breakdowns the net training loss was only about approx 10% downtime, i.e. >90% effective training time. Metas’ engineering notes also mention a 4D parallelism strategy to fit this model. They used tensor parallelism (TP) (splitting matrix multiplications across GPUs), pipeline parallelism (PP) (assigning different layer chunks to different devices), data parallelism (DP) (replicating model copies and splitting the batch), and a clever context parallelism (CP) for long sequences. The context parallelism cuts a very long input sequence into pieces across GPUs to reduce memory, allowing the 128K-token context to be handled without running out of memory. All of these simultaneously helped scale to the 126-layer, 405B model. Another interesting training detail: LLaMA 3 models were trained on a target sequence length of 8,192 tokens (the context length of LLaMA 3.0 models) using document-level masking (not letting the model attend across document boundaries). Despite training at this fixed length, Meta found a surprising phenomenon: the models did not hit diminishing returns at Chinchilla’s famous data regime optimum. In fact, their smaller 8B model saw continued log-linear gains well past the Chinchilla quota for that size (~200 billion tokens) when they used an order of magnitude more (15 trillion). Put differently, far from wasting compute, training on such huge data gave the smaller models extra punch; presumably this insight motivated the enormous 15T token corpus and the capacity of LLaMA 3 parallel to it. Fine-Tuning and Alignment. Once the dense models were pretrained, Meta applied multiple stages of fine-tuning to prepare them for interactive use. First was supervised fine-tuning (SFT) on instruction and dialogue tasks. All LLaMA 3 models (8B, 70B, 405B) were tuned on curated datasets of instruction-response pairs, using over 10 million human-annotated examples. This included a wide variety of formats (chat-style, questions, code, etc.), leveraging both public data and newly crowdsourced examples. The goal was to make the models behave well as chat assistants after the raw pretraining. Next came aligning with human preferences. The team employed both rejection sampling and Reinforcement Learning from Human Feedback (RLHF) variants. Concretely, they ran at least one round of policy optimization: generating multiple candidates for a prompt and ranking them via crowdworkers, then using those rankings in PPO and Direct Preference Optimization (DPO) to nudge the model towards better outputs. Notably, Meta’s blog comments that preference training had an outsized effect on reasoning and coding capabilities. One quip from Meta engineers was that sometimes the pretrained model “knew” the correct reasoning trace but couldn’t select the right answer. Training on preference data essentially taught it which answer to pick. For instance, LLaMA 3 showed big gains on math and logic questions after the PPO/DPO step. They iterated this process: one SFT round, then preference tuning, then possibly another answer ranking pass, etc. Each phase involved careful vetting of the training data and prompts. Meta reports that much of the performance improvement came from curating high-quality instruction prompts and doing strict quality checks on the human labels. These rigorous alignment procedures help explain why the instruction-tuned LLaMA 3 achieves top-tier performance in evaluations. Performance and Benchmarks. How good are the LLaMA 3 models? The paper’s headline claim is that LLaMA 3 “delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks”. In absence of full benchmark tables in the arXiv abstract, Meta’s statements and blogs give some context. The April 2024 blog for LLaMA 3 already said the initial 8B/70B models had state-of-the-art performance on industry benchmarks, especially in reasoning. And the July report reiterates that across many categories LLaMA 3 rivals GPT-4. (Meta does not typically release raw scores, but does emphasize that these open models match closed models in key areas.). We do know some specifics in qualitative terms: for example, on coding challenges and math puzzles, LLaMA 3 beats Llama 2 by a wide margin. The enhanced reasoning is attributed to the extra data and preference training. On multilingual tasks, the added non-English data likely improves performance on Chinese, Spanish, Arabic, etc. On broad tasks like factual Q&A or writing, LLaMA 3 with 405B generally appears competitive with other top LLM APIs. In fact, the LLaMA team notes that with its 128K context, LLaMA 3 can handle extremely long documents (hundreds of pages) – something few models support at all. In short, all indications are that LLaMA 3 sets new records for an open‐weight model family, and it benchmarks within shouting distance of GPT-4 on many leaderboards. (Formal head-to-head comparisons are an active area of research as third parties start using LLaMA 3.). One concrete demonstration of quality is that Meta openly compares LLaMA 3 results to GPT-4 on similar tasks, often finding parity. For example, the research abstract explicitly claims “comparable quality to. GPT-4 on a plethora of tasks”. Social evidence also comes from independent testing: some QA evaluations posted online show LLaMA 3 405B matching or even exceeding GPT-4 answers on reasoning benchmarks. Of course, these models still make mistakes and struggle with tricky bias issues, but the raw language and reasoning ability is competitive. On technical grounds, LLaMA 3 benefits from architectural efficiencies. Its larger vocabulary and GQA boost throughput, so it can give similar answers faster (using fewer tokens) than older models. Meta’s benchmarks on the tokenizer suggest a ~15% gain in tokens-per-input over LLaMA 2. And because they optimized the software stack, single-accelerator performance (FLOPS/GPU) is extremely high, meaning the model can infer quickly on optimized hardware. Meta even collaborated with hardware partners: LLaMA 3 runs are supported on major cloud platforms and edge devices via TorchTune and other tooling, broadening real-world use. In terms of specific tasks, some highlights from related LLaMA 3 analyses include: very high scores on code generation benchmarks (substantiating the “coding” claim), excellent accuracy on math (MATH, GSM8K), strong performance on GLUE/SuperGLUE tasks, and top performance on the new instruction-following evaluations Meta designed (covering advice, classification, Q&A, coding, etc.). The instruction alignment seems to pay off especially for dialogue and reasoning prompts. We’ll have to wait for independent leaderboards, but it’s safe to say that LLaMA 3 comes out as one of the most capable open models yet. Multimodal Extensions: Images, Video, and Speech. Beyond text, LLaMA 3 was designed with an eye toward multimodality. Rather than training a single monolithic text+vision+audio model (which can be very complex and expensive), Meta took a compositional approach: they built adapters on top of the text model. In their words, this lets them develop vision/speech capabilities in parallel without retraining the core LLM or confusing the token stream between modalities. The result is a suite of prototype models for image understanding, video, and speech tasks – all reusing the frozen LLaMA backbone. Vision and Image Understanding. For vision-language, Meta built vision adapters. The core idea: attach a pre-trained image encoder to the Transformer by adding cross-attention blocks that inject visual features. Specifically, LLaMA 3’s vision system has three parts:. Image encoder: a standard large vision Transformer (ViT). Meta used a ViT-H/14 variant with 630M parameters, initialized from CLIP-like pretraining on 2.5 billion image-text pairs. This encoder turns an image into a sequence of image tokens. Image adapter: Once the image is tokenized by the encoder, a small set of learned layers project those tokens into a form the LLM can attend to. (This might be a simple linear layer or a few transformer cross-attention layers.). Video adapter (for video, see next section). For static images, the procedure is: feed the image through the ViT encoder and the image adapter, then let the LLaMA model attend to the resulting tokens (usually via added cross-attention layers in the Transformer). Crucially, during training of this vision pipeline, the original LLaMA text weights remain frozen. Only the image encoder and adapter weights are fine-tuned on vision-language data. This ensures that adding visual capability does not degrade the model’s text-only performance. Meta trained these vision adapters on pairs of (image, caption/question). They used large image-text datasets, applying their usual cleaning (or CLIP-style filtering) to align images with relevant text. After this training, the LLaMA 3 “Vision” models can accept an image plus a text prompt (e.g. “Describe this picture” or “Answer the question about the image:.”) and generate a text answer. The results are impressive even with surprisingly small models. In the September 2024 LLaMA 3.2 preview, Meta introduced 11B and 90B “Vision” models by attaching a vision adapter to the 11B and 90B text models. Evaluations showed the 11B-Vision LLaMA outperforming Anthropic’s Claude 3 Haiku on image understanding tasks, and even comparable to a smaller variant of GPT-4 (GPT-4o mini). In other words, just 11B of LLaMA plus a good vision encoder was enough to match models that were trained end-to-end on multimodal data. This highlights the effectiveness of the compositional approach: the text model doesn’t have to “re-learn” text tasks from scratch, it simply “listens” to the image tokens. Video Understanding. For video, the pipeline is similar but with an extra temporal component. A video is first decomposed into sampled frames. In LLaMA 3’s case, they uniformly subsample up to 64 frames from the video clip for each query. Each frame is run through the same ViT-H/14 image encoder and image adapter. Then video-specific modules come into play: a “temporal aggregator” and extra cross-attention layers, which we can call the video adapter. The temporal aggregator (implemented as a Perceiver Resampler) takes groups of consecutive frame tokens and compresses them: for example, it might combine every 32 frames into one summary token. This reduces the sequence length so that the model can handle the longer input efficiently. Then additional cross-attention layers are inserted periodically into the Transformer to allow the model to attend over time and fuse the aggregated frame features. Meta’s design freezes the original LLaMA text weights and the base image encoder during video adapter training. They only train the new temporal aggregation and cross-attention parameters. (A technical detail: for LLaMA 3, the video-specific parameters amount to about 0.6B for the 7B model and 4.6B for the 70B model. That is, these adapters are relatively small compared to the host LLM.). The upshot is a model that can perform video-question-answering tasks zero-shot. In the LLaMA 3 experiments, the 8B/70B text models with their (small) video adapters achieved very competitive results on a range of video understanding benchmarks. In fact, without any special fine-tuning for most tasks, the LLaMA 3 models often matched or beat other big multimodal models on long-range video questions. For example, they report that LLaMA 3 exhibited especially strong temporal reasoning: on the “Perception Test” (designed to probe complex time-based reasoning) the video‐enabled LLaMA 3 was the best performer of all tested models. Even on long video QA like ActivityNet-QA, where the model might see only one frame every 3 seconds, LLaMA 3 still scored “strong results” by some margin. These results suggest that a small video adapter on LLaMA 3 can extract a lot of temporal understanding. Speech and Audio. Meta also experimented with adding speech capabilities via a similar compositional trick. The LLaMA 3 speech interface is built around a speech encoder plus a small set of “speech adapter” layers, connected to the LLaMA core. In practice, a waveform is fed into a speech encoder (trained separately, e.g. a Wav2Vec-like model), producing a sequence of audio tokens or features. These are then processed by adapter layers and injected into the LLaMA Transformer, again via cross-attention. Interestingly, LLaMA 3’s speech system has modes. By default, with no special prompt, the model acts like a general spoken dialogue assistant: you can speak to it, and it will respond as if in a normal conversation. But Meta also implemented system prompts that switch the mode to tasks like Automatic Speech Recognition (ASR) or Automatic Speech Translation (AST). For example, if you start with a prompt like “Transcribe this audio:” then the LLaMA model will output a text transcription of the speech. If you use a prompt like “Translate this speech to English:”, it attempts speech-to-text translation. The clever part is that all this is done without fine-tuning the LLM each time: the prompts and the adapters do the work. Crucially, the LLaMA 3 speech interface is multilingual. The team reports support for up to 34 languages in ASR/AST mode. It even allows you to interleave text and speech in the input: for instance you could ask it a question in text and then clarify with a voice snippet, and LLaMA 3 will handle both. This makes it a very flexible voice assistant. Meta also built a streaming text-to-speech (TTS) pipeline on top of LLaMA 3’s outputs (using a separate neural TTS model), so the entire loop (audio in → LLM → audio out) can operate with low latency. The training for the speech modules also involved scale. Meta used about 15 million hours of unlabeled audio to pretrain the speech encoder (for self-supervised representation learning), and then tens of thousands of hours of transcribed and translated speech to fine-tune for ASR/AST. They even generated synthetic speech and transcripts using the LLaMA 3 model itself to augment low-resource languages. All of that is detailed in the paper’s appendices. For our purposes, the key takeaway is that by the end, LLaMA 3 can do live multi-lingual speech conversations, transcription, and translation all in one framework. That’s a striking capability for a model family born as a text LLM. In summary, Meta’s “compositional” design for LLaMA 3 means they didn’t attempt to train a single beast that natively takes text, image, video, and audio together. Instead, they built specialized front ends for each new medium and used system prompts to steer the model. This avoids huge joint training costs and preserves the text performance. The results are quite competitive: LLaMA 3 line-of-sight matches state-of-the-art on image classification/QA, video Q&A, and even speech tasks, all while using the same underlying language model. Deployment, Safety, and Impact. One of Meta’s big goals is open access. LLaMA 3 is released with weights and code, following LLaMA 2’s open ethos. The main 8B and 70B base/instruct models went public in April 2024, and the 405B model weights became available by August 2024. These models carry the Meta LLaMA 3 license (which is research-friendly and non-commercial). Along with the model weights, Meta published evaluation code, tokenizer specs (128K vocabulary), and the LLaMA Guard 3 safety model (an 8B filter that screens content). They also released the training recipes in TorchTune (Meta’s new PyTorch-based LLM library) and integration examples for inference. TorchTune provides fine-tuning recipes and kernels that other researchers can use to reproduce or adapt the LLaMA 3 training process. Meta partnered with many platforms – Hugging Face, AWS, Google, Azure, etc. – to provide cloud instances of LLaMA 3, and with hardware vendors to optimize inference (including INT8 quantization). Security and ethical use have been emphasized, too. LLaMA 3 comes with updated trust-and-safety tools: besides LLaMA Guard 3 for output safety, Meta introduced Code Shield (which scans generated code for vulnerabilities at inference time) and an updated CyberSec Eval to test for malicious code output. In short, they want the ecosystem to use LLaMA 3 responsibly. Since launch, the real-world impact of the LLaMA line has been massive. Independent coverage (and Meta’s own reports) indicate that the LLaMA models in aggregate have been downloaded tremendously. By April 2025, Meta announced that the entire LLaMA family had surpassed 1.2 billion downloads. (For perspective, they had reported 650 million by early Dec 2024, reaching 1.0 billion by March 2025.) This means millions of developers around the world are running LLaMA models. Onstage at a developer conference, Meta’s CPO noted “thousands of developers contributing tens of thousands of derivative models” from LLaMA, each downloaded and used frequently. In effect, LLaMA 3 plus its relatives have become the most widely used open LLMs in the ecosystem. Meta has also integrated LLaMA3 tech into its own products. For example, the Meta AI assistant (a ChatGPT-like conversational app) is powered by LLaMA 3, and Meta reports on the order of a billion users interacting with that assistant. Meanwhile, major companies (Ferrari, NASA, etc.) and open-source projects have started building on LLaMA 3. Many third-party inference providers (OpenRouter, NVIDIA NIM, etc.) offer APIs for the new models, making them easy to experiment with. The innovation has not stopped. In late 2024 and 2025 we’ve seen LLaMA 3.2 and even early LLaMA 4 teaser releases. These extend the herd with new “small and medium” sizes (1B, 3B, 11B, 90B in vision form) and improved architectures. But the LLaMA 3 Herd of Models forms the core: the flagship 405B patent and its smaller siblings. By open-sourcing these powerful models, Meta hopes to spur research and applications across the board, from academic labs to industry products. Looking back, LLaMA 3 is a culmination of years of scale-up from LLaMA 1 (2023) and LLaMA 2 (2023). It demonstrates that large open models can rival closed offerings. The key contributions are the sheer scale of data/hardware, the smart engineering (like 128K context, GQA at all scales), and the thoughtful modular design for multimodality. 30 minutes of deep analysis might not cover every nuance, but in summary: LLaMA 3 represents a new milestone for open AI. Meta’s official “Herd” report provides the full technical details, and we’ve tried to distill the essence here for the embodied AI community. The era of multi-trillion-token, 100+ billion parameter open models is now upon us – and LLaMA 3 is right at the forefront. References: The above details are drawn from Meta’s official “LLaMA 3 Herd of Models” paper and accompanying blog posts, as well as analyses like the LLaMA 3 wiki and TechCrunch coverage. All the specific figures and components cited can be found in those sources. (See the inline citations for exact references.).