Stanford CME295 Transformers and LLMs | Autumn 2026 | Lecture 2 - Large Language Models

Stanford Online: University Courses & Lectures · Oct 25, 2026 · 1h 49m

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-educationTo follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/Chapters:00:00:00 Introduction00:04:47 BERT and encoder-only models00:13:47 GPT and decoder-only models00:16:10 Encoder-decoder versus decoder-only00:19:23 Scaling laws00:26:49 Large Language Model definition00:29:05 Mixture of experts (MoE)00:33:11 MoE-based LLMs00:36:03 Load balancing loss00:45:36 Position embeddings01:01:58 Rotary position embeddings (RoPE)01:09:28 Sliding Window…