Stanford CME295 Transformers and LLMs | Autumn 2025 | Lecture 5 - LLM tuning

Stanford Online: University Courses & Lectures · Oct 17, 2026 · 1h 54m

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-educationOctober 31, 2025This lecture covers: • Preference tuning• RLHF overview• Reward modeling• RL approaches (PPO and variants)• DPOTo follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/Chapters:00:00:00 Introduction00:04:50 Preference tuning00:11:31 Data collection00:17:43 RLHF overview00:28:24 Reward model00:29:46 Bradley-Terry formulation00:46:02 Reinforcement learning00:53:54 PPO00:57:56 Advantage, value function01:03:39 PPO variants: Clip…