Stanford CME295 Transformers and LLMs | Autumn 2026 | Lecture 4 - Reinforcement Learning with LLMs

Stanford Online: University Courses & Lectures · Oct 21, 2026 · 1h 50m

For more information about Stanford’s graduate programs, visit: https://online.stanford.edu/graduate-educationTo follow along with the course schedule and syllabus, visit: https://cme295.stanford.edu/syllabus/Chapters:00:00:00 Introduction00:06:35 RL setup00:09:32 Reward design00:12:23 Policy gradient00:17:12 Derivation of the gradient00:23:50 REINFORCE00:25:51 Importance sampling ratio00:33:22 Challenges of policy gradients00:40:34 Proximal Policy Optimization (PPO)00:42:49 Value function and advantage00:51:16 Policy ratio 00:53:03 Ratio clipping00:57:45 KL Penalty00:59:48 Connection with…