“I trained GPT-2 with full-batch gradient descent so you don’t have to” by shawnghu

LessWrong (30+ Karma) · Oct 15, 2026 · 3m

The impetus for doing this was provided in large part by Charles Renshaw-Whitman.

Learning dynamics people suggested to me that noise was essential in the training of neural networks. I thought this was kind of counterintuitive--- I usually naively operate from a model in which the pretraining data basically specifies the outcome by virtue of the samples all "pulling" on the loss landscape in some relatively uniform way which incentivizes converging to a point that generalizes well, but instead it seems to be the case that the actual dynamics of the optimization process itself have…