Latest / The Information Bottleneck
Pierre-Carl Langlais on Building Models from Data You Can Account For
Most labs build language models by scraping the web and filtering afterward. Pierre-Carl Langlais runs it the other way around. At Pleias, the French-German lab he co-founded, the models are built from data he can actually account for, which in practice means open and public-domain sources plus a lot of synthetic data the lab generates itself. It sounds like a self-imposed handicap. It mostly isn't. One of their models is a 600 million parameter system that runs live inside the Paris subway's monitoring pipeline.
We cover the SYNTH pretraining dataset and why he thinks "ethical data" has to…
The skinny
The skinny isn't ready yet — notes appear once the transcript is processed.