skinny.

Latest / learning with yacine show

How to Generate Synthetic Pretraining Data? with Joël Niklaus from Hugginface

today we are taking a deeper look at of the secret behind LLM pretraining which is synthetic data pipelines with Joel Niklaus machine learning engineer at Hugging Face!joel an his team ran 90 controlled experiments and burned over a trillion tokens to figure out what actually makes good pretraining data and in huggingface fashion provided all of their findings/artifacts openly!

lots of things to learn from experiment like this and to be honest there is still so much more to do!

couple of useful links:

👉Joel Website: https://niklaus.ai👉 Twitter: https://x.com/joelniklaus👉 Paper : arXiv…

Log in to save or queue this episode.

The skinny

The skinny isn't ready yet — notes appear once the transcript is processed.