Latest / learning with yacine show / How to Generate Synthetic Pretraining Data? with Joël Niklaus from Hugginface
Transcript
- 0:00SPEAKER_01: Originally,
- 0:00SPEAKER_01: pre-training data for large language models was mostly built by scraping the web and
- 0:05SPEAKER_01: then filtering it down aggressively with quality classifiers.
- 0:08SPEAKER_01: Now, basically every recent flagship models like Nemotron 3,
- 0:13SPEAKER_01: Gwentry,
- 0:14SPEAKER_01: GLM, He's leaning heavily on adding synthetic data to their training recipes.
- 0:18SPEAKER_01: The way it works is that you take a web document,
- 0:21SPEAKER_01: you run it through a strong enough LM,
- 0:23SPEAKER_01: and ask it to produce a variant that keep the meaning but change the presentation.
- 0:27SPEAKER_01: Something like rewrite this as a tutorial,
- 0:30SPEAKER_01: Then you mix the synthetic data back into the real web data and you pre-train on
- 0:38SPEAKER_01: the mixture.
- 0:39SPEAKER_01: For example, NVIDIA rephrase about 2 trillion tokens this way for Nimetron CC.
- 0:44SPEAKER_01: The issue is that the literature on how to properly build such
- 0:47SPEAKER_01: a synthetic Pre-training pipeline is pretty sparse.
- 0:51SPEAKER_01: The frontier labs like OpenAI, Google,
- 0:53SPEAKER_01: Entropic,
- 0:53SPEAKER_01: and even the Chinese labs usually don't publish their training data recipes.
- 0:58SPEAKER_01: And the academic groups that would publish them usually don't have the compute
- 1:01SPEAKER_01: to run all the ablation In this video,
- 1:05SPEAKER_01: we're going to look at the amazing work from the team at Hugging Face
- 1:08SPEAKER_01: that actually ran a full ablation suite on the synthetic data pipeline.
- 1:12SPEAKER_01: We're talking about 90 rephrasing configurations,
- 1:15SPEAKER_01: 300 evaluation experiments,
- 1:18SPEAKER_01: There's a trillion token generated just for the ablation,
- 1:21SPEAKER_01: which total up to about 12 plus GPU years of compute.
- 1:24SPEAKER_01: We'll even have the chance of interviewing Joel Niklas,
- 1:27SPEAKER_01: who led the work into making this synthetic data pipeline playbook possible.
- 1:31SPEAKER_01: The core idea is that there are three axes you can vary
- 1:34SPEAKER_01: when you generate synthetic pre-turning data.
- 1:36SPEAKER_01: The prompt,
- 1:37SPEAKER_01: used to rephrase,
- 1:38SPEAKER_01: the generator model,
- 1:39SPEAKER_01: doing the rephrasing,
- 1:40SPEAKER_01: and the source data, plus whatever real data you mix it back in.
- 1:44SPEAKER_01: The result Hugging Face found was pretty counterintuitive.
- 1:49SPEAKER_01: For instance, the prompt is by far the biggest lever, and what work is mainly restructuring.
- 1:54SPEAKER_01: Meanwhile, the generator model barely mattered past the billion parameters.
- 1:58SPEAKER_01: The dataset,
- 2:00SPEAKER_01: every prompt template,
- 2:01SPEAKER_01: all the adlation data,
- 2:02SPEAKER_01: the generation framework,
- 2:04SPEAKER_01: and the full benchmark result are all open source,
- 2:07SPEAKER_01: which is You can check out right here their 90-page document on
- 2:11SPEAKER_01: the full methodology.
- 2:13SPEAKER_01: Speaking of fully open technical artifacts,
- 2:15SPEAKER_01: let's talk a bit about RC, the open-weight lab that is sponsoring this interview.
- 2:19SPEAKER_01: If you're familiar with the open AI model landscape,
- 2:21SPEAKER_01: you know it's pretty much Well,
- 2:24SPEAKER_01: RC is a pretty cool lab right here in North America
- 2:27SPEAKER_01: that not only open weight their model,
- 2:29SPEAKER_01: which you can download and run on whatever system you want,
- 2:32SPEAKER_01: but it's also very open about their training methodology and release a lot
- 2:35SPEAKER_01: of like their research artifact.
- 2:37SPEAKER_01: You can get their latest model on Open Router and if you're
- 2:39SPEAKER_01: into personal agent you can literally shove it into your Hermes setup.
- 2:41SPEAKER_01: You can also run it straight into
- 2:53SPEAKER_01: One fresh new thing that they recently released is Knack,
- 2:56SPEAKER_01: which is an open-source harness for long-running agentic work.
- 2:59SPEAKER_01: It's pretty useful for research that are complex but decomposable
- 3:03SPEAKER_01: with well-separated part and concrete acceptance criteria.
- 3:07SPEAKER_01: Stuff that you would do in like a machine learning experiment.
- 3:10SPEAKER_01: How it works is that you give your model in that harness a high-level objective,
- 3:13SPEAKER_01: some constraint stated upfront,
- 3:15SPEAKER_01: an acceptance criteria,
- 3:16SPEAKER_01: and then you let it roll on a research idea for days and dates.
- 3:19SPEAKER_01: The team at RC is actually building their next version of Trinity,
- 3:23SPEAKER_01: their flagship model using that.
- 3:25SPEAKER_01: You can try out this harness out there on GitHub or install the current Edge build
- 3:30SPEAKER_01: by checking their blog post.
- 3:31SPEAKER_01: Big thanks to RC for sponsoring this video, a very cool app to keep your eye on.
- 3:34SPEAKER_01: Let's now dive into the interview.
- 3:35SPEAKER_02: Yeah, my name is Joel.
- 3:37SPEAKER_02: Actually,
- 3:37SPEAKER_02: I started studying sports science and then switched to computer science
- 3:42SPEAKER_02: in my bachelor's.
- 3:43SPEAKER_02: Then I did specialization in data science in the master and then PhD in NLP.
- 3:49SPEAKER_02: The PhD was focused on legal NLP, like mainly evaluation.
- 3:55SPEAKER_02: The project was about anonymization of people inside Swiss court decisions.
- 4:00SPEAKER_02: And then after the PhD,
- 4:02SPEAKER_02: I worked for Harvey for a year as a research scientist,
- 4:07SPEAKER_02: made a benchmark for translating Swiss legal documents in five languages and worked
- 4:14SPEAKER_02: on improving retrieval systems.
- 4:16SPEAKER_02: And then in August last year,
- 4:18SPEAKER_02: I joined Hugging Face and my first project was the Synthetic Data Playbook.
- 4:23SPEAKER_01: That's a good first project.
- 4:25SPEAKER_01: This is pretty good.
- 4:27SPEAKER_01: I think what's nice here is that it ties up to a lot of your work
- 4:30SPEAKER_01: that you've done previously, which is really cool.
- 4:33SPEAKER_01: You work in like you did a PhD, you work in academia and industry.
- 4:38SPEAKER_01: Like now you're doing kind of a kind of mix of the two with like ugly face.
- 4:43SPEAKER_01: What's your overall general philosophy with research problem?
- 4:48SPEAKER_01: I know you've done benchmark stuff.
- 4:50SPEAKER_01: You worked on various different projects.
- 4:53SPEAKER_01: But when you're tackling a research project that has at the edge of
- 5:00SPEAKER_01: the knowledge quality, what's your philosophy to tackle it?
- 5:04SPEAKER_02: Yeah, great question.
- 5:06SPEAKER_02: I think similar a bit to how startups operate in general,
- 5:10SPEAKER_02: I would say like if you're tackling a new research project,
- 5:13SPEAKER_02: you don't really know where it's going,
- 5:15SPEAKER_02: what's going to work and what's not going to work.
- 5:18SPEAKER_02: So in the beginning,
- 5:19SPEAKER_02: you want to explore as many different parts of the problem as fast as possible.
- 5:24SPEAKER_02: So for example,
- 5:25SPEAKER_02: in the Fine Phrase project,
- 5:28SPEAKER_02: a big issue was that I had to rephrase like 10 billion tokens just
- 5:34SPEAKER_02: to run one experiment and that even on the Huggy Face cluster, it takes a while.
- 5:40SPEAKER_02: So you can't really run that many experiments, right?
- 5:44SPEAKER_02: So you wanna figure out ways to like speed up The iterations as much as possible,
- 5:52SPEAKER_02: and then try to explore the tree of possible options as broadly as possible,
- 5:59SPEAKER_02: as early as possible,
- 6:01SPEAKER_02: so that you can then narrow in on the ones that sound promising.
- 6:05SPEAKER_02: I did some work on Monte Carlo Tree Search in my master's.
- 6:11SPEAKER_02: And I feel like, yeah,
- 6:13SPEAKER_02: maybe to be a bit nerdy about that,
- 6:17SPEAKER_02: I would say I would follow Monte Carlo Tree Search, where I first explore.
- 6:21SPEAKER_02: And then when I find one branch of the tree that looks promising,
- 6:27SPEAKER_02: I exploit until I have a project that is well-rounded and can be finished.
- 6:34SPEAKER_01: Yeah, same thought here.
- 6:36SPEAKER_01: Because I've done startup stuff for a while, research stuff also.
- 6:41SPEAKER_01: Same mental model I have.
- 6:43SPEAKER_01: I've done a lot of graph theory things in my research for the project that I've been doing.
- 6:51SPEAKER_01: And my view is that if you can just rephrase and rethink about your problem
- 6:56SPEAKER_01: in this way,
- 6:57SPEAKER_01: it's usually much easier to understand the scale of what you're trying to accomplish
- 7:02SPEAKER_01: and How useful it is to kind of go fast at the beginning,
- 7:06SPEAKER_01: even if like it's a bit messy and you're just trying to explore a bit the space.
- 7:11SPEAKER_01: But once you have something good that like fit your criteria,
- 7:14SPEAKER_01: like just going over there and getting like juicing up as much
- 7:17SPEAKER_01: as possible is usually like very, very good.
- 7:20SPEAKER_01: And then you're able to kind of get the good stuff.
- 7:23SPEAKER_01: I also see that you've done a lot of benchmark,
- 7:27SPEAKER_01: which is funny because I think you're the third or fourth person I'm interviewing
- 7:33SPEAKER_01: that has a heavy background in making benchmarks.
- 7:37SPEAKER_01: First off, can you talk about these different benchmarking work that you've done?
- 7:43SPEAKER_02: So one of the most recent ones that we made was, it's called Lexam.
- 7:48SPEAKER_02: So it's legal reasoning evaluation based on Law exams from the University of Zurich
- 7:55SPEAKER_02: in Switzerland.
- 7:55SPEAKER_02: And the new thing there was that we really evaluated like long form reasoning.
- 8:01SPEAKER_02: And that was like one of the first works that they test that in the legal domain.
- 8:05SPEAKER_02: So like really hard, really hard.
- 8:07SPEAKER_02: Provide long-form reasoning questions in the law domain.
- 8:11SPEAKER_02: Another one I've touched upon briefly before was called SwiltraBench,
- 8:15SPEAKER_02: where we benchmarked translation models on law translation and headnote translation
- 8:22SPEAKER_02: from Swiss court decisions, and then made some before on general...
- 8:28SPEAKER_02: Like back in the days in NLP when we looked at named entity recognition tasks
- 8:34SPEAKER_02: or text classification tasks for multilingual legal data sets.
- 8:39SPEAKER_01: In the ancient time, right?
- 8:41SPEAKER_01: Like the ancient time of machine learning, right?
- 8:44SPEAKER_01: 2021 or something like that?
- 8:47SPEAKER_01: Hey, this is crazy how like time in...
- 8:51SPEAKER_01: The field moved.
- 8:53SPEAKER_01: I had a question that I got from somebody that I was collaborating with.
- 8:58SPEAKER_01: It was mainly on manufacturing data with their company.
- 9:04SPEAKER_01: One thing that they found very hard is trying to get an internal benchmark for their own stuff.
- 9:14SPEAKER_01: What would you say is the first few steps?
- 9:16SPEAKER_01: Let's say you're working in a legal business or whatever it is, right?
- 9:22SPEAKER_01: What's the first few steps that you do in order to get started
- 9:25SPEAKER_01: and have this benchmark set up, this eval for yourself?
- 9:29SPEAKER_02: Yeah, I think especially like in the commercial setting,
- 9:32SPEAKER_02: it's very important that the tasks that you look at are like as close as possible
- 9:37SPEAKER_02: to what the system is actually going to do in the end, right?
- 9:43SPEAKER_02: And then diversity is really important.
- 9:45SPEAKER_02: I think these are probably the most, like the two things I would focus on first.
- 9:54SPEAKER_02: From the academic side, I always have a bias to making the benchmark as hard as possible.
- 9:59SPEAKER_02: Maybe in the commercial side,
- 10:02SPEAKER_02: you have a bit of a different incentive sometimes to show
- 10:06SPEAKER_02: that your system really performs well.
- 10:08SPEAKER_02: You don't want the hardest tasks in your benchmark.
- 10:12SPEAKER_02: Maybe more for an internal one to see where still the gaps are and where you can improve.
- 10:18SPEAKER_02: But for the academic side,
- 10:20SPEAKER_02: I think that's the most interesting part,
- 10:21SPEAKER_02: especially now that the methods develop so fast.
- 10:26SPEAKER_02: For example,
- 10:26SPEAKER_02: Lexam is almost saturated already and we only published it a bit over a year ago,
- 10:33SPEAKER_02: so it goes fast.
- 10:34SPEAKER_01: So you take like six months to do it and then saturate.
- 10:38SPEAKER_01: This is good.
- 10:39SPEAKER_01: Yeah, because also like I've worked with,
- 10:43SPEAKER_01: I am currently working with like a few clients and like every time we get into like
- 10:47SPEAKER_01: the internal evaluation part,
- 10:49SPEAKER_01: it's always like almost a challenge because like they never done That's in
- 10:54SPEAKER_01: a rigorous fashion that much.
- 10:56SPEAKER_01: And like it's hard for somebody from the outside to kind of like do it for them
- 11:00SPEAKER_01: because they have the most context right about like the problem set
- 11:04SPEAKER_01: and like what they actually want.
- 11:06SPEAKER_01: Right.
- 11:07SPEAKER_01: So they kind of have to help like us do it in some way.
- 11:11SPEAKER_01: But this is usually the part that is taking so long with your work
- 11:15SPEAKER_01: that you're doing right now with the synthetic data pipeline.
- 11:17SPEAKER_01: Do you see like a connection and impact for like benchmark making?
- 11:23SPEAKER_02: Interestingly,
- 11:23SPEAKER_02: I've just reviewed a paper for NeurIPS that suggested creating a benchmark as
- 11:29SPEAKER_02: a benchmark task itself, which I found really interesting.
- 11:34SPEAKER_02: Basically, they measured the LLM's ability of benchmark creation as a task.
- 11:41SPEAKER_02: So that's one interesting side.
- 11:45SPEAKER_02: And then on the legal side,
- 11:48SPEAKER_02: Harvey recently published the legal agent benchmark
- 11:53SPEAKER_02: that was entirely synthetically created.
- 11:57SPEAKER_02: In the legal domain,
- 11:58SPEAKER_02: we always have, like,
- 11:59SPEAKER_02: when we're talking about public documents,
- 12:02SPEAKER_02: like court decisions or legislation, then it's fine, right?
- 12:06SPEAKER_02: We can use those to create benchmarks.
- 12:10SPEAKER_02: But like most tasks that law firms care about, like they use client data, right?
- 12:17SPEAKER_02: And obviously client data, you cannot publish it.
- 12:20SPEAKER_02: So for that, synthetic data is also a great application.
- 12:24SPEAKER_02: And that's exactly what they did there.
- 12:26SPEAKER_01: Cool.
- 12:27SPEAKER_01: Cool.
- 12:27SPEAKER_01: So it's good.
- 12:28SPEAKER_01: Yeah.
- 12:28SPEAKER_01: So like,
- 12:29SPEAKER_01: can I use it as a,
- 12:30SPEAKER_01: a data augmentation mechanism or like from the get go,
- 12:33SPEAKER_01: just trying to optimize for like a few,
- 12:36SPEAKER_01: um, kind of criterion and then like the model will build this on that.
- 12:40SPEAKER_01: And then you evaluate basically like,
- 12:41SPEAKER_01: uh, how close to this criterion it is just like the rough idea.
- 12:45SPEAKER_02: Yeah, so basically they created the,
- 12:48SPEAKER_02: like the whole benchmark is completely synthetically generated.
- 12:51SPEAKER_02: So I think they had like some descriptions of how they,
- 12:57SPEAKER_02: like they want, like of what tasks they want to be present in the benchmark.
- 13:03SPEAKER_02: And then they had the models generate sample documents that would be used as inputs
- 13:08SPEAKER_02: for the agents to accomplish certain tasks.
- 13:12SPEAKER_02: And then they had like rubrics that are being used to create
- 13:17SPEAKER_00: the agent outputs.
- 13:25SPEAKER_02: Yeah, today what we'll cover first, why synthetic data in the first place, right?
- 13:31SPEAKER_02: A bit of a background about pre-training data in 2026.
- 13:35SPEAKER_02: Then how we structured the experiments.
- 13:39SPEAKER_02: So we looked at three axes mainly, the prompt, the generator model, and the source data.
- 13:45SPEAKER_02: Then we ran 333 experiments in total to figure out what moves the needle,
- 13:51SPEAKER_02: what leads to good pre-training data.
- 13:54SPEAKER_02: And then we did some analysis to satisfy our curiosity,
- 14:00SPEAKER_02: like why does it work, what issues are there still.
- 14:03SPEAKER_02: Then, as I mentioned already a bit,
- 14:07SPEAKER_02: running these experiments at scale is kind of expensive,
- 14:11SPEAKER_02: so you want to make sure that you really utilize the GPU as well,
- 14:16SPEAKER_02: and there's a few infrastructure considerations there.
- 14:20SPEAKER_02: And then finally,
- 14:22SPEAKER_02: we ran the final pipeline over 350
- 14:27SPEAKER_02: billion tokens of fine web edu data four times
- 14:31SPEAKER_02: for four prompts, which created this fine phrase data set of 486 billion tokens.
- 14:37SPEAKER_02: So why synthetic data in the first place?
- 14:42SPEAKER_02: Synthetic data is everywhere now,
- 14:44SPEAKER_02: so here we have an example for GLM,
- 14:50SPEAKER_02: but most recent flagship LLMs heavily use synthetic data like Mnemotron 3,
- 14:56SPEAKER_02: Qn3, 5.4, RC, Trinity for example.
- 15:02SPEAKER_02: NVIDIA even released a huge synthetic dataset called Mnemotron CC that we compare against.
- 15:09SPEAKER_02: CEII used synthetic data in mid-training for GLM 4.5 and small LM
- 15:14SPEAKER_02: from Huggy Face also used synthetic data a lot in post-training.
- 15:19SPEAKER_01: The issue is the recipes are
- 15:21SPEAKER_02: mostly secret.
- 15:22SPEAKER_02: So the frontier labs, they don't tell you how they build their training data, right?
- 15:28SPEAKER_02: They don't know how they filter it,
- 15:30SPEAKER_02: how they construct it, and also obviously not how they use synthetic
- 15:36SPEAKER_02: Also not the Chinese labs that are releasing their model's open weight.
- 15:41SPEAKER_02: And training data is arguably the most important ingredient in building
- 15:47SPEAKER_02: a world-class LLM, but the recipe is still a black box.
- 15:52SPEAKER_02: So if you can't build the data,
- 15:54SPEAKER_02: you can't build the model, and that's why we're focusing on the data side here.
- 16:01SPEAKER_02: Early on, lots of the compute or most of the compute was spent on model training,
- 16:08SPEAKER_02: but now more and more of the compute goes into actually creating
- 16:13SPEAKER_02: or curating training data, and this both in pre-training and in post-training.
- 16:19SPEAKER_02: For generating synthetic data and then post-training during the rollouts for RL, for example.
- 16:25SPEAKER_02: So you might ask model collapse, right?
- 16:29SPEAKER_02: Like there is this well-known paper from Shumailov.
- 16:35SPEAKER_02: They basically said if you're training on synthetic data, at some point your model collapses.
- 16:42SPEAKER_02: Here we have to be a bit careful because the setup
- 16:45SPEAKER_02: that they used was quite constructed to showcase that you can collapse a model,
- 16:52SPEAKER_02: so they used a closed-loop training.
- 16:56SPEAKER_02: Ran inference on the model and then trained the model on its own output.
- 17:00SPEAKER_02: No human signal in between, like no seed documents.
- 17:04SPEAKER_02: And then the model collapsed.
- 17:06SPEAKER_02: But you would not really do this in practice, right?
- 17:10SPEAKER_02: When you actually create synthetic data in practice, usually you start with a seed document.
- 17:15SPEAKER_02: You have a strategy of how you want to do it.
- 17:18SPEAKER_02: And you always mix in real data as well.
- 17:22SPEAKER_02: In Findphrase, we didn't see...
- 17:28SPEAKER_02: And also in the fine web paper before that,
- 17:32SPEAKER_02: I think they trained on certain documents that were already known to be AI generated
- 17:40SPEAKER_02: on the web at that point.
- 17:41SPEAKER_01: We are still in a phase of alchemy and we want to
- 17:45SPEAKER_02: turn this alchemy into chemistry
- 17:47SPEAKER_02: and make it a bit more known how this exactly works.
- 17:51SPEAKER_02: And for that, we use these three axes, the model, the prompt and the data.
- 17:58SPEAKER_02: And we do this with systematic and reproducible experiments.
- 18:03SPEAKER_02: So to give you a bit of the numbers, in the end, we did 90 rephrasing configurations.
- 18:09SPEAKER_02: So 90 times we generated synthetic data and then had 333 evaluate experiments on top
- 18:17SPEAKER_02: of that, where we trained a model and evaluated it on the downstream benchmarks.
- 18:21SPEAKER_02: In total,
- 18:22SPEAKER_02: just for the ablations,
- 18:24SPEAKER_02: this generated 1 trillion plus tokens,
- 18:27SPEAKER_02: just synthetic data, and this used 12.7 GPU years or 111,800 hours to generate.
- 18:36SPEAKER_02: In the end, we used the final recipe to generate these 486 billion tokens of Findphrase.
- 18:43SPEAKER_02: So to start with the headline figure,
- 18:46SPEAKER_02: using our data,
- 18:47SPEAKER_02: you can train a 1.7 billion model,
- 18:50SPEAKER_02: either to 31% better benchmark performance or 3.2 times faster than
- 18:54SPEAKER_02: the best baseline that we
- 19:01SPEAKER_02: Compared to at the time.
- 19:02SPEAKER_02: So this is the table prompt, more about that later.
- 19:07SPEAKER_02: And we mixed it at its optimal 70% synthetic ratio, 30% fine web edu high quality data.
- 19:16SPEAKER_02: So we have a little bit of an animation that I had fun generating with Claude.
- 19:23SPEAKER_02: It's not rendering in the presentation,
- 19:25SPEAKER_02: but maybe we can have a look at it later in the blog post.
- 19:28SPEAKER_02: So part two, what is rephrasing exactly?
- 19:32SPEAKER_02: So the way we see it is you start with a random web document and then you run it
- 19:40SPEAKER_02: through an LLM to produce variants that Change the presentation,
- 19:45SPEAKER_02: but keep the meaning intact.
- 19:46SPEAKER_02: So here you can imagine all kinds of different reformulations, right?
- 19:51SPEAKER_02: You could write the tutorial,
- 19:53SPEAKER_02: write FAQs, turn the article into tables and ask questions about it and so on.
- 20:00SPEAKER_02: So there is a lot of prior work on this, but it was still the alchemy phase.
- 20:06SPEAKER_02: No one really had done a systematic comparison against prompts.
- 20:12SPEAKER_02: Across prompts.
- 20:13SPEAKER_02: Nematron CC, I think they tried like five different prompts, which was the most I can remember.
- 20:21SPEAKER_02: We look at three axis of synthetic data, the prompt, the generator model, and the source data.
- 20:28SPEAKER_02: So for the source data,
- 20:30SPEAKER_02: we have DCLM,
- 20:31SPEAKER_02: FineWeb,
- 20:33SPEAKER_02: High-quality data,
- 20:34SPEAKER_02: FineWeb low-quality data, and Cosmopedia, another synthetic data set by itself.
- 20:40SPEAKER_02: Then we had nine different prompt strategies ourselves and compared also against
- 20:46SPEAKER_02: The nemotron ones and the rewire ones.
- 20:49SPEAKER_02: And for the model families, we used the best ones that we saw at the 1 billion scale here.
- 20:56SPEAKER_02: And for Gemma,
- 20:57SPEAKER_02: we also scaled up from 270 million to 27 billion parameters to see what difference
- 21:04SPEAKER_02: the size makes.
- 21:05SPEAKER_01: I understand that the frontier labs don't want to reveal their data stuff
- 21:11SPEAKER_01: because it's very secret sauce sensitive and stuff like that.
- 21:15SPEAKER_01: And I also understand that there's the different open labs that try different recipes and stuff.
- 21:22SPEAKER_01: But rarely they've been doing an ablation at this store in this, right?
- 21:30SPEAKER_01: Why do you think that is?
- 21:31SPEAKER_01: Why do you think that this experiment now is possible to run before it was not actually done?
- 21:40SPEAKER_01: Is it because it's in infancy and we're still figuring out what made sense
- 21:46SPEAKER_01: and what didn't make sense?
- 21:48SPEAKER_01: Or is it the compute that is too much?
- 21:50SPEAKER_01: What's your thought here?
- 21:51SPEAKER_02: I think for the Frontier Labs,
- 21:53SPEAKER_02: I would not be surprised if they have done similar things internally.
- 21:57SPEAKER_02: I just assume they don't publish it.
- 22:00SPEAKER_02: For example, Google,
- 22:02SPEAKER_02: where I know some people,
- 22:04SPEAKER_02: they just tell me that they can't really publish these things anymore.
- 22:08SPEAKER_02: And OpenAI Anthropic, they didn't really start in the first place, I guess.
- 22:12SPEAKER_02: And for the academic community,
- 22:14SPEAKER_02: I think there have been some works on this,
- 22:17SPEAKER_02: just that usually,
- 22:18SPEAKER_02: I guess, they don't really have the compute to do it at this scale.
- 22:22SPEAKER_01: And yeah,
- 22:22SPEAKER_01: it does seem like,
- 22:23SPEAKER_01: I don't know,
- 22:25SPEAKER_01: they are talking about some aspect of the modelization,
- 22:28SPEAKER_01: I would say,
- 22:29SPEAKER_01: like the Frontier Lab and stuff, but the data stuff, it's mostly silent.
- 22:34SPEAKER_01: We get some information through,
- 22:36SPEAKER_01: I don't know, like Entropic burning all these books and stuff like that, right?
- 22:39SPEAKER_01: But it's never something that is disclosed in such a way.
- 22:47SPEAKER_01: Okay, cool.
- 22:48SPEAKER_01: Thank you.
- 22:49SPEAKER_02: Yeah, so how do we do the rephrasing?
- 22:52SPEAKER_02: We have generator models from 270 million to 27 billion parameters,
- 22:57SPEAKER_02: the models from the families I mentioned before.
- 23:00SPEAKER_02: We have different source corpora that we rephrase.
- 23:04SPEAKER_02: So we looked specifically at different quality sets of fine web to differentiate
- 23:11SPEAKER_02: whether low quality data is enough for rephrasing, for example.
- 23:16SPEAKER_02: So we have this eduScore classifier that Five is the highest score and zero the lowest one.
- 23:24SPEAKER_02: So we have two sets of low quality and high quality one.
- 23:27SPEAKER_02: And then DCLem and Cosmopedia.
- 23:29SPEAKER_02: And each experiment rephrases roughly 10 billion tokens and then applies on
- 23:37SPEAKER_02: a 1.7 billion student and evaluates on 12 benchmarks.
- 23:43SPEAKER_02: We used VLLM for inference,
- 23:47SPEAKER_02: ran speculative decoding because that made sense in the first initial trials
- 23:52SPEAKER_02: that I did,
- 23:53SPEAKER_02: then we also scored each output by The FineWeb Edu and DCLM quality classifiers
- 24:01SPEAKER_02: to figure out if there is like a proxy that could help us figure out
- 24:06SPEAKER_02: the data quality beforehand, before training the ablation model.
- 24:10SPEAKER_02: And then we, yeah,
- 24:11SPEAKER_02: we distributed this to a hundred parallel SLURM tasks and each
- 24:16SPEAKER_02: of those SLURM tasks runs one VLLM model instance.
- 24:20SPEAKER_02: And this enabled us to like scale up on the cluster Like quite well,
- 24:26SPEAKER_02: also allowing other people in the company to use the cluster, right?
- 24:30SPEAKER_02: So just whenever the cluster was free,
- 24:33SPEAKER_02: it would rephrase web data and whenever someone else used it, it would pause.
- 24:38SPEAKER_02: So yeah, how do we measure your success?
- 24:41SPEAKER_02: We just used the ablation methodology from FineWeb because they took a lot of care
- 24:47SPEAKER_02: into making sure that the methodology makes sense at this scale because there's
- 24:53SPEAKER_02: a few gotchas to be aware of.
- 24:56SPEAKER_02: So at the 1.7 billion scale for just 20 billion tokens,
- 25:02SPEAKER_02: These models, they can't really do a lot.
- 25:04SPEAKER_02: So you wouldn't be able to evaluate them with the normal multiple choice setup.
- 25:12SPEAKER_02: You would just see random performance at that point, right?
- 25:15SPEAKER_02: So that's why we use the close format.
- 25:18SPEAKER_02: And then we evaluate on 12 benchmarks across six categories with 3Shot.
- 25:25SPEAKER_02: And yeah, a single seed.
- 25:27SPEAKER_02: So general knowledge,
- 25:28SPEAKER_02: reading comprehension,
- 25:29SPEAKER_02: reasoning, natural language understanding, math and table understanding.
- 25:33SPEAKER_02: So it's quite a broad coverage of the benchmarks that we use.
- 25:37SPEAKER_02: So now to the actual results.
- 25:43SPEAKER_02: First, we'll start with the baselines.
- 25:46SPEAKER_02: So, TCLM performed the best on these benchmarks at this 20 billion scale.
- 25:54SPEAKER_02: And then second is the NVIDIA
- 25:59SPEAKER_02: Rewire is a synthetic dataset from Facebook, and then we have more web dataset.
- 26:07SPEAKER_02: Interestingly,
- 26:08SPEAKER_02: Cosmopedia and Synth from Playus did not perform very well,
- 26:13SPEAKER_02: just a little bit better than the low quality fine web dataset.
- 26:18SPEAKER_02: So from here on, we just compare to DCLM as the best baseline.
- 26:24SPEAKER_02: Then some of these synthetic datasets,
- 26:27SPEAKER_02: they consist actually of multiple different parts generated with different prompts.
- 26:34SPEAKER_02: And they were generated with different models,
- 26:37SPEAKER_02: but just to compare the prompts,
- 26:40SPEAKER_02: we just used the prompts and regenerated and evaluated on our benchmark
- 26:45SPEAKER_02: so we could compare.
- 26:49SPEAKER_02: Particularly for the NemoTron dataset,
- 26:52SPEAKER_02: the performance seems to be mainly carried by this QA pairs prompt,
- 26:57SPEAKER_02: whereas the other ones fall behind.
- 27:01SPEAKER_02: And then there is like the summarize and continuation prompt from the From
- 27:07SPEAKER_02: the etology paper that were very simple,
- 27:11SPEAKER_02: just continue this web document or summarize it.
- 27:16SPEAKER_02: And the guided rewrite prompt from rewire is quite complicated.
- 27:20SPEAKER_02: They use the Lama 70b model for that.
- 27:24SPEAKER_02: And yeah, it performs at the baseline level for us here.
- 27:30SPEAKER_02: Then we saw a recurring trade-off,
- 27:33SPEAKER_02: for example, that certain datasets perform better on certain benchmarks.
- 27:40SPEAKER_02: For example, DCLM performed well on common sense tasks.
- 27:44SPEAKER_02: And then rephrased data,
- 27:46SPEAKER_02: for example, performed well on knowledge and reading comprehension tasks.
- 27:50SPEAKER_01: And I've seen this repeated in the paper and stuff.
- 27:53SPEAKER_01: I know that you may not have an answer here,
- 27:55SPEAKER_01: but just your general intuition,
- 27:58SPEAKER_01: because I have some ideas,
- 27:59SPEAKER_01: but I think you have more of a structured understanding of the problem space here.
- 28:06SPEAKER_01: Why this is?
- 28:08SPEAKER_01: Why when we rephrase and change synthetically the structure of the content,
- 28:16SPEAKER_01: we lose this common sense element?
- 28:18SPEAKER_02: Yeah, I'm very curious to hear your ideas as well,
- 28:21SPEAKER_02: because yeah, I don't really have a good explanation for it.
- 28:24SPEAKER_02: Maybe for Hellaswag, this is about continuation of sentences.
- 28:30SPEAKER_02: Maybe the benchmark sentences, they're created by humans, right?
- 28:36SPEAKER_02: And maybe that's just a different distribution than the LLM.
- 28:40SPEAKER_02: Generations and then the model is thrown off.
- 28:44SPEAKER_02: And for the knowledge and reading comprehension ones,
- 28:48SPEAKER_02: maybe this is like a format thing where for squat,
- 28:52SPEAKER_02: for example, it's like a question answering benchmark.
- 28:55SPEAKER_02: And if we train on FAQs, that may help.
- 29:00SPEAKER_02: But yeah, that's as far as my hypothesis go.
- 29:04SPEAKER_01: My hypothesis is that like a bit in the distribution element is
- 29:08SPEAKER_01: that like you saw off all of the kind of little noise that
- 29:13SPEAKER_01: the humans are like producing,
- 29:15SPEAKER_01: even like the low quality noise that will be like invariably present in
- 29:20SPEAKER_01: the actual test set.
- 29:23SPEAKER_01: So it doesn't know how to handle these, right?
- 29:26SPEAKER_01: Because it never kind of saw it because it did kind of like saw it off.
- 29:30SPEAKER_01: That's kind of how I'm intuitively looking at it.
- 29:34SPEAKER_01: Which this could be tested by taking the rephrased one,
- 29:38SPEAKER_01: adding back some of this noise synthetically, right?
- 29:42SPEAKER_01: And you could maybe see if you're able to get that common sense back.
- 29:47SPEAKER_01: Because if it's really a noise element,
- 29:50SPEAKER_01: and then now because you've seen all these weird edges around the data,
- 29:55SPEAKER_01: it knows how to not think too much about that and go in and answer the task
- 30:02SPEAKER_01: or stuff, yeah.
- 30:03SPEAKER_01: That's my rough understanding of this, but I think this is testable.
- 30:09SPEAKER_01: What's the value problem doing rephrasing?
- 30:12SPEAKER_01: Why do it at all and what is it trying to do?
- 30:15SPEAKER_02: I guess one of the easy answers is that we have lots of seed documents.
- 30:22SPEAKER_02: We have basically the whole internet and if we rephrase it to a different structure,
- 30:30SPEAKER_02: Then we can reuse a lot of the data.
- 30:33SPEAKER_02: Another early hypothesis or reason for us was
- 30:38SPEAKER_02: that maybe we can upcycle lower quality web documents by rephrasing them
- 30:45SPEAKER_02: and synthetically Generating a higher quality document that still keeps
- 30:53SPEAKER_02: the overall content.
- 30:56SPEAKER_02: And by starting with the general diverse web data, you can keep this diversity, right?
- 31:06SPEAKER_02: Because if you don't start with any seed documents,
- 31:11SPEAKER_02: the model collapse would be More of an issue, I would assume.
- 31:15SPEAKER_02: We even see it a bit later with the Quen versus SmallLM analysis we did,
- 31:22SPEAKER_02: where Quen generated higher quality math data sets.
- 31:28SPEAKER_02: Problems when you look at it personally,
- 31:31SPEAKER_02: but then it led to lower quality ablation models trained on for the benchmarks.
- 31:38SPEAKER_02: And our hypothesis there is that the Quent's rephrasing leads to less diversity in
- 31:44SPEAKER_02: the output data because it's just like it's too templatic.
- 31:51SPEAKER_01: I think this is also what prompted me for this to think about noise and little edges
- 31:58SPEAKER_01: that are not there, right?
- 32:00SPEAKER_01: Because I think on this part,
- 32:02SPEAKER_01: it's a bit different,
- 32:03SPEAKER_01: but still with the small versus the quen, you see kind of this element.
- 32:08SPEAKER_01: And here on the raw data and then synthetic, you see kind of a similar situation.
- 32:14SPEAKER_01: Math is a type of rephrasing, question mark.
- 32:17SPEAKER_01: Maybe explaining the flow from the document to a math problem.
- 32:23SPEAKER_02: Yeah.
- 32:23SPEAKER_02: Yeah.
- 32:24SPEAKER_02: So yeah,
- 32:24SPEAKER_02: basically the,
- 32:25SPEAKER_02: the idea was just like turning some article about like some,
- 32:31SPEAKER_02: like let's assume we have like some art, some Wikipedia article, right?
- 32:35SPEAKER_02: Like similar how you would just ask like, um, questions about it.
- 32:40SPEAKER_02: Um, you could also think of generating some math problems about that article,
- 32:45SPEAKER_02: like grade school math,
- 32:47SPEAKER_02: like, uh, some sentence problems you, you would remember from, from primary school.
- 32:53SPEAKER_02: There is this limitation that some articles may just not be suited for math problems, right?
- 33:01SPEAKER_02: And I think that would be interesting future work, right?
- 33:05SPEAKER_02: Similar to my colleague did it for fine translations,
- 33:08SPEAKER_02: where he basically had a gate inside the prompt for the translation.
- 33:17SPEAKER_02: So yeah,
- 33:17SPEAKER_02: a bit of context,
- 33:19SPEAKER_02: he translated all kinds of web documents in all kinds of languages to English,
- 33:26SPEAKER_02: and that can be used to train translation models that go back to these languages in
- 33:30SPEAKER_02: the end.
- 33:32SPEAKER_02: And what he did was he had like in the prompt saying that
- 33:40SPEAKER_02: if this document is like low quality,
- 33:42SPEAKER_02: then don't translate and just like output some flag
- 33:47SPEAKER_02: so they can easily filter it out.
- 33:49SPEAKER_02: And we could do something similar here for the math problems, right?
- 33:53SPEAKER_02: Like if this document is not suitable for math problems, just ignore it.
- 33:57SPEAKER_01: Yeah, here are the nine prompts that we
- 33:58SPEAKER_02: tried out.
- 33:59SPEAKER_02: So, article like writing an encyclopedic write-up or an opinionated analysis as
- 34:08SPEAKER_02: a commentary or a discussion and so on, right?
- 34:11SPEAKER_02: You can imagine all kinds of different structures that you want to use to rephrase the data.
- 34:17SPEAKER_02: And the four ones with the stars were the ones that performed best.
- 34:22SPEAKER_02: So in this case, we use the Gemma 1B model as a default.
- 34:30SPEAKER_02: And we see that the math prompt performs best.
- 34:35SPEAKER_02: And up to the tutorial, we see gains.
- 34:41SPEAKER_02: Compared to the baseline.
- 34:42SPEAKER_02: These are all mixed 50-50 with original data, which is FineWeb, Edu, high quality data.
- 34:51SPEAKER_02: If we would just do synthetic data by itself,
- 34:54SPEAKER_02: we wouldn't see these gains,
- 34:55SPEAKER_02: but we have another experiment showing that clearly later on.
- 34:58SPEAKER_02: Yeah, so it's more of a restructuring as well, not really like a rephrasing.
- 35:04SPEAKER_02: I think we started a bit with rephrasing,
- 35:07SPEAKER_02: just like a simple prompt rephrase this,
- 35:10SPEAKER_02: and then we saw that restructuring actually helps a lot.
- 35:14SPEAKER_02: And we also see that different prompts help on different benchmarks, right?
- 35:20SPEAKER_02: So then to what extent does the rephrasing model matter?
- 35:26SPEAKER_02: So here we look at the generator size.
- 35:28SPEAKER_02: So from Gemma 3, 207 million to 27 billion parameters.
- 35:34SPEAKER_02: Here I was a bit surprised to be honest, right?
- 35:36SPEAKER_02: Like you would expect that the 27 billion model generates the best data and then
- 35:41SPEAKER_02: the 270 million generates the worst, right?
- 35:44SPEAKER_02: For the 270 million,
- 35:46SPEAKER_02: we actually see that,
- 35:47SPEAKER_02: but then We saw that the 1B model generated at least as good data as the model
- 35:54SPEAKER_02: that was almost 30 times bigger.
- 35:56SPEAKER_02: I was a bit curious about why that's the case and to not tell you everything about it yet.
- 36:03SPEAKER_02: But the reason is probably because we measured on a 1.7 billion student model.
- 36:10SPEAKER_02: So if we change the size, we get different results and they have to be adapted.
- 36:15SPEAKER_02: Then another hypothesis we had was, do we need bigger models?
- 36:18SPEAKER_02: Results for low quality data, right?
- 36:20SPEAKER_02: So we rephrased high quality data and we rephrased low quality data,
- 36:26SPEAKER_02: and then we used different model sizes to do that.
- 36:29SPEAKER_02: And interestingly, as well here, we don't see better results with the bigger models.
- 36:35SPEAKER_02: We also see that the low quality data overall leads to a bit worse performance than
- 36:40SPEAKER_02: the high quality source data, but the difference is not that big.
- 36:45SPEAKER_02: To what extent does the model family matter, right?
- 36:48SPEAKER_02: And here we see that SmallLM outperforms all the other models and Quen
- 36:54SPEAKER_02: in this case performed the worst.
- 36:57SPEAKER_02: We think that the reason for that is Like a rewrite heavy data set in
- 37:04SPEAKER_02: the instruction tuning data set for smaller length.
- 37:06SPEAKER_02: Then to what extent does the model generation matter?
- 37:10SPEAKER_02: Here we see what we would expect.
- 37:12SPEAKER_02: We see from QN 1.4,
- 37:14SPEAKER_02: So to recap this part,
- 37:16SPEAKER_02: it looks like you can just use a small model for rephrasing in pre-training.
- 37:29SPEAKER_02: So you just really get better data if you use large models.
- 37:34SPEAKER_02: Then SmallLM is the model family that dominates across PROT.
- 37:40SPEAKER_02: Newer generations of models are better.
- 37:44SPEAKER_02: Cost matters.
- 37:46SPEAKER_02: So if you can rephrase with a 1B model,
- 37:49SPEAKER_02: you're so much faster than if you have to use a 27B model for rephrasing.
- 37:56SPEAKER_02: This makes a huge difference in how much data you can rephrase for the same amount of compute.
- 38:02SPEAKER_01: And I had a question about this part because like at this point,
- 38:09SPEAKER_01: I know you're going to do something different afterward,
- 38:11SPEAKER_01: but at this point you're training like a Quinn-like model here.
- 38:18SPEAKER_01: From well understood,
- 38:19SPEAKER_01: the architecture is closer to,
- 38:22SPEAKER_01: I think, a Quen 2.5 or something like that, if I remember correctly.
- 38:26SPEAKER_01: I think Quen 2, yeah.
- 38:28SPEAKER_01: Okay.
- 38:29SPEAKER_01: So do you think this is a form of self-teaching distillation that is going on here
- 38:36SPEAKER_01: for the formatting of things?
- 38:40SPEAKER_01: And just intuitively,
- 38:42SPEAKER_01: if the architecture is closer to Quen,
- 38:45SPEAKER_01: I would have assumed that a Quen model would have been the best generator for this,
- 38:51SPEAKER_01: but it doesn't seem to be the case.
- 38:52SPEAKER_01: It seems to be like the small one is a bit better on our front here.
- 38:58SPEAKER_01: So what's your intuition here about the closeness between the generator and the actual student?
- 39:07SPEAKER_02: Yeah.
- 39:08SPEAKER_02: Yeah.
- 39:08SPEAKER_02: It's a good question.
- 39:09SPEAKER_02: I think to me, this just makes clear that the data matters more than the architecture, right?
- 39:14SPEAKER_02: I would also assume that the I mean,
- 39:17SPEAKER_02: I'm not a specialist on that at all,
- 39:20SPEAKER_02: but I would assume that the architectures of the smaller lamb, quen,
- 39:24SPEAKER_02: llama and so on are fairly close,
- 39:27SPEAKER_02: but the data just makes much bigger of a difference during the training as well.
- 39:32SPEAKER_01: Yeah.
- 39:33SPEAKER_02: So next, to what extent do the dataset choices matter?
- 39:38SPEAKER_02: Yeah, here we were interested in seeing whether synthetic data is enough on its own, right?
- 39:45SPEAKER_02: Can we just train on synthetic data in the limit, basically?
- 39:49SPEAKER_02: And here we have quite a clear answer.
- 39:53SPEAKER_02: Synthetic data by itself pretty much always underperforms synthetic data mixed
- 40:00SPEAKER_02: with some original dataset.
- 40:03SPEAKER_02: And I think this goes together with the thing we've talked about before a bit,
- 40:09SPEAKER_02: that certain datasets are good for certain tasks, right?
- 40:13SPEAKER_02: And when you can switch,
- 40:15SPEAKER_02: when you can mix in DCLM,
- 40:17SPEAKER_02: which is good for Hellaswag, then you will see an increase in Hellaswag, right?
- 40:22SPEAKER_02: And then if you just remove it altogether in the synthetic data only case.
- 40:28SPEAKER_01: So, yeah,
- 40:28SPEAKER_01: if you look here, right,
- 40:29SPEAKER_02: like table plus DCLM versus table only or yeah,
- 40:36SPEAKER_02: so math only and math DCLM is like the only exception where it's like not better,
- 40:42SPEAKER_02: right?
- 40:43SPEAKER_02: FAQ plus DCLM is better than FAQ only and so on.
- 40:48SPEAKER_02: There is also other work that did this and they came to a conclusion
- 40:54SPEAKER_02: that 30% rephrased and 70% web data gave the best speedup.
- 41:01SPEAKER_02: For us, it's the other way around almost.
- 41:05SPEAKER_02: So for us,
- 41:06SPEAKER_02: we saw that around 70%,
- 41:09SPEAKER_02: depending on the prompt, synthetic data gives us the best results.
- 41:15SPEAKER_02: Table being the one that outperforms in total.
- 41:20SPEAKER_01: This part was very interesting to me.
- 41:24SPEAKER_01: Here it could be multiple reasoning why.
- 41:29SPEAKER_01: You just need to have the proper distribution of the two things.
- 41:34SPEAKER_01: Do you think it's a proportion thing or that the model needs to
- 41:39SPEAKER_01: at least see X amount Of token that are real, right?
- 41:46SPEAKER_01: And then whatever you add on top will just lead to better performance.
- 41:50SPEAKER_01: Even if it's synthetic, it doesn't matter.
- 41:53SPEAKER_01: Do you think it's more like this or literally a proportion where like,
- 41:56SPEAKER_01: let's say like you have a lot of token for like the real one,
- 42:00SPEAKER_01: but you make so much synthetic data that now it's kind of confused and stuff.
- 42:05SPEAKER_01: So which part are you aligning more here on the proportion side or like a threshold
- 42:12SPEAKER_01: of a capacity for the model?
- 42:13SPEAKER_02: Yeah, it's a good question.
- 42:16SPEAKER_02: I don't have the Kang et al work very present.
- 42:22SPEAKER_02: But one reason that I could imagine for them arriving at
- 42:26SPEAKER_02: the different rephrased mix Maybe their rephrase data is not as good as
- 42:32SPEAKER_02: the one we generated.
- 42:34SPEAKER_02: It takes quite a bit of experimentation to find prompts that work well versus others that don't.
- 42:42SPEAKER_02: For example, here the commentary prompt just worked much worse than the math prompt.
- 42:48SPEAKER_02: And it's rephrasing the exact same data, right?
- 42:51SPEAKER_02: Training the exact same model.
- 42:54SPEAKER_02: So the prompt makes a huge difference, right?
- 42:59SPEAKER_02: Maybe for different prompts, the synthetic data almost hurts, right?
- 43:04SPEAKER_02: And then you just want to have more real data, whereas in this case...
- 43:09SPEAKER_02: The synthetic data pulled the other ones up.
- 43:14SPEAKER_02: That could be one hypothesis,
- 43:17SPEAKER_02: but it's also quite difficult to compare across benchmark setup
- 43:23SPEAKER_02: and training configuration.
- 43:27SPEAKER_02: I don't know how many tokens they trained for.
- 43:30SPEAKER_02: It could be a scale issue as well.
- 43:32SPEAKER_02: Maybe they trained for longer or less long.
- 43:35SPEAKER_01: Yeah, that's really cool.
- 43:36SPEAKER_01: Can you go back to the previous slide also?
- 43:39SPEAKER_01: Because this was also very stuck.
- 43:41SPEAKER_01: You have the commentary that kind of stuck, right?
- 43:46SPEAKER_01: And then you have the table mat stuff that is kind of at the top.
- 43:51SPEAKER_01: Is there any other format,
- 43:53SPEAKER_01: wacky format that you've tried to say, I don't know, generating... What did I say?
- 44:00SPEAKER_01: Generating HTML website or even XML type of structure Over the content, right?
- 44:07SPEAKER_01: Same content,
- 44:08SPEAKER_01: but like an XML structure,
- 44:09SPEAKER_01: same content in the HTML structure,
- 44:12SPEAKER_01: or like same content,
- 44:13SPEAKER_01: but in a kind of discussion format, like a forum type of stuff, right?
- 44:19SPEAKER_01: Like a Reddit or Acuneews.
- 44:21SPEAKER_01: Have you explored like different other ways of creating these prompts?
- 44:26SPEAKER_02: Yeah, so I've tried these nine ones.
- 44:28SPEAKER_02: Yeah, just like thinking about how I would rephrase it that like makes sense, right?
- 44:36SPEAKER_02: XML, HTML, I didn't try.
- 44:39SPEAKER_02: Usually in pre-training, you do the opposite, right?
- 44:42SPEAKER_02: Like you extract the text from the HTML and from the XML.
- 44:46SPEAKER_02: And it's probably also not measured by the benchmarks,
- 44:50SPEAKER_02: like to what extent the model can generate HTML or XML.
- 44:54SPEAKER_02: Like in the ones that we measure, right?
- 44:56SPEAKER_02: So there you would probably see no gain for that.
- 44:59SPEAKER_02: If you have benchmarks that actually measure that, then I guess it would be useful.
- 45:06SPEAKER_02: Another thing we tried early was just rephrase in general, right?
- 45:10SPEAKER_02: Just telling it, make this higher quality.
- 45:14SPEAKER_02: But yeah, that's really exciting.
- 45:19SPEAKER_01: Is there also the order of these data sets being used for pre-training that matter
- 45:23SPEAKER_01: for the distribution?
- 45:24SPEAKER_01: That's actually also very interesting.
- 45:26SPEAKER_01: What's your thought here?
- 45:27SPEAKER_02: The order.
- 45:28SPEAKER_02: Yeah.
- 45:28SPEAKER_02: I mean, that's like curriculum learning is a whole different field of study, right?
- 45:35SPEAKER_02: I'm not an expert,
- 45:36SPEAKER_02: but to the extent of my knowledge,
- 45:41SPEAKER_02: it's quite clear that curriculum learning doesn't really work for pre-training.
- 45:47SPEAKER_02: There's no clear gains as far as I know.
- 45:49SPEAKER_02: I think it's very important for RL, I've been told, but... Yeah,
- 45:55SPEAKER_01: post-training, yes.
- 45:57SPEAKER_01: Yeah.
- 45:58SPEAKER_01: I think he has quite a lot of gain,
- 46:01SPEAKER_01: how you engineer the ordering of the different tasks that you're giving,
- 46:07SPEAKER_01: especially for continual learning.
- 46:09SPEAKER_01: I've been talking to Aiden, and he seems to have an impact on it, especially on the...
- 46:17SPEAKER_01: The difficulty of the question that you're asking and ordering them properly,
- 46:24SPEAKER_01: but also this is a whole other mess.
- 46:27SPEAKER_02: Yeah, it's a very interesting approach.
- 46:30SPEAKER_02: I'm also a bit sad that it doesn't work for pre-training,
- 46:33SPEAKER_02: but it's good to see that it works for post-training.
- 46:36SPEAKER_01: Do you think that like,
- 46:37SPEAKER_01: I don't know if a 1D model is too much for it,
- 46:40SPEAKER_01: but like maybe bigger, like adding multilinguality to the format, right?
- 46:45SPEAKER_01: It's the same thing, but like now it's translated in like this other language.
- 46:50SPEAKER_01: I would assume that like.
- 46:51SPEAKER_01: The generator then needs to be kind of a bit better, let's say, than 1B.
- 46:55SPEAKER_01: I think that then will maybe make a difference.
- 46:59SPEAKER_01: But would it be even useful as a synthetic,
- 47:03SPEAKER_01: I would say, element for the model that are being trained?
- 47:06SPEAKER_02: Yeah.
- 47:07SPEAKER_02: That's a, that's a great question.
- 47:08SPEAKER_02: Yeah.
- 47:08SPEAKER_02: I think, um, like,
- 47:11SPEAKER_02: would you do the rephrase,
- 47:12SPEAKER_02: like the,
- 47:13SPEAKER_02: the translation at the same time as the restructuring, um, or in a next step?
- 47:19SPEAKER_01: Yeah.
- 47:20SPEAKER_01: Like there's two ways of, of doing it, right?
- 47:21SPEAKER_01: You either have it as the same thing,
- 47:25SPEAKER_01: but now it's like a rephrase in the language,
- 47:28SPEAKER_01: so we can keep the same format and stuff like that.
- 47:30SPEAKER_01: Or what you could do is you have your nine things,
- 47:33SPEAKER_01: your nine different prompt,
- 47:35SPEAKER_01: but then you Translate all of these into another language.
- 47:41SPEAKER_01: Then you have this bigger augmentation.
- 47:43SPEAKER_01: But I would assume that it would be better to do the first one.
- 47:47SPEAKER_01: You just gain one of it,
- 47:49SPEAKER_01: but you keep the noise and then the particularity around the actual messier text.
- 47:57SPEAKER_01: That's my hunch, but I'm basing it on absolutely nothing.
- 48:01SPEAKER_02: So basically just do the translation, right?
- 48:03SPEAKER_02: Yeah, I mean like fine translations did basically this,
- 48:08SPEAKER_02: but from the low resource languages to the high resource languages,
- 48:13SPEAKER_02: to English basically,
- 48:15SPEAKER_02: because we know that,
- 48:17SPEAKER_02: Language models seem to be similar to humans,
- 48:20SPEAKER_02: that they understand languages they don't know very well,
- 48:25SPEAKER_02: but they can't speak them very well.
- 48:27SPEAKER_02: So they can translate much better from a low resource language to English than
- 48:31SPEAKER_02: the other But
- 48:36SPEAKER_02: the value of this fine translation dataset is mainly for training
- 48:39SPEAKER_02: in translation model.
- 48:40SPEAKER_02: If your goal is to get a dataset that can be used
- 48:44SPEAKER_02: to train better multilingual models,
- 48:46SPEAKER_02: then we would need to do it the other way around.
- 48:49SPEAKER_02: And that's exactly what you're asking for.
- 48:52SPEAKER_02: I think for that, it would be a very valuable thing to do, yes.
- 48:56SPEAKER_01: My hunch is that like,
- 48:58SPEAKER_01: since they understand like these,
- 49:00SPEAKER_01: like a low resource language,
- 49:02SPEAKER_01: but like they're not that fluent to kind of move in it, right?
- 49:05SPEAKER_01: And they would much prefer to have it in, I don't know, English, which they've seen more.
- 49:10SPEAKER_01: My hunch is that like, it would just introduce another kind of noise, right?
- 49:18SPEAKER_01: But that still fits what we're trying to get them to output, right?
- 49:24SPEAKER_01: That's kind of my hunch around it, right?
- 49:27SPEAKER_01: It will create a bit more diversity.
- 49:29SPEAKER_01: But for this to be correct, you need to have a good translator.
- 49:33SPEAKER_01: Otherwise, it will be mostly kind of garbage translation.
- 49:38SPEAKER_01: Yeah.
- 49:39SPEAKER_02: Okay, cool.
- 49:40SPEAKER_02: So you're saying it might not be that bad if translation is not perfect, right?
- 49:45SPEAKER_01: I think it will not be that bad.
- 49:47SPEAKER_01: It still needs to be kind of coherent in structure, right?
- 49:50SPEAKER_01: So that like the other model can kind of pick up on that, right?
- 49:55SPEAKER_01: Yeah.
- 49:56SPEAKER_01: I think like there's different type of translation to different type of like
- 50:00SPEAKER_01: if you do this from like English to Arabic,
- 50:03SPEAKER_01: I mean, that's going to be a pretty huge lift for the model.
- 50:07SPEAKER_01: But like language that are kind of adjacent,
- 50:10SPEAKER_01: I would say,
- 50:12SPEAKER_01: to English might introduce some of the I don't know,
- 50:16SPEAKER_01: the particularity of a language that can be captured for the benchmark.
- 50:21SPEAKER_01: And it's not going to be too hard of a task,
- 50:24SPEAKER_01: but then you create this diversity in formatting that we're capturing with the rest
- 50:28SPEAKER_01: of the format.
- 50:29SPEAKER_02: I mean, yeah, I guess you would need multilingual benchmarks to measure the gain, right?
- 50:34SPEAKER_01: Yeah, yeah, maybe.
- 50:36SPEAKER_02: But yeah, I think that could be a really cool project.
- 50:38SPEAKER_02: Maybe I should consider working on that next.
- 50:41SPEAKER_02: Continuing on the dataset choices.
- 50:43SPEAKER_02: Which non-synthetic dataset should you mix in, right?
- 50:46SPEAKER_02: And here we see that basically FineWeb, Edu, HQ and DCLM, they're like the same basically.
- 50:53SPEAKER_02: It doesn't really matter.
- 50:54SPEAKER_02: And Cosmopedia and FineWeb, Edu, low quality are worse.
- 51:01SPEAKER_02: And interestingly,
- 51:02SPEAKER_02: there is quite a big lift from just training on FineWeb,
- 51:06SPEAKER_02: Edu, high quality data to mixing it into like synthetic data.
- 51:13SPEAKER_02: Whereas for DCLM, we don't really see that gain, which I was a bit surprised by.
- 51:19SPEAKER_02: Then here we test what happens if the mix-in is the same as the source.
- 51:28SPEAKER_02: That means we have the source,
- 51:31SPEAKER_02: we rephrase low quality fine web edu data here,
- 51:36SPEAKER_02: and then we mix in the same data again.
- 51:39SPEAKER_02: And the same for Cosmopedia, DCLM, and high quality, right?
- 51:43SPEAKER_02: Here we see that, yeah, FineWebEdu, high quality, and DCLM just clearly beat the rest.
- 51:50SPEAKER_02: So it looks like the source does matter.
- 51:54SPEAKER_02: But if we fix to a strong mix-in,
- 51:59SPEAKER_02: I think in this case we fix to fineweb edu high quality,
- 52:04SPEAKER_02: then the source data doesn't really matter that much anymore.
- 52:10SPEAKER_02: And this is a really cool result,
- 52:11SPEAKER_02: I think,
- 52:11SPEAKER_02: because this tells us that the data wall that we've been hearing about
- 52:17SPEAKER_02: a bit is not really an issue, right?
- 52:21SPEAKER_02: You can even take very low quality data here,
- 52:24SPEAKER_02: like FindWeb Edu low quality is really the worst of the worst almost in terms
- 52:32SPEAKER_02: of web data available.
- 52:34SPEAKER_02: It's Edu score zero to one, right?
- 52:38SPEAKER_02: It's really bad.
- 52:41SPEAKER_02: But if you rephrase this and mix it with high-quality data,
- 52:45SPEAKER_02: you can get almost DCLM-level performance, which is a cool result, I feel.
- 52:52SPEAKER_02: So yeah, this opens up a much larger pool of data,
- 52:55SPEAKER_02: because DCLM,
- 52:56SPEAKER_02: I think,
- 52:57SPEAKER_02: they start with an initial pool and then filter it down to less than 1% of the data
- 53:03SPEAKER_02: that they actually use for pre-training in the end.
- 53:06SPEAKER_01: I haven't looked at like fine web,
- 53:07SPEAKER_01: like low quality, but like at this point, what is it that we're generating?
- 53:12SPEAKER_01: Like, uh,
- 53:12SPEAKER_01: if it's like,
- 53:13SPEAKER_01: this is low quality,
- 53:14SPEAKER_01: it's like,
- 53:15SPEAKER_01: is it more of a sort of literally desolation of, uh,
- 53:19SPEAKER_01: whatever the generator is,
- 53:21SPEAKER_01: the one B and that's kind of what's, what's, uh, what's going into the, the model.
- 53:25SPEAKER_01: And then like the edu low quality just doesn't matter anymore.
- 53:28SPEAKER_01: Like, is this, this, or like, it literally is fixing these, um,
- 53:37SPEAKER_01: What's your thought here?
- 53:38SPEAKER_02: To me,
- 53:39SPEAKER_02: I don't think it's distillation because the rephrasing is being done by a 1B model,
- 53:47SPEAKER_02: right?
- 53:48SPEAKER_02: I mean, the 1B model is trained for a long time and the student model is from scratch.
- 53:57SPEAKER_02: So it could be, I guess, yeah.
- 54:00SPEAKER_02: How would you test that?
- 54:02SPEAKER_02: I don't know.
- 54:03SPEAKER_01: And if it's distillation, a bigger model should get better performance, right?
- 54:08SPEAKER_01: Or maybe the capacity of the model is capped at 1B and stuff.
- 54:12SPEAKER_01: So this, I'm not sure.
- 54:14SPEAKER_01: But then what you could test is just try to lower the quality even more.
- 54:18SPEAKER_01: At some point,
- 54:20SPEAKER_01: if you're just feeding it pure garbage,
- 54:22SPEAKER_01: like literal,
- 54:23SPEAKER_01: literal,
- 54:24SPEAKER_01: you stripped away all information,
- 54:27SPEAKER_01: and then you shove it through this model and say,
- 54:31SPEAKER_01: this is interesting data, you should go ahead and make these in different templates.
- 54:38SPEAKER_01: And you see what happened.
- 54:39SPEAKER_01: And if it is able to boost the performance,
- 54:42SPEAKER_01: then you could theoretically say that that portion of the gain
- 54:46SPEAKER_01: from this literal noise Data that has a structural language,
- 54:51SPEAKER_01: but there's nothing in there for sure.
- 54:53SPEAKER_01: We removed all possible information.
- 54:58SPEAKER_01: This is the portion that can be attributable to distillation.
- 55:02SPEAKER_01: And this other portion is the actual gain from the dataset.
- 55:07SPEAKER_01: And you can have the dataset being varied here.
- 55:10SPEAKER_01: I think that will be a way to test this out.
- 55:13SPEAKER_02: I think there is also work on like literally training on like random stuff, right?
- 55:20SPEAKER_02: Like in pre-training.
- 55:23SPEAKER_02: I think even there they see like surprisingly strong models.
- 55:31SPEAKER_01: The model just want to learn, right?
- 55:33SPEAKER_01: You fit it, whatever.
- 55:35SPEAKER_02: Yeah, so then I think another natural question is to what extent does it help
- 55:40SPEAKER_02: to combine different prompts, right?
- 55:43SPEAKER_02: But here I was a bit surprised about the results.
- 55:46SPEAKER_02: So I would expect if you combine all the different datasets, you would get better performance.
- 55:54SPEAKER_02: And we see it to some degree, but not really clearly, right?
- 56:01SPEAKER_02: All the prompts,
- 56:03SPEAKER_02: it's just a little bit better than just using the math prompt together
- 56:07SPEAKER_02: with FineWeb high quality.
- 56:10SPEAKER_02: I think the interesting result here is that all prompts,
- 56:17SPEAKER_02: they can basically almost replace the source, right?
- 56:21SPEAKER_02: If we have lots of different synthetic data with different prompts
- 56:25SPEAKER_02: that are all good, we don't need the high-quality source data anymore.
- 56:30SPEAKER_01: We're almost at the math-level quality here.
- 56:34SPEAKER_02: Another thing, just like a small experiment I ran out of curiosity was the typos.
- 56:44SPEAKER_02: The rewire paper, the prompt there was full of typos.
- 56:50SPEAKER_02: And I was like,
- 56:51SPEAKER_02: does this make a difference if I improve it, if I fix the typos and the grammar?
- 56:57SPEAKER_02: But interestingly, it turns out to be the opposite way or no difference at all.
- 57:02SPEAKER_02: So for the 1B model, they engineered the typos very well,
- 57:08SPEAKER_01: it looks like.
- 57:09SPEAKER_02: And yeah,
- 57:10SPEAKER_02: so to summarize here,
- 57:13SPEAKER_02: It really looks like prompt design is the most important part,
- 57:19SPEAKER_02: maybe a bit sad as a result but that's what it is.
- 57:27SPEAKER_02: Size, it seems like beyond 1B,
- 57:31SPEAKER_02: it doesn't really help that much, maybe 4B, but 12B or 27B doesn't improve.
- 57:39SPEAKER_02: Then you should always mix with original data.
- 57:44SPEAKER_02: Don't just train on synthetic data alone.
- 57:47SPEAKER_02: And the source seems secondary if you have like a strong mix in data set.
- 57:53SPEAKER_02: So let's look a bit in more detail how and why this works.
- 57:59SPEAKER_02: So first we were interested in seeing if rephrasing like leaks the benchmark.
- 58:06SPEAKER_01: This is so funny because like on Twitter, there's a guy that has the same question.
- 58:10SPEAKER_01: He says like, what about benchmark contamination?
- 58:13SPEAKER_01: Have we covered that yet?
- 58:14SPEAKER_01: We're literally here now.
- 58:16SPEAKER_02: We ran a 10 gram overlap audit of the benchmarks to the rephrased corpora,
- 58:24SPEAKER_02: and we just sampled 5 billion tokens for each of the corpora.
- 58:29SPEAKER_02: The reason for that is that depending on the prompt,
- 58:31SPEAKER_02: there is different amounts of tokens being generated.
- 58:35SPEAKER_02: So we always started with 20 billion tokens input data,
- 58:39SPEAKER_02: and then some models, some prompts lead to much more rephrased data than others.
- 58:45SPEAKER_02: And yeah,
- 58:46SPEAKER_02: we see that the rewire is like the most contaminated,
- 58:53SPEAKER_02: but still it's like at a low rate.
- 58:57SPEAKER_02: So that's like the number of documents per billion training tokens.
- 59:02SPEAKER_02: And so, for example, here, like the FindWebEduHQ also is contaminated to some degree.
- 59:09SPEAKER_02: And the rephrased corpora that we produced are not...
- 59:12SPEAKER_02: They're on average less contaminated.
- 59:14SPEAKER_02: There's just one that is more.
- 59:17SPEAKER_02: Then another question is how much is seed noise?
- 59:21SPEAKER_02: We have a data seed and a training seed, like model initialization seed fixed.
- 59:27SPEAKER_02: But what if these seeds are lucky for some prompts and unlucky for others?
- 59:35SPEAKER_02: So we ran the same recipe across nine seed pairs,
- 59:39SPEAKER_02: so three data seeds and three model in a And across four student sizes.
- 59:45SPEAKER_02: And we see that the difference is very small.
- 59:50SPEAKER_02: So just like a gain of 0.05 is like already way above the noise.
- 59:58SPEAKER_02: It depends a bit on the benchmark.
- 1:00:00SPEAKER_02: So for drop...
- 1:00:05SPEAKER_02: The noise is much bigger than for Hellaswag, for example.
- 1:00:08SPEAKER_02: Also, the model initialization seat dominates the noise much more than the data ordering seat.
- 1:00:18SPEAKER_02: Here we see an overview of all the rephrasing runs plotted across GPU time and
- 1:00:27SPEAKER_02: the
- 1:00:30SPEAKER_02: aggregate macro score on all the benchmarks of the student model trained on
- 1:00:36SPEAKER_02: the dataset,
- 1:00:37SPEAKER_02: and we see that the The Pareto frontier is dominated by the small models.
- 1:00:43SPEAKER_02: So, yeah, like we don't really see big gains like as we've seen before, right?
- 1:00:50SPEAKER_02: Like from rephrasing with huge models that generate a lot of data.
- 1:00:55SPEAKER_02: For example, here,
- 1:00:57SPEAKER_02: like we spent a year of compute for one rephrasing run and it is worse
- 1:01:03SPEAKER_02: than spending one week with this other model, right?
- 1:01:07SPEAKER_01: Oh, this is kind of crazy.
- 1:01:08SPEAKER_01: This is insane.
- 1:01:10SPEAKER_01: And I think this has to do with maybe the limitation of the 1B that is addressed
- 1:01:15SPEAKER_01: with the other experiment that you've done, right?
- 1:01:17SPEAKER_02: Yeah, exactly.
- 1:01:19SPEAKER_02: So this ties back a bit to the original question that you asked, how to do research.
- 1:01:25SPEAKER_02: And I really wanted this to work, but it didn't, unfortunately.
- 1:01:29SPEAKER_02: The current setup is really time consuming, right?
- 1:01:35SPEAKER_02: You first have to generate 10 billion tokens.
- 1:01:38SPEAKER_02: As you see here, if you choose the wrong model, it takes you a year of GPU time.
- 1:01:45SPEAKER_02: And then you need to train a 1.7 billion model on 20 billion tokens,
- 1:01:50SPEAKER_02: which also
- 1:01:52SPEAKER_02: takes three hours on eight nodes, so 64 GPUs for three hours.
- 1:01:58SPEAKER_02: That's also quite an investment, right?
- 1:02:02SPEAKER_02: And only then you still need to evaluate, right?
- 1:02:06SPEAKER_02: It's quite an investment for each run.
- 1:02:10SPEAKER_02: So it would be great if we could just have a small proxy like the quality filters
- 1:02:15SPEAKER_02: that were used to create the Web Edu or the DCLM pre-training data sets.
- 1:02:20SPEAKER_02: But unfortunately, it doesn't really work that well for synthetic data.
- 1:02:26SPEAKER_02: So the highest correlation with macro average scores after the benchmark is
- 1:02:32SPEAKER_02: the DCLM score difference between the input data and the synthetic data,
- 1:02:38SPEAKER_02: which is 0.6.
- 1:02:41SPEAKER_02: Not bad, but still not great.
- 1:02:47SPEAKER_02: And the EDU score,
- 1:02:51SPEAKER_02: like if we just look for the output score from the EDU classifier,
- 1:02:57SPEAKER_02: there's like No correlation at all.
- 1:03:00SPEAKER_02: Interestingly,
- 1:03:03SPEAKER_02: if we have a high output EDU score, we actually get bad benchmark performance.
- 1:03:08SPEAKER_02: So the EDU score even seems to be sometimes negatively correlated with
- 1:03:14SPEAKER_02: the final benchmark performance, as we can also see here.
- 1:03:17SPEAKER_01: I've noticed that LM have a favorite vernacular.
- 1:03:20SPEAKER_01: Does that vernacular survive rephrasing, like Opus, for example, really like load-bearing?
- 1:03:25SPEAKER_01: Do you see,
- 1:03:26SPEAKER_01: depending on which of the generator,
- 1:03:28SPEAKER_01: if it's using more the ticks of these models,
- 1:03:34SPEAKER_01: or there's not that much difference at the scale?
- 1:03:39SPEAKER_02: Yeah, I haven't noticed that, but maybe I need to look into the data a bit more.
- 1:03:46SPEAKER_01: And there's another question that says,
- 1:03:48SPEAKER_01: do you think that the technique like rephrasing could be contributing toward
- 1:03:51SPEAKER_01: the model better understanding a true language semantic invariant to
- 1:03:53SPEAKER_01: the exact word choice?
- 1:03:55SPEAKER_01: I mean, like,
- 1:03:57SPEAKER_01: I think like a bit similar to my thought about like,
- 1:04:01SPEAKER_01: it's solving And it's like more standardizing.
- 1:04:08SPEAKER_01: I mean, the language that is being used and like,
- 1:04:11SPEAKER_01: because there's no different format,
- 1:04:13SPEAKER_01: like, do you think that this plays a role in this?
- 1:04:16SPEAKER_01: I mean, my worry about the standardization is a bit like the loss of diversity.
- 1:04:41SPEAKER_02: Similar to the thing we've seen with the Quen model, right?
- 1:04:44SPEAKER_01: But yeah,
- 1:04:44SPEAKER_01: the original hypothesis
- 1:04:46SPEAKER_02: was that the original web text can be very noisy and full
- 1:04:51SPEAKER_02: of grammar errors or typos or OCR errors and stuff,
- 1:04:56SPEAKER_02: and that we can get rid of those.
- 1:04:58SPEAKER_02: And this generates more high-quality data that improves for pre-training performance.
- 1:05:05SPEAKER_02: But yeah, I'm not sure about the exact mechanisms behind it.
- 1:05:09SPEAKER_01: Yeah.
- 1:05:09SPEAKER_02: So we also looked a bit at diversity metrics.
- 1:05:13SPEAKER_02: So can maybe diversity metrics can predict performance.
- 1:05:17SPEAKER_02: There is different ones like Vendy score over sentence embeddings of the output.
- 1:05:26SPEAKER_02: This helps on some benchmarks, but not on others.
- 1:05:30SPEAKER_02: On the aggregate score, it doesn't help.
- 1:05:33SPEAKER_02: There is gradient diversity,
- 1:05:36SPEAKER_02: like GVENDY score,
- 1:05:37SPEAKER_02: and this is the best predictor That we found for reasoning performance.
- 1:05:42SPEAKER_02: So if we go back, we see that for reasoning, there's like no really good score, right?
- 1:05:52SPEAKER_02: But the GVENDY score works well for that one, but then it doesn't work for others.
- 1:05:56SPEAKER_02: So maybe like if you combine different scores,
- 1:06:00SPEAKER_02: then you could achieve like a higher than 0.6 correlation.
- 1:06:05SPEAKER_02: But yeah, I don't know.
- 1:06:06SPEAKER_02: To me, it seems a bit over-engineered and a bit Like overfitted in the end.
- 1:06:12SPEAKER_02: So yeah, I didn't do that.
- 1:06:14SPEAKER_02: So yeah,
- 1:06:15SPEAKER_02: it looks like there really isn't a good proxy that you can use and you really need
- 1:06:22SPEAKER_02: to train and evaluate each dataset independently.
- 1:06:26SPEAKER_01: It looks like, unfortunately.
- 1:06:28SPEAKER_02: Then another thing we saw, like we talked about the 1B generator a lot, right?
- 1:06:31SPEAKER_02: So one reason for that could be that the student model that we train is just too small, right?
- 1:06:38SPEAKER_02: Like maybe if you train a 1B model,
- 1:06:42SPEAKER_02: then it just can't learn from like better data generated by the bigger models.
- 1:06:48SPEAKER_02: Right.
- 1:06:49SPEAKER_02: And when we look at this,
- 1:06:50SPEAKER_02: we train student models of different other sizes from 0.5 billion to 6.2.
- 1:06:58SPEAKER_02: Billion on the exact same data mixes.
- 1:07:01SPEAKER_02: And then we sweep the Gemma 3 generators from 270 million to 27 billion
- 1:07:07SPEAKER_02: on three different prompts.
- 1:07:08SPEAKER_02: And again, always mix in the FineWeb Edu.
- 1:07:11SPEAKER_02: And what we see here is basically the bigger the student gets,
- 1:07:16SPEAKER_02: the easier it can differentiate between good and bad data.
- 1:07:21SPEAKER_02: So for the 1B model,
- 1:07:24SPEAKER_02: what we saw was that like after the 1B generator,
- 1:07:29SPEAKER_02: it just basically squashes everything together, right?
- 1:07:32SPEAKER_02: It just can't differentiate anymore.
- 1:07:34SPEAKER_02: And same for the 0.5B, right?
- 1:07:37SPEAKER_02: Here, it can't even distinguish the 1B model from the 270 million model,
- 1:07:44SPEAKER_02: and the difference between the smallest and the biggest model is just very small.
- 1:07:49SPEAKER_02: But then when we go to the 2.9 billion and the 6.2 billion,
- 1:07:53SPEAKER_02: we see a much bigger difference, and here we actually see the
- 1:08:00SPEAKER_02: The 12B model performing quite well.
- 1:08:03SPEAKER_02: We see that the 207 million model across all prompts and student sizes is really bad.
- 1:08:11SPEAKER_02: The 1B model as a generator is quite good already, much better than the 270 million.
- 1:08:18SPEAKER_02: And then if we train a bigger student model,
- 1:08:23SPEAKER_02: then we see that bigger models actually have benefits.
- 1:08:28SPEAKER_01: For me, this was also super interesting.
- 1:08:32SPEAKER_01: When you look at the chart,
- 1:08:33SPEAKER_01: the gain with the bigger generator,
- 1:08:35SPEAKER_01: at some point you have to do the trade-off with the compute that you're taking
- 1:08:38SPEAKER_01: because it jumps.
- 1:08:39SPEAKER_01: Because the 270 still is doing something.
- 1:08:41SPEAKER_01: It's not doing nothing.
- 1:08:50SPEAKER_01: But do you think that like purpose built trained 270 million parameters, right?
- 1:08:56SPEAKER_01: Like that is specifically like a fine tune or something like that to be a very good generator.
- 1:09:03SPEAKER_01: Do you think that there will be gain over there to kind of climb up to
- 1:09:06SPEAKER_01: the I don't know, the 4B and stuff like that.
- 1:09:09SPEAKER_01: Because then you would just need to train and make sure that whatever prompt
- 1:09:12SPEAKER_01: that you're using for the translation here from one data format
- 1:09:19SPEAKER_01: to another is well done, right?
- 1:09:21SPEAKER_01: But then you have the gain of having this very compute efficient model.
- 1:09:27SPEAKER_01: I would assume it's The perfect task for a small model because you have
- 1:09:33SPEAKER_01: the information and then you have the output.
- 1:09:38SPEAKER_01: There's very little unknown that you have to work with here.
- 1:09:41SPEAKER_01: You just have to be able to understand the different templates.
- 1:09:45SPEAKER_01: What do you think?
- 1:09:46SPEAKER_02: For sure, yeah.
- 1:09:47SPEAKER_02: I think here I just focused on prompt engineering, right?
- 1:09:50SPEAKER_02: Because the design space was so huge already.
- 1:09:53SPEAKER_02: But I think this is a very interesting future work.
- 1:09:55SPEAKER_02: Concurrently, an intern at Huggy Face did something similar where I think he fine-tuned.
- 1:10:01SPEAKER_02: I don't recall the results, but yeah, I think this could be really promising.
- 1:10:07SPEAKER_02: I agree, a 270 million model should be able to do this, right?
- 1:10:11SPEAKER_02: It's not like the task is super hard.
- 1:10:15SPEAKER_01: Maybe there's some templates that are harder than others,
- 1:10:17SPEAKER_01: let's say the matte one,
- 1:10:18SPEAKER_01: maybe it's like the hardest and stuff,
- 1:10:20SPEAKER_01: but then you can kind of mix and match the different model size and you don't have
- 1:10:24SPEAKER_01: to kind of use like the 27B for like just making tables, right?
- 1:10:29SPEAKER_01: Maybe there's a trade-off also at like the different difficulties that you can Yeah,
- 1:10:34SPEAKER_02: it also just matters a bit like on the prompt, right?
- 1:10:38SPEAKER_02: Like for the math prompt,
- 1:10:41SPEAKER_02: we see the 1b model being the best even with the small student size, right?
- 1:10:49SPEAKER_02: Also, one thing we saw in the plots here before, right?
- 1:10:53SPEAKER_02: Like the student size just makes a huge difference in terms of the macro Macro score
- 1:11:00SPEAKER_02: on the benchmarks.
- 1:11:01SPEAKER_02: I mean, it's not very surprising,
- 1:11:03SPEAKER_02: I guess, but still, after 20 billion tokens, the difference is quite big.
- 1:11:11SPEAKER_02: And the generator size.
- 1:11:15SPEAKER_02: The data doesn't make that big of a difference.
- 1:11:19SPEAKER_02: You can still scale a larger model by quite a lot,
- 1:11:24SPEAKER_02: but just having a larger model by itself, you see quite big gains.
- 1:11:30SPEAKER_01: Then also
- 1:11:33SPEAKER_02: here we investigated how chatty the models are with different prompts.
- 1:11:37SPEAKER_02: So overall,
- 1:11:39SPEAKER_02: the table prompt led to very short outputs and the FAQ and the tutorial ones led
- 1:11:46SPEAKER_02: to much more verbose ones.
- 1:11:50SPEAKER_02: And also the variance across models was much bigger there.
- 1:11:54SPEAKER_01: Yeah, we also looked a bit about compression ratio versus downstream performance.
- 1:11:59SPEAKER_02: Very early in the project,
- 1:12:01SPEAKER_02: I chatted with a friend of mine and his hypothesis was
- 1:12:05SPEAKER_02: that synthetic data generation is basically like Doing compression or like because
- 1:12:11SPEAKER_02: of compression, it leads to better pre-training performance.
- 1:12:15SPEAKER_02: Like you start with a certain amount of information,
- 1:12:19SPEAKER_02: say like in a thousand tokens,
- 1:12:20SPEAKER_02: and then you rephrase it into 200 tokens and you still have the same information
- 1:12:25SPEAKER_02: or most of it, but now represented in 200 tokens.
- 1:12:29SPEAKER_02: And since you still train for 20 billion tokens,
- 1:12:33SPEAKER_02: you have much more information in the synthetic data now than in the original one,
- 1:12:36SPEAKER_02: right?
- 1:12:37SPEAKER_02: That was his hypothesis.
- 1:12:39SPEAKER_02: It's a bit difficult with our results now because for some,
- 1:12:45SPEAKER_02: we do see that relationship, but then also it goes in the other direction.
- 1:12:49SPEAKER_02: We just see very big variance in the
- 1:12:55SPEAKER_02: space where the compression happens.
- 1:12:57SPEAKER_01: So I can't really answer this question, unfortunately.
- 1:13:00SPEAKER_02: Yeah, I've talked a bit about this already.
- 1:13:02SPEAKER_02: So, like,
- 1:13:03SPEAKER_02: we were a bit surprised when we looked at the results,
- 1:13:06SPEAKER_02: like the output from QN and smaller LAM.
- 1:13:09SPEAKER_02: When you look at the QN output, it looks great.
- 1:13:12SPEAKER_02: And when you look at the smaller LAM output, it looks weird.
- 1:13:15SPEAKER_02: And then you see the small LM outputs being much better on the benchmarks.
- 1:13:20SPEAKER_02: So we tried to figure out why that's the case.
- 1:13:22SPEAKER_02: And yeah,
- 1:13:23SPEAKER_02: the hypothesis we arrived at was in the end that
- 1:13:26SPEAKER_02: the small LM just creates much more diverse outputs.
- 1:13:30SPEAKER_02: Outputs and diversity seems to be important for pre-training.
- 1:13:35SPEAKER_02: So, for example,
- 1:13:36SPEAKER_02: we see that Quen only has 40 different 10-char openings,
- 1:13:43SPEAKER_02: whereas SmallLM has like Almost 2000 ones, right?
- 1:13:47SPEAKER_02: Then we also looked a bit at the math output.
- 1:13:52SPEAKER_02: So for example, SmallLM has like in 23% of cases, there's no numbers in the math rephrased data.
- 1:14:00SPEAKER_02: Whereas for Quen, it's just 0.1%, right?
- 1:14:03SPEAKER_02: Like we see that Quen follows the instructions much better
- 1:14:07SPEAKER_01: than SmallLM.
- 1:14:08SPEAKER_02: We also checked the accuracy of the equations and saw more accurate equations
- 1:14:16SPEAKER_02: for Gwen and more valid math overall,
- 1:14:20SPEAKER_02: but still for pre-training, this doesn't seem to matter.
- 1:14:24SPEAKER_02: For pre-training at this stage,
- 1:14:26SPEAKER_02: it's probably just not about correctness, but yeah, more about that.
- 1:14:31SPEAKER_01: And does these results hold for small and Quinn as the model grows in size?
- 1:14:37SPEAKER_01: For the student,
- 1:14:37SPEAKER_01: if the student is bigger,
- 1:14:38SPEAKER_01: let's say it's a six or whatever,
- 1:14:41SPEAKER_01: does this still hold that the diversity of the small one doing whatever is better
- 1:14:48SPEAKER_01: than Quinn just following the instruction?
- 1:14:50SPEAKER_02: That's a good question.
- 1:14:51SPEAKER_02: That I don't know.
- 1:14:52SPEAKER_02: I haven't run that experiment.
- 1:14:54SPEAKER_02: Yeah.
- 1:14:54SPEAKER_01: Because for me,
- 1:14:56SPEAKER_01: this is the dumping down with the numbers,
- 1:14:59SPEAKER_01: I think,
- 1:14:59SPEAKER_01: almost make me think that the problems are now more in the capacity of the 1B model
- 1:15:07SPEAKER_01: to kind of understand.
- 1:15:08SPEAKER_01: I'm thinking more from a post-training curriculum learning lens here where
- 1:15:13SPEAKER_01: The problems are easier and then like you can just kind of like figure it out better
- 1:15:17SPEAKER_01: than like just Quint being this kind of distilled kind of god model
- 1:15:21SPEAKER_01: that like knows much better than like this poor one beat train from scratch.
- 1:15:26SPEAKER_01: And it's like just putting numbers everywhere and like I don't understand numbers yet.
- 1:15:30SPEAKER_01: Right.
- 1:15:31SPEAKER_01: That's my that's like the rough feeling I had when I saw these.
- 1:15:36SPEAKER_01: But it doesn't explain the diversity of messed up information.
- 1:15:39SPEAKER_01: Maybe it's adding enough noise with simple enough structure and output
- 1:15:45SPEAKER_01: that it can pick it up and remove the noise and say,
- 1:15:48SPEAKER_01: yeah, this is what matters for me now.
- 1:15:50SPEAKER_01: Versus the other one is like, Kind of a ruthless teacher in some way.
- 1:15:54SPEAKER_01: We're like, no, no, this is the thing that you need to understand.
- 1:15:58SPEAKER_01: It's clean.
- 1:15:59SPEAKER_01: This is it.
- 1:16:00SPEAKER_01: But it doesn't have enough diversity to figure out the hardness of it.
- 1:16:05SPEAKER_01: And it just doesn't get it, right?
- 1:16:07SPEAKER_01: Well, the other kind of climb up a bit.
- 1:16:10SPEAKER_01: And even if the teacher is messy,
- 1:16:12SPEAKER_01: the problems are kind of more in the ballpark of this poor train from scratch 1v.
- 1:16:18SPEAKER_02: Yeah, that's a good explanation.
- 1:16:22SPEAKER_02: Maybe it's too hard, right?
- 1:16:25SPEAKER_02: It can't learn.
- 1:16:26SPEAKER_02: Math is just at this training stage at 10 billion tokens or something
- 1:16:31SPEAKER_02: that it has seen, just not possible to solve.
- 1:16:33SPEAKER_02: And since the small LLM output has less math in it, Yeah, so some summaries about this.
- 1:16:40SPEAKER_02: Yeah, we tested whether the results are within the noise,
- 1:16:44SPEAKER_02: but we can pretty confidently rule this out.
- 1:16:49SPEAKER_02: The student scale matters a lot,
- 1:16:51SPEAKER_02: so we need bigger students to be able to differentiate
- 1:16:55SPEAKER_02: between bigger generation models and smaller ones.
- 1:16:59SPEAKER_02: Unfortunately,
- 1:17:01SPEAKER_02: quality scores can't be used as proxies,
- 1:17:04SPEAKER_02: also not diversity scores,
- 1:17:07SPEAKER_02: even though G-Wendy helps for finding good reasoning data.
- 1:17:12SPEAKER_02: So there is unfortunately no substitute for training so far.
- 1:17:15SPEAKER_02: And it looks like diversity is more important than polish,
- 1:17:20SPEAKER_02: at least for
- 1:17:21SPEAKER_01: the math example that we looked at.
- 1:17:23SPEAKER_02: This is the content part.
- 1:17:27SPEAKER_02: Now it's about how we made this run.
- 1:17:32SPEAKER_02: So we're looking a bit at the infrastructure now.
- 1:17:34SPEAKER_02: So each experiment rephrases 10 billion tokens.
- 1:17:38SPEAKER_02: And as you've seen before, right?
- 1:17:40SPEAKER_02: If you choose the wrong model,
- 1:17:41SPEAKER_02: it takes you a year of GPU compute and also the serving setup makes
- 1:17:47SPEAKER_02: a big difference, right?
- 1:17:48SPEAKER_02: Since I ran this for 90 configurations,
- 1:17:51SPEAKER_02: I wanted to make sure that I don't burn months of GPU time For nothing
- 1:17:56SPEAKER_02: when I could have optimized it a bit more before, right?
- 1:17:59SPEAKER_02: BLM and SGLang already are like quite good at serving models fast,
- 1:18:04SPEAKER_02: but still like configuring them correctly can make a huge difference.
- 1:18:09SPEAKER_02: Leandro and Carlos from Huggy Face I had a challenge together with Google recently
- 1:18:15SPEAKER_02: about serving Gemma really fast and they had like a challenge
- 1:18:21SPEAKER_02: with agents optimizing like throughput for Gemma
- 1:18:25SPEAKER_02: and I think they pushed it like fourfold within a few days with
- 1:18:30SPEAKER_02: a few optimizations there.
- 1:18:32SPEAKER_02: So there is quite a lot of potential here.
- 1:18:34SPEAKER_02: I didn't go as far, but still there is quite a lot that can be done.
- 1:18:40SPEAKER_02: How did we run this?
- 1:18:41SPEAKER_02: We used Datatroth for generating the data at scale.
- 1:18:47SPEAKER_02: So the way it works, we start with a HagiFace hub dataset.
- 1:18:51SPEAKER_02: We have a reader, then a transformer, which
- 1:18:56SPEAKER_02: makes calls to an inference engine like VLLM here.
- 1:19:01SPEAKER_02: This also supports more More complicated rollouts,
- 1:19:05SPEAKER_02: like where you have like a verifier inside as well,
- 1:19:09SPEAKER_02: and then you reject certain outputs and so on.
- 1:19:11SPEAKER_02: But here we just wanted to optimize for throughput, so we had the most simple setup possible.
- 1:19:16SPEAKER_02: And then you write again to Huggy Face dataset usually, right?
- 1:19:22SPEAKER_02: You can run this locally or on the Slurm cluster as we did it for scaling it up then.
- 1:19:26SPEAKER_02: Here we ran like a throughput sweep across different models.
- 1:19:31SPEAKER_02: And we saw that for some models, you can speed them up quite a lot.
- 1:19:35SPEAKER_02: For example,
- 1:19:35SPEAKER_02: for smaller M2,
- 1:19:36SPEAKER_02: 1.7 billion,
- 1:19:39SPEAKER_02: the model used for generating the final fine phrase data set,
- 1:19:43SPEAKER_02: we saw a 1.75x speed up in the end.
- 1:19:49SPEAKER_02: So I had two tiers here.
- 1:19:51SPEAKER_02: First, tweaking tensor parallel, maximum sequences, and maximum batch tokens.
- 1:19:58SPEAKER_02: And then in tier one, afterwards, the GPU memory utilization and speculative decoding.
- 1:20:03SPEAKER_02: Different bottlenecks need different fixes.
- 1:20:07SPEAKER_02: So for large models, they seem to be more memory bound.
- 1:20:15SPEAKER_02: If you add more tensor parallelism,
- 1:20:18SPEAKER_02: even though the model fits on one GPU already,
- 1:20:24SPEAKER_02: it adds to KVCache headroom and you can fit more concurrent sequences and that leads
- 1:20:31SPEAKER_02: to a speedup.
- 1:20:33SPEAKER_02: And for smaller models,
- 1:20:35SPEAKER_02: This type of speculative decoding helped here,
- 1:20:38SPEAKER_02: like ngram and suffix,
- 1:20:40SPEAKER_02: where you look at the inputs and then you just copy them over and verify with
- 1:20:48SPEAKER_02: the model how much of the input is correct.
- 1:20:52SPEAKER_02: And for the small models,
- 1:20:54SPEAKER_02: we saw quite an increase with So with the small LM2,
- 1:21:01SPEAKER_02: we arrived at 9200 tokens per second per GPU in the end.
- 1:21:06SPEAKER_02: And for, we also managed to optimize like GPT OSS 120 billion.
- 1:21:13SPEAKER_02: To 6,000 tokens, so we could also have run that, would not have taken it much longer.
- 1:21:19SPEAKER_02: You see there's like quite a big gain for each experiment in terms of GPU costs
- 1:21:27SPEAKER_02: that can be saved.
- 1:21:29SPEAKER_02: There's another visualization here, but viewed better in the blog post.
- 1:21:33SPEAKER_02: We also added some convenience features for automated updating of dataset cards
- 1:21:40SPEAKER_02: that show you the progress of how the data is being generated,
- 1:21:46SPEAKER_02: which can be helpful if you run it for multiple days or weeks.
- 1:21:51SPEAKER_02: So finally, we arrived at the completed dataset.
- 1:21:55SPEAKER_02: We used the smaller than 2, 1.7b instruct model that won the family ablation.
- 1:22:01SPEAKER_02: And with speculative decoding, we arrived at 9,000 tokens per second.
- 1:22:06SPEAKER_02: We used the fineweb edu as a source dataset and used the four best prompts we had
- 1:22:14SPEAKER_02: and ran it on 100 H100s in parallel.
- 1:22:17SPEAKER_02: So we generate around 3.3 billion tokens per hour.
- 1:22:22SPEAKER_02: In total, we had 339 million documents, each rephrased four times with the different prompts.
- 1:22:29SPEAKER_02: In the end, we arrived at 486 billion tokens rephrased.
- 1:22:34SPEAKER_02: So 100 GPUs, we get to 920,000 tokens per second.
- 1:22:38SPEAKER_02: So 3.3 billion tokens per hour takes 612 GPU days in total.
- 1:22:42SPEAKER_02: So around six days to...
- 1:22:50SPEAKER_02: To really generate it if we had access to 100 GPUs
- 1:22:54SPEAKER_01: at all times.
- 1:22:55SPEAKER_02: And here are a few examples of the dataset, but it's also better viewed in the blog post.
- 1:23:02SPEAKER_02: Again, here to compare to the baselines with the different prompts,
- 1:23:07SPEAKER_02: the table one is the best we found in this case,
- 1:23:10SPEAKER_02: but the math is very close and FAQ and tutorial are also still quite a bit better
- 1:23:16SPEAKER_02: than The best synthetic baselines we compared to.
- 1:23:21SPEAKER_02: We also compared a bit the cost per token to the peers,
- 1:23:25SPEAKER_02: like Cosmopedia,
- 1:23:27SPEAKER_02: for example,
- 1:23:28SPEAKER_02: generated 2.5 million tokens per GPU hours and rewire only 1.1 million
- 1:23:35SPEAKER_02: because they used a much bigger model and didn't optimize as much maybe.
- 1:23:40SPEAKER_02: So we used 24 times less compute, but generated more tokens.
- 1:23:47SPEAKER_02: So to conclude, prompt design wins.
- 1:23:51SPEAKER_02: It's by far the biggest lever.
- 1:23:52SPEAKER_02: Structured prompts like table, math, FAQ, format, tutorial work quite well.
- 1:23:59SPEAKER_02: Always mixing raw web data.
- 1:24:02SPEAKER_02: Synthetic data alone loses on common sense tasks.
- 1:24:07SPEAKER_02: 60 to 80% synthetic data worked the best for us.
- 1:24:11SPEAKER_02: Small models seem to be enough.
- 1:24:12SPEAKER_02: 1B or 4B models seem to be good.
- 1:24:16SPEAKER_02: If you're training like a bigger student model,
- 1:24:18SPEAKER_02: then it makes sense to use bigger models for synthetic data generation.
- 1:24:25SPEAKER_02: And diversity seems to be better than or more important than polish
- 1:24:30SPEAKER_01: at this scale.
- 1:24:30SPEAKER_02: There's still many open questions.
- 1:24:33SPEAKER_02: So for example, like,
- 1:24:35SPEAKER_02: can you repeat data more often without performance loss if some of
- 1:24:40SPEAKER_02: the repetitions are rephrased, right?
- 1:24:42SPEAKER_02: So is the 60 to 80% optimum?
- 1:24:49SPEAKER_02: What does it depend on, right?
- 1:24:52SPEAKER_02: Does it change with larger student sizes or longer training?
- 1:24:58SPEAKER_02: I fixed all the generation parameters, so what happens if we change temperature or others?
- 1:25:06SPEAKER_02: Best of n filtering, we could generate multiple.
- 1:25:09SPEAKER_02: Rollouts for each example and then have a verifier that scores and only keeps
- 1:25:14SPEAKER_02: the best one, but obviously would be much more expensive again.
- 1:25:19SPEAKER_02: Could use tools like prompt optimization to improve the rephrasing performance.
- 1:25:25SPEAKER_02: I did that a bit, had negative results in the beginning.
- 1:25:30SPEAKER_02: And longer pre-training.
- 1:25:33SPEAKER_02: So far, I've trained for 21 billion tokens,
- 1:25:36SPEAKER_02: so I don't have to generate too much synthetic data and
- 1:25:40SPEAKER_02: that I could have fast experimentation cycles.
- 1:25:45SPEAKER_02: To what extent does it hold at longer scales?
- 1:25:49SPEAKER_02: I did an initial experiment at the 100 billion token and it held.
- 1:25:55SPEAKER_02: So it was like pretty much the same ranking, which is good to see.
- 1:26:00SPEAKER_02: But what happens if we scale to a trillion?
- 1:26:03SPEAKER_01: I don't know.
- 1:26:04SPEAKER_02: So we open source everything,
- 1:26:07SPEAKER_02: the whole code,
- 1:26:08SPEAKER_02: all prompt templates,
- 1:26:09SPEAKER_02: evaluation details,
- 1:26:11SPEAKER_02: fine phrase data sets, ablation data, generation framework, and all the results.
- 1:26:17SPEAKER_02: Everything's open source.
- 1:26:19SPEAKER_02: You'll find it and all the links in the paper or on the blog post.
- 1:26:24SPEAKER_02: And here as well, happy to share the slides as well later.
- 1:26:27SPEAKER_02: Thanks a lot.
- 1:26:28SPEAKER_01: That was awesome, man.
- 1:26:29SPEAKER_01: This is great.
- 1:26:32SPEAKER_01: It's so thorough.
- 1:26:34SPEAKER_01: I mean like all the different permutations you've done and then there's still
- 1:26:39SPEAKER_01: so much to do,
- 1:26:39SPEAKER_01: but then like the fact that like you went and then you did all these ablations
- 1:26:44SPEAKER_01: and all these different experiments and you still kept everything kind of coherent.
- 1:26:49SPEAKER_01: And order.
- 1:26:50SPEAKER_01: That's super cool.
- 1:26:53SPEAKER_01: These are actually my favorite experiment because,
- 1:26:55SPEAKER_01: I don't know,
- 1:26:56SPEAKER_01: you can ask so many questions back to back and still get a lot of insight.
- 1:27:00SPEAKER_01: But then you can distill the stuff that matters afterwards and then kind
- 1:27:03SPEAKER_01: of sprawl more questions, right?
- 1:27:05SPEAKER_01: But with a better ground that you're in.
- 1:27:08SPEAKER_01: Just like your general thought here about, like this is for pre-training, right?
- 1:27:13SPEAKER_01: What are you taught with the usefulness of synthetic data But on the post training side, right?
- 1:27:21SPEAKER_01: Did you have early experiment or like taught around that already?
- 1:27:25SPEAKER_02: So I'm not really well versed in post-training,
- 1:27:29SPEAKER_02: but members of the team use synthetic data quite heavily for post-training.
- 1:27:36SPEAKER_02: I think there was recent work called QED Nano where they generated lots
- 1:27:43SPEAKER_02: of reasoning traces with big models and then fine-tuned the model there.
- 1:27:48SPEAKER_02: And I think they were able to match
- 1:27:52SPEAKER_02: Like Gemini 3's performance on math reasoning tasks like IMO.
- 1:28:00SPEAKER_02: So, yeah,
- 1:28:01SPEAKER_02: I think you can use it to distill knowledge from bigger models to smaller ones,
- 1:28:09SPEAKER_02: for example.
- 1:28:09SPEAKER_01: And also my thought here is that like it would be very useful kind of documentation
- 1:28:15SPEAKER_01: in an environment to have like this sort of like on the fly rephrase or like
- 1:28:20SPEAKER_01: in different like structure,
- 1:28:22SPEAKER_01: but still kind of keep like the The overall problem set that you're using.
- 1:28:28SPEAKER_01: So it can make the problem easier a bit by taking you step by step as a tutorial,
- 1:28:33SPEAKER_01: or it can make the problem a bit harder by asking it as an actual problem,
- 1:28:38SPEAKER_01: or handling it as a discussion.
- 1:28:40SPEAKER_01: But then you distill it down to like not thinking too much about the format,
- 1:28:43SPEAKER_01: but thinking about like the actual thing that you want the model to be training
- 1:28:47SPEAKER_01: about, which is pretty cool.
- 1:28:49SPEAKER_01: What's next on this direction for you,
- 1:28:52SPEAKER_01: like on the field and stuff like where you want to put and bring this next?
- 1:28:57SPEAKER_01: I think you're trying to get the paper accepted at the conference, right?
- 1:29:01SPEAKER_02: Yeah, we got some good reviews, so I hope it will work out.
- 1:29:06SPEAKER_02: Yeah, I think you had a really cool idea about making it multilingual,
- 1:29:10SPEAKER_02: like basically find translations in reverse to generate
- 1:29:15SPEAKER_02: a really big pre-training data set for multilingual pre-training.
- 1:29:21SPEAKER_02: That could help like low resource languages a lot.
- 1:29:24SPEAKER_02: I think that could be a very cool direction.
- 1:29:28SPEAKER_02: At the moment, I'm working more a bit on harness engineering.
- 1:29:32SPEAKER_02: So I helped out a bit on a project called Physics Engineering.
- 1:29:36SPEAKER_02: In turn, that was about engineering a harness to solve physics problems much better.
- 1:29:43SPEAKER_02: So we were able to push performance of Gemini from 17% to 30% just by using
- 1:29:50SPEAKER_02: a different harness, which seems to make a huge difference.
- 1:29:53SPEAKER_02: So quite excited about that work currently.
- 1:29:56SPEAKER_02: But synthetic data also seems like a really cool direction, so I might go back later.
- 1:30:01SPEAKER_01: Cool.
- 1:30:02SPEAKER_01: I mean, like everything is kind of connected.
- 1:30:04SPEAKER_01: I was talking to Florian from Prime Intellect and like,
- 1:30:06SPEAKER_01: I don't know,
- 1:30:07SPEAKER_01: like the harnesses, like the evals, the benchmark, the environment, right?
- 1:30:11SPEAKER_01: It's all, it's all roughly the same structure.
- 1:30:15SPEAKER_01: It's just the infra.
- 1:30:16SPEAKER_01: Behind that is a bit like it might be a bit different.
- 1:30:20SPEAKER_01: But at the end of the day,
- 1:30:21SPEAKER_01: just like trying to give the right format and data to the model so
- 1:30:25SPEAKER_01: that they can just kind of learn or to evaluate them in like specific configuration.
- 1:30:30SPEAKER_01: Really cool stuff, man.
- 1:30:31SPEAKER_01: Thank you a lot for coming and going through all my questions.
- 1:30:36SPEAKER_02: Sounds great.
- 1:30:37SPEAKER_02: Amazing questions.
- 1:30:39SPEAKER_02: Was a lot of fun.
- 1:30:40SPEAKER_01: Have a great day.