Latest / learning with yacine show / Inside a Frontier Open Model: Nemotron 3 Ultra Explained ft. NVIDIA's Chris Alexiuk
Transcript
- 0:00SPEAKER_00: Originally Frontier LM were dense transformer with full attention
- 0:04SPEAKER_00: in every single layer and making them smarter mostly meant making them bigger
- 0:08SPEAKER_00: and slower.
- 0:09SPEAKER_00: Now basically every Frontier open model is a sparse mixture of experts and
- 0:13SPEAKER_00: the way we use them The way it works is that an agent gets a goal,
- 0:16SPEAKER_00: reason, calls tools, observe the result, and loops.
- 0:30SPEAKER_00: None of these iterations pays the full inference cost of the model.
- 0:33SPEAKER_00: This has led to token count explosion where a single agentic task can burn
- 0:37SPEAKER_00: through tens of thousands of output tokens across dozens of turns.
- 0:41SPEAKER_00: Now, in this new era,
- 0:42SPEAKER_00: the issue is that raw intelligence stopped being The only thing
- 0:45SPEAKER_00: that matters anymore.
- 0:46SPEAKER_00: A model that is slightly smarter,
- 0:48SPEAKER_00: but six times slower,
- 0:49SPEAKER_00: lose on tasks that takes iterations because the faster model just crush
- 0:54SPEAKER_00: through more attempt in a similar time window.
- 0:56SPEAKER_00: In this video,
- 0:57SPEAKER_00: we're going to take a look at NVIDIA NEMOTRON 3 Ultra A frontier open model built
- 1:01SPEAKER_00: for agentic workload.
- 1:03SPEAKER_00: And engineered around one pretty cool thesis, which is a faster model is a smarter model.
- 1:08SPEAKER_00: We're talking about 550 billion parameters with 55 billion active,
- 1:13SPEAKER_00: pre-trained on 20 trillion tokens We'll have the chance to interview Chris
- 1:17SPEAKER_00: from NVIDIA,
- 1:18SPEAKER_00: a Senior Product Research Engineer who knows this model like the back of his pocket.
- 1:30SPEAKER_00: Our idea of Nemotron 3 Ultra is that speed is engineered into every single layer of the stack.
- 1:35SPEAKER_00: For example,
- 1:35SPEAKER_00: the architecture is mostly linear member 2 layers with just a handful
- 1:39SPEAKER_00: of full attention layers.
- 1:40SPEAKER_00: The latent MOE part is routing 512 expert into a compressed latent
- 1:45SPEAKER_00: The pre-training is done,
- 1:50SPEAKER_00: like I said,
- 1:51SPEAKER_00: in NVFP4, so 4-bit training, so you train more base model per unit of compute.
- 1:57SPEAKER_00: The interesting thing here is that most of these speed bets didn't Anyway,
- 2:01SPEAKER_00: some of them, like latent MOE, actually improved it.
- 2:04SPEAKER_00: The weights,
- 2:04SPEAKER_00: the pre-training and post-training dataset,
- 2:06SPEAKER_00: the oral environments,
- 2:07SPEAKER_00: the training recipe,
- 2:08SPEAKER_00: the post-mortem of their two training divergence are all open,
- 2:12SPEAKER_00: which is actually pretty intense.
- 2:14SPEAKER_00: So to get started understanding this beast,
- 2:16SPEAKER_00: you can check out right here their 60-page technical report.
- 2:19SPEAKER_00: Speaking of fully open model energetic harness,
- 2:22SPEAKER_00: let's talk a bit about RC, the open-weight lab that is sponsoring this interview.
- 2:26SPEAKER_00: If you're familiar with the open AI model landscape,
- 2:28SPEAKER_00: you know it's pretty much dominated by a lab outside of North America.
- 2:31SPEAKER_00: Well, RC is a cool lab in North America that not only open-weight their model,
- 2:36SPEAKER_00: which you can download and run like on whatever you want,
- 2:38SPEAKER_00: but is also very open about their training methodology and release a lot of
- 2:42SPEAKER_00: the research artifacts they are developing internally, like this.
- 2:44SPEAKER_00: The open weight model, but also their cleaning recipe and even the harness.
- 2:48SPEAKER_00: You can get their latest model on open router
- 2:50SPEAKER_01: over here.
- 2:50SPEAKER_00: Or if you're into personal agent, you can run them on Hermesys.
- 2:55SPEAKER_00: You can also use it straight in their cloud environment over here.
- 2:58SPEAKER_00: One fresh new thing that they released What I recently is NAC,
- 3:00SPEAKER_00: which I'm pretty excited about,
- 3:02SPEAKER_00: an open-source harness for long-running energetic work.
- 3:04SPEAKER_00: It's pretty useful for research work that are complex but decomposable,
- 3:07SPEAKER_00: with well-separated parts and concrete acceptance criteria,
- 3:12SPEAKER_00: like machine learning experiments are a good example of what Big thanks to RC
- 3:15SPEAKER_00: for sponsoring this interview.
- 3:40SPEAKER_00: A very cool lab to keep an eye on.
- 3:41SPEAKER_00: Let's now dive into the interview.
- 3:42SPEAKER_01: My name is Chris Alexiuk.
- 3:44SPEAKER_01: I'm a what's called a product research engineer.
- 3:49SPEAKER_01: Basically, I just do all three of those words.
- 3:53SPEAKER_01: That's what you can think about it as.
- 3:58SPEAKER_01: My background kind of is, I would say, like, normal.
- 4:03SPEAKER_01: I started as a physics undergrad,
- 4:07SPEAKER_01: and then I learned that you can teach computer stuff,
- 4:11SPEAKER_01: and I've just been doing that since then, and, you know, it's That's how it goes.
- 4:16SPEAKER_01: I come from like the OG ML days though,
- 4:19SPEAKER_01: back when we called this whole field machine learning instead of AI.
- 4:23SPEAKER_01: So it's been a wild ride to see how we got to where we are here.
- 4:28SPEAKER_00: It's crazy because like half of the people I'm interviewing Exactly the same setup.
- 4:35SPEAKER_00: They came from a physics background and then they went into ML back in the days
- 4:39SPEAKER_00: when ML was like, not this, right?
- 4:43SPEAKER_00: But like you have to do and grind and do the stuff and this and data and Android like that.
- 4:48SPEAKER_00: And I don't know,
- 4:50SPEAKER_00: I feel like especially this type of background, they're very well grounded into like
- 5:00SPEAKER_00: The understanding of how things work,
- 5:04SPEAKER_00: but also they're okay with not understanding the stuff and
- 5:09SPEAKER_00: the thing is partially working.
- 5:12SPEAKER_00: They have this mental model that is very iterative.
- 5:15SPEAKER_00: It's like, yeah,
- 5:16SPEAKER_00: it's working kind of like this,
- 5:17SPEAKER_00: but this part, we just had to make it work and then we moved on.
- 5:22SPEAKER_00: Well, other background,
- 5:23SPEAKER_00: I feel like they always have this anxiousness of like, Oh,
- 5:26SPEAKER_00: I should have,
- 5:27SPEAKER_00: I should have known a bit more and like,
- 5:29SPEAKER_00: I don't know exactly what each of the pieces are doing.
- 5:31SPEAKER_00: Yeah, man, I think like this, this type of background is great.
- 5:36SPEAKER_01: Yeah, you have to be comfortable in physics,
- 5:38SPEAKER_01: for sure, with not knowing what the hell is going on.
- 5:41SPEAKER_01: You know,
- 5:41SPEAKER_01: that's something you've got to learn fast,
- 5:43SPEAKER_01: and I think it's something that applies to AI as well.
- 5:46SPEAKER_01: I think a lot of people get caught up in the classic Feynman interview, right?
- 5:52SPEAKER_01: Like, why, why, why?
- 5:53SPEAKER_01: We can always ask why at the next layer down.
- 5:57SPEAKER_01: At some point, it works because it works, and that's good enough, man.
- 5:59SPEAKER_01: Let's do the next thing.
- 6:03SPEAKER_00: Yeah, let's go, let's move.
- 6:04SPEAKER_00: We have another model to train.
- 6:05SPEAKER_00: That's right.
- 6:07SPEAKER_00: This is good, man.
- 6:08SPEAKER_00: Can you give us also a background on Nemotron Lab, like your overall mission?
- 6:13SPEAKER_00: Because I think there's two things.
- 6:14SPEAKER_00: There's the Nemotron Labs and then there's the Coalition.
- 6:17SPEAKER_00: I would just like to have a few words about both.
- 6:21SPEAKER_01: Of course, yeah.
- 6:21SPEAKER_01: So there's actually three things.
- 6:22SPEAKER_01: So there's Nemotron, which is kind of NVIDIA's production model.
- 6:28SPEAKER_01: That's what most people will know of is Nemotron,
- 6:32SPEAKER_01: which is kind of like our marquee language models.
- 6:35SPEAKER_01: This is stuff.
- 6:36SPEAKER_01: It's like Ultra,
- 6:37SPEAKER_01: Super Nano,
- 6:39SPEAKER_01: you know, it extends though to like embedding models and all this other stuff.
- 6:44SPEAKER_01: That's kind of like our,
- 6:45SPEAKER_01: I would say like Neumotron's like our production model shop part of the company.
- 6:50SPEAKER_01: Neumotron Labs is a, we use this word a lot, but it's a bunch of different things.
- 6:57SPEAKER_01: In large part though,
- 6:58SPEAKER_01: it's like, How we get to Nemotron is not straightforward every time, right?
- 7:03SPEAKER_01: I think there's some questions down the road that we'll talk a little bit more about that.
- 7:06SPEAKER_01: But the idea is like,
- 7:08SPEAKER_01: you know,
- 7:09SPEAKER_01: for a given Nemotron,
- 7:12SPEAKER_01: it's like there's not just like,
- 7:13SPEAKER_01: oh, we nail it each time and every step along the way we nail it.
- 7:17SPEAKER_01: And so Nemotron Labs is kind of where we're exploring these techniques that may wind up in
- 7:27SPEAKER_01: So, you know,
- 7:28SPEAKER_01: recent things from the Neotron Labs are like,
- 7:31SPEAKER_01: you know,
- 7:32SPEAKER_01: the fusion models,
- 7:33SPEAKER_01: the text diffusion, and then things like, you know, Star Elastic or the big ones.
- 7:38SPEAKER_01: One for us,
- 7:39SPEAKER_01: of course,
- 7:39SPEAKER_01: was Nemotron H,
- 7:40SPEAKER_01: which is kind of the precursor to the hybrid Mamba family of Nemotrons that we had.
- 7:46SPEAKER_01: So Lab is kind of where the research teams are playing.
- 7:50SPEAKER_01: I use the word playing, but I mean it in a very serious engineering way, not
- 7:58SPEAKER_01: in a silly way.
- 7:59SPEAKER_01: But that's kind of like their playground.
- 8:00SPEAKER_01: And you can kind of...
- 8:02SPEAKER_01: Look at what's happening there and get a sense for the direction
- 8:06SPEAKER_01: that Nemotron proper is going to go.
- 8:07SPEAKER_01: And then finally,
- 8:08SPEAKER_01: the Nemotron Coalition is this kind of implicit understanding
- 8:14SPEAKER_01: that there's no way our teams are literally the best in the world at everything
- 8:19SPEAKER_01: to do this and to bring
- 8:23SPEAKER_01: specifically open American or West models to the frontier
- 8:28SPEAKER_01: and to do that without any help Help or anyone else
- 8:32SPEAKER_01: the ecosystem feels very silly.
- 8:34SPEAKER_01: It feels like it's leaving a lot on a table.
- 8:36SPEAKER_01: One of the things we're going to talk about,
- 8:38SPEAKER_01: MOPD, I think this is like a real life version of MOPD, right?
- 8:44SPEAKER_01: There are so many specialized experts and we want to distill all
- 8:47SPEAKER_01: of their capabilities into a great model.
- 8:51SPEAKER_01: And so that's kind of what the coalition is doing.
- 8:53SPEAKER_01: And I think it's this idea that like,
- 8:56SPEAKER_01: there's no way, like doing this together is way better than doing it not together.
- 9:01SPEAKER_01: And so the coalition's trying to bring as many people as we can together.
- 9:05SPEAKER_00: 100%.
- 9:05SPEAKER_00: And like,
- 9:06SPEAKER_00: I feel like there's like, yes,
- 9:07SPEAKER_00: the coalition,
- 9:07SPEAKER_00: but like just the back and forth that is happening in the,
- 9:10SPEAKER_00: and then the eco that you can see in the open, open weight, larger ecosystem, right?
- 9:19SPEAKER_00: Like the stuff that you guys did with the latent MOE,
- 9:21SPEAKER_00: then you see it back in another paper, another model, like right afterward.
- 9:27SPEAKER_00: And you can see this exchange of information.
- 9:29SPEAKER_00: And even when you talk to the different labs,
- 9:32SPEAKER_00: you see that they're studying what the others are doing.
- 9:35SPEAKER_00: And you get all of these benefits.
- 9:37SPEAKER_00: And everybody's life is a bit easier.
- 9:39SPEAKER_00: Every time there's an open model launch,
- 9:43SPEAKER_00: everybody that I know that is working at the open lab, it's like almost a party.
- 9:48SPEAKER_00: It's going to be the Kimi K3 day or it's going to be the Nemo 2 Ultra day
- 9:52SPEAKER_00: and they'll just look at the stuff and they get ideas and you see them racing.
- 9:57SPEAKER_00: And then they now have more information about how to do their own stuff and
- 10:02SPEAKER_00: to make their product better.
- 10:04SPEAKER_00: Which I find it's so cool, man.
- 10:07SPEAKER_00: It's rare that you're going to see that sort of stuff anywhere.
- 10:11SPEAKER_00: Like, okay,
- 10:11SPEAKER_00: the competitor or like the other people are launching stuff and then you're excited
- 10:16SPEAKER_00: for that and you're going to go and look at this stuff, right?
- 10:19SPEAKER_00: But we're in such a special place here where I don't,
- 10:23SPEAKER_00: I feel like we get the best of the open source and then like
- 10:27SPEAKER_00: the ML community all together.
- 10:30SPEAKER_01: Yeah, I mean, it's so deeply true what you're saying.
- 10:32SPEAKER_01: I mean, and I think it's something that we're really happy to be, like, supportive of.
- 10:37SPEAKER_01: And I think, you know, we had, like, the letter apocalypse the last few days or whatever.
- 10:41SPEAKER_01: But, like, I think this is part of why, you know, part of why is that.
- 10:46SPEAKER_01: It's important for people to hear from the people working at these companies that,
- 10:50SPEAKER_01: like,
- 10:50SPEAKER_01: they're dedicated to this thing,
- 10:52SPEAKER_01: you know,
- 10:52SPEAKER_01: and it's important to see the ecosystem kind of,
- 10:56SPEAKER_01: you know, banding together across all players, right?
- 11:01SPEAKER_01: So I think that's super important.
- 11:04SPEAKER_01: But also, like, it is Kimmy Day.
- 11:07SPEAKER_01: And, like, today was Inkling Day, right?
- 11:09SPEAKER_01: Like, Inkling Small.
- 11:10SPEAKER_01: So it's just, like,
- 11:12SPEAKER_01: You can't not read some of these tech reports and immediately say,
- 11:15SPEAKER_01: oh, but what if we did that?
- 11:18SPEAKER_01: Like scalar reasoning effort from Inkling, right?
- 11:21SPEAKER_01: Why doesn't Mimotron have scaling reasoning effort, man?
- 11:24SPEAKER_01: That would be better than discrete reasoning effort.
- 11:27SPEAKER_01: We should do that.
- 11:29SPEAKER_01: This is the world that I want to live in,
- 11:33SPEAKER_01: where people read our stuff and And so like, oh, great idea.
- 11:36SPEAKER_01: And then they make it even better.
- 11:38SPEAKER_01: And then we get to say, oh, what a great idea, right?
- 11:40SPEAKER_01: Like, I feel like we all progressed so fast and together in that world.
- 11:45SPEAKER_00: It's really great.
- 11:46SPEAKER_00: And like,
- 11:46SPEAKER_00: one thing I like also is that you could see,
- 11:49SPEAKER_00: however,
- 11:49SPEAKER_00: that like each of the different labs has their own kind of taste
- 11:55SPEAKER_00: and like their own direction that they're following.
- 11:59SPEAKER_00: And in you guys' case,
- 12:00SPEAKER_00: I really liked,
- 12:01SPEAKER_00: like throughout everything I was reading,
- 12:04SPEAKER_00: Speed is something that is so important and it shows everywhere.
- 12:09SPEAKER_00: If you look at the architecture,
- 12:10SPEAKER_00: it just pops up again and again that it was made for being able to be very fast and
- 12:17SPEAKER_00: then being able to be used into these agentic system and agentic harnesses
- 12:23SPEAKER_00: that can be run for days.
- 12:26SPEAKER_00: But you see that in all of the different labs in their own kind of direction.
- 12:32SPEAKER_00: Like incorporate some of the stuff from the other labs that are like maybe going
- 12:35SPEAKER_00: after something different, right?
- 12:38SPEAKER_00: But they all are able to share a bit of this.
- 12:41SPEAKER_00: It's really cool.
- 12:43SPEAKER_01: Yeah, it is super cool to see.
- 12:46SPEAKER_00: I just want you to explain this ethos of like a fast AI is smart AI, right?
- 12:53SPEAKER_00: I think, like,
- 12:53SPEAKER_00: I see this in your research page that says,
- 12:56SPEAKER_00: like, NVDA comes to AI as an accelerated computing company.
- 13:00SPEAKER_00: We believe that faster AI is smarter AI,
- 13:03SPEAKER_00: which I find beautiful, and I kind of 100% agree with that.
- 13:06SPEAKER_00: But can you, like, just break it down in your own word, like, what do you mean in practice here?
- 13:13SPEAKER_01: Yeah, faster models are smarter models is the way I like to say it.
- 13:16SPEAKER_01: It's the way that I first heard it said, and I just can't get away from it in such a...
- 13:21SPEAKER_01: Like you said, there is beauty to it, right?
- 13:22SPEAKER_01: Like faster models are smarter models.
- 13:25SPEAKER_01: But the idea is it's very straightforward.
- 13:28SPEAKER_01: Models should be fast, right?
- 13:31SPEAKER_01: Like, I mean, intelligence, It should be fast.
- 13:38SPEAKER_01: I think it's not good enough just for something to be smart, especially now.
- 13:44SPEAKER_01: Maybe in 2023,
- 13:45SPEAKER_01: in the peak of the deep research days,
- 13:48SPEAKER_01: or when we were like, oh, the model took 13 minutes, and I'm so happy about that.
- 13:55SPEAKER_01: Now it's definitely not.
- 13:59SPEAKER_01: We want it to be done faster and faster.
- 14:04SPEAKER_01: Almost every kind of work that a model might do is in some way likely iterative.
- 14:11SPEAKER_01: And the faster you can do iterations, the faster you can get to the final result.
- 14:19SPEAKER_01: I like to think about it as like,
- 14:20SPEAKER_01: so NemoTron Ultra compared to like GPT-5,
- 14:22SPEAKER_01: 6 or something like that,
- 14:23SPEAKER_01: like, is, they're different models in terms of their intelligence profiles.
- 14:28SPEAKER_01: But like,
- 14:28SPEAKER_01: if what you need to do is to hammer a task until it's done,
- 14:32SPEAKER_01: and that task is reasonably Like, Ultra can just get it done faster.
- 14:37SPEAKER_01: Even if it takes more iterations than a smarter model,
- 14:41SPEAKER_01: it will crush those iterations faster and therefore get it done faster.
- 14:47SPEAKER_01: You know, test time compute is still a big part of how we make models smarter, right?
- 14:53SPEAKER_01: It's still a big...
- 14:55SPEAKER_01: Kind of scaling access.
- 14:56SPEAKER_01: The faster we are, you know, say the best of end, right?
- 15:01SPEAKER_01: The larger ends we can have in the same period of time on the same endpoint.
- 15:05SPEAKER_01: So I think faster models,
- 15:06SPEAKER_01: smarter models,
- 15:07SPEAKER_01: this concession of the fact that like, We live in a very iterative world.
- 15:12SPEAKER_01: We solve very iterative problems.
- 15:15SPEAKER_01: And there's a lot of work that needs to be done for any given prompt.
- 15:20SPEAKER_01: And the faster you can complete that work, the better.
- 15:22SPEAKER_01: And so as we'll talk about,
- 15:25SPEAKER_01: About Neotron, I mean, almost every decision is a concession to speed, right?
- 15:32SPEAKER_01: Speed being like throughput or interactivity or latency.
- 15:36SPEAKER_01: There's a bunch of different kinds of ways you can talk about speed,
- 15:39SPEAKER_01: but the whole model is built to be fast, and we think that's incredibly important.
- 15:46SPEAKER_00: Even the pipeline, right?
- 15:47SPEAKER_00: If you think about it,
- 15:48SPEAKER_00: this is what I mean by like,
- 15:51SPEAKER_00: The ethos of this is showing kind of everywhere because the way
- 15:57SPEAKER_00: that you guys sequenced also some of the longer agentic RL rollout by keeping some
- 16:05SPEAKER_00: of the other stuff there that was maybe a bit smaller at first and
- 16:09SPEAKER_00: then you do these ones that are bigger.
- 16:11SPEAKER_00: Or like the async RL.
- 16:13SPEAKER_00: I feel like it shows back again and again that Like you have,
- 16:17SPEAKER_00: yes, this ethos at the model level where it's true.
- 16:21SPEAKER_00: Like if I'm coding and then my model is just like going and
- 16:25SPEAKER_00: then just asking me feedback straight away,
- 16:28SPEAKER_00: I'm going to stay in the flow and then I'm going to run and then do the stuff,
- 16:30SPEAKER_00: right?
- 16:31SPEAKER_00: If it takes like two hours, I don't even know what I ask, right?
- 16:34SPEAKER_00: And it's going in the wrong direction and stuff.
- 16:36SPEAKER_00: So like that iteration is there, yes.
- 16:38SPEAKER_00: But you see that even more like at the whole loop level.
- 16:44SPEAKER_00: Just like a thinking machine,
- 16:47SPEAKER_00: the time between Inkling and Inkling Small was like,
- 16:50SPEAKER_00: I don't know, something like four weeks, something like this.
- 16:54SPEAKER_00: Having this iteration speed baked into how you're iterating over
- 17:01SPEAKER_00: the model deployment and the model construction,
- 17:05SPEAKER_00: I also think that is something that is making the AI much smarter, right?
- 17:09SPEAKER_00: There's a lot of good ideas.
- 17:10SPEAKER_00: Like you said,
- 17:11SPEAKER_00: you're reading the Kimi-Ki-Tree paper or the Inkling paper,
- 17:14SPEAKER_00: and then you look at the stuff and you're like, yeah, we should do that too, right?
- 17:17SPEAKER_00: And if you have a good loop,
- 17:18SPEAKER_00: you're going to get smarter model straight away,
- 17:20SPEAKER_00: but you have to be able to kind of integrate them pretty fast.
- 17:24SPEAKER_00: Anyway.
- 17:25SPEAKER_00: Yeah, fast is everywhere.
- 17:26SPEAKER_00: Very pumped.
- 17:28SPEAKER_00: Yeah.
- 17:28SPEAKER_01: Yeah, like NVF before pre-training is fast, right?
- 17:32SPEAKER_01: Like, you know, moving to like pivot RL, what you described, is fast.
- 17:39SPEAKER_01: Like, all of these things help you do all the stuff you want to do, but in less time.
- 17:44SPEAKER_01: I mean, at the end of the day, we're all compute constrained, right?
- 17:49SPEAKER_01: Like, true, but we're also time constrained.
- 17:51SPEAKER_01: And like, you can run...
- 17:54SPEAKER_01: Even with the same compute,
- 17:56SPEAKER_01: if you can run more experiments in the same delta T or whatever,
- 18:03SPEAKER_01: then you're going to wind up with better systems at the end of the day, likely.
- 18:09SPEAKER_01: The other thing, too, is search space...
- 18:16SPEAKER_01: Tasks, okay?
- 18:17SPEAKER_01: So this is like a lot of actually science and like a lot of work is like exploring
- 18:23SPEAKER_01: a search space, right?
- 18:25SPEAKER_01: And like that exploration being fast means you find at least like local resources.
- 18:31SPEAKER_01: Maximum is faster.
- 18:33SPEAKER_01: And so I think the tightness of your loop, again,
- 18:38SPEAKER_01: like you said,
- 18:39SPEAKER_01: from when I'm talking to a model on my keyboard all the way to when
- 18:43SPEAKER_01: a model is being trained is extraordinarily important.
- 18:48SPEAKER_01: Something I want to say as well,
- 18:50SPEAKER_01: when you're sitting with a terminal,
- 18:51SPEAKER_01: there are kinds of models that the thing you do is slash goal and
- 18:54SPEAKER_01: then you walk away.
- 18:56SPEAKER_01: And I think those are awesome models.
- 18:58SPEAKER_01: But there's also models that I want to be able to actually be in the loop As a human
- 19:03SPEAKER_01: at my keyboard, like doing work.
- 19:07SPEAKER_01: And you need the model to be fast for that.
- 19:09SPEAKER_01: It can't be slow.
- 19:10SPEAKER_01: And so that's another like interactivity,
- 19:13SPEAKER_01: you know,
- 19:13SPEAKER_01: literally not as in like the,
- 19:16SPEAKER_01: you know,
- 19:17SPEAKER_01: the performance measurement,
- 19:19SPEAKER_01: but like how much can I interact with this thing in front of me?
- 19:23SPEAKER_00: But it's funny because like I work with a lot of enterprise, right?
- 19:27SPEAKER_00: Where like literally I'm going in there and I'm trying to implement the AI project they have to.
- 19:32SPEAKER_00: To do with my team.
- 19:34SPEAKER_00: And more often than not,
- 19:37SPEAKER_00: having something be able to move quickly opens up so much that you can do.
- 19:43SPEAKER_00: And then the subsequent adoption by the user is much faster if the thing is very reactive.
- 19:55SPEAKER_00: They get this like, okay, wow, it's something I can use now and it can support me.
- 20:03SPEAKER_00: Every time it's a bit slower,
- 20:06SPEAKER_00: the feedback,
- 20:07SPEAKER_00: even for the same amount of output quality,
- 20:11SPEAKER_00: it has a big impact,
- 20:12SPEAKER_00: especially if we're talking about this diffusion across the market, right?
- 20:19SPEAKER_00: Yep.
- 20:21SPEAKER_00: Strongly agree.
- 20:22SPEAKER_00: Perfect.
- 20:23SPEAKER_00: I think we can go into your presentation.
- 20:25SPEAKER_00: Excited for this?
- 20:26SPEAKER_00: Sure.
- 20:27SPEAKER_00: Too much question?
- 20:29SPEAKER_01: So basically, this particular presentation was more specifically focused on...
- 20:35SPEAKER_01: The RL side of the house.
- 20:36SPEAKER_01: So there's going to be a lot of things on the RL side of the house.
- 20:39SPEAKER_01: But I think your questions more broadly cover Nemotron generally.
- 20:44SPEAKER_01: So this is the thing.
- 20:47SPEAKER_01: And then we're not going to worry about this agenda.
- 20:50SPEAKER_01: I'm going to very briefly go over this stuff
- 20:51SPEAKER_01: because I think probably anyone watching your video knows it.
- 20:54SPEAKER_01: But like we've been doing this forever.
- 20:57SPEAKER_01: So this is just kind of a slide where we get to brag
- 20:58SPEAKER_01: about how long we've been doing this and how big our models were even back then.
- 21:03SPEAKER_01: So this is like from The OG game days of, I think we still call it NLP at this point even.
- 21:13SPEAKER_01: So this is like your autocomplete models.
- 21:17SPEAKER_01: This is the base model, we'd call it now, right?
- 21:19SPEAKER_01: Then we were like, nah, that's garbage.
- 21:21SPEAKER_01: We should, you know, autocomplete's good, but like we need some way to interact with it.
- 21:26SPEAKER_01: And then the Emergent Behaviors paper came out and suddenly Instructune Took off to the moon.
- 21:31SPEAKER_01: And so then we did that for a while.
- 21:34SPEAKER_01: And then we said, hey,
- 21:36SPEAKER_01: what if instead of just input-output,
- 21:39SPEAKER_01: we gave the model some kind of scratch pad to take down notes
- 21:43SPEAKER_01: about what it was thinking and how it was thinking through processes.
- 21:46SPEAKER_01: And what we can do is we can kind of trick the model into producing outputs that
- 21:50SPEAKER_01: through in context learning, you know, let the model cook up a better response.
- 21:56SPEAKER_01: So the idea is like we fill the context window up with a bunch of reasoning tokens.
- 22:00SPEAKER_01: And then some combination of those reasoning tokens allow the model
- 22:04SPEAKER_01: through context learning to produce a final response was like pretty dope.
- 22:08SPEAKER_01: And then we were like,
- 22:09SPEAKER_01: what if we also let the model with JSON blobs and we sent those JSON blobs to And
- 22:17SPEAKER_01: the code was executed on those JSON blobs,
- 22:21SPEAKER_01: and now we're using tools, and now we have many different back and forths.
- 22:28SPEAKER_01: So instead of just me, model,
- 22:29SPEAKER_01: me, model, it's like me, tool, or me, model, tool, tool, tool, model, me, right?
- 22:36SPEAKER_01: Like, this is the idea.
- 22:38SPEAKER_01: And then finally...
- 22:40SPEAKER_01: We have this word agentic.
- 22:42SPEAKER_01: Agentic means a lot of things to some people and almost nothing to others.
- 22:48SPEAKER_01: But the basic idea is like this is going to run in some kind of loop and
- 22:51SPEAKER_01: that loop is going to call the model over and over again based on some conditions
- 22:57SPEAKER_01: until the model has in some way completed work as defined either by itself
- 23:03SPEAKER_01: or some extra model process.
- 23:06SPEAKER_01: So that's like the history lesson.
- 23:07SPEAKER_01: There you go.
- 23:08SPEAKER_01: What does it mean for Model B-Agentic?
- 23:10SPEAKER_01: We've already talked about it.
- 23:11SPEAKER_01: Boom.
- 23:12SPEAKER_01: Good stuff.
- 23:12SPEAKER_01: So for Neymotron 3 Super, this is kind of how we...
- 23:17SPEAKER_01: So basically, post-training.
- 23:19SPEAKER_01: So you'll see, like, we started with a base model...
- 23:21SPEAKER_01: With 1 million context length.
- 23:23SPEAKER_01: So this is a pre-trained model checkpoint that went through CLE,
- 23:29SPEAKER_01: which is like context length extension.
- 23:31SPEAKER_01: So taking it from the native length to this 1 million.
- 23:35SPEAKER_01: And then we did a bunch of SFT on it.
- 23:38SPEAKER_01: We would love for this number to be as close to zero as possible.
- 23:43SPEAKER_01: The only reason to do SFT right now is because we need it to make RL good.
- 23:50SPEAKER_01: That's like...
- 23:52SPEAKER_01: Kind of a joke, but it's not as much of a joke as you think it would be.
- 23:55SPEAKER_01: I think our post-training lead would be happiest if we could do zero SFT,
- 24:00SPEAKER_01: but alas, we still have to do some.
- 24:02SPEAKER_01: And then we kind of do like RLVR.
- 24:04SPEAKER_01: RLVR is like, you all know and love it.
- 24:08SPEAKER_01: Reinforced Learning Verifiable Rewards.
- 24:09SPEAKER_01: The idea is like, we have the model do stuff and then we, you know, it gets better.
- 24:14SPEAKER_01: We're going to talk about it in later slides.
- 24:16SPEAKER_01: But you'll notice there's like these stages, right?
- 24:18SPEAKER_01: So...
- 24:19SPEAKER_01: Instead of doing this in one big batch,
- 24:26SPEAKER_01: we're going to phase out and roll out the RL,
- 24:31SPEAKER_01: and we're even going to break it down further outside of this box.
- 24:36SPEAKER_01: The reason we're going to do this is...
- 24:39SPEAKER_01: There's so many reasons,
- 24:40SPEAKER_01: but one of the reasons we're going to do this is that these kinds of envs
- 24:44SPEAKER_01: that you see grouped here share similar...
- 24:48SPEAKER_01: ITL, OS, or ISL, OSL characteristics, right?
- 24:53SPEAKER_01: So, like,
- 24:53SPEAKER_01: what that means, like,
- 24:55SPEAKER_01: you kind of expect that they're going to start and finish near-ish the same times,
- 24:59SPEAKER_01: and that their end-to-end latency is going to be near-ish to the same times, right?
- 25:05SPEAKER_01: We're grouping kind of like environments.
- 25:07SPEAKER_01: And the reason we group like environments is because we want to spend as little time
- 25:11SPEAKER_01: as possible in downtime.
- 25:13SPEAKER_01: There's going to be a slide that kind of goes into that a little bit better.
- 25:16SPEAKER_01: Once we do all this multi-environment,
- 25:18SPEAKER_01: and multi-environment here just means like instead of a group
- 25:21SPEAKER_01: of rollouts getting scored by one environment,
- 25:23SPEAKER_01: they're going to be scored by many environments.
- 25:25SPEAKER_01: And then we're going to do some mixing to determine like overall What we want
- 25:30SPEAKER_01: to learn,
- 25:30SPEAKER_01: but the idea is like that we're going to talk about why we do that
- 25:33SPEAKER_01: a little bit later.
- 25:34SPEAKER_01: And then we go to SWE focused RLVR.
- 25:36SPEAKER_01: This is by itself because it takes for damn ever.
- 25:38SPEAKER_01: If you've ever trained a model on SWE tasks using SWE environments,
- 25:42SPEAKER_01: you know that they,
- 25:44SPEAKER_01: and SWE here just means software engineering,
- 25:46SPEAKER_01: you know that they take for damn ever, right?
- 25:48SPEAKER_01: Like it is just true that if we like,
- 25:52SPEAKER_01: if we like Made any of these tasks wait for this task, right?
- 25:57SPEAKER_01: Like it would be the least efficient use of our time possible.
- 26:01SPEAKER_01: And we want to be as efficient as possible.
- 26:03SPEAKER_01: And then we still do our LHF and then some MTP and some,
- 26:09SPEAKER_01: this is basically to make the model protect many tokens.
- 26:12SPEAKER_01: But, you know, we add some extra bells and whistles for Ultra.
- 26:17SPEAKER_01: Uh, SFT, uh, why do you do it?
- 26:19SPEAKER_01: Uh, because you need to, uh, how do we do it?
- 26:22SPEAKER_01: Huge SDGs, synthetic data generation pipelines.
- 26:25SPEAKER_01: That's what it's all about.
- 26:26SPEAKER_01: Uh, we have a big pipeline.
- 26:28SPEAKER_01: It does pipeline stuff.
- 26:29SPEAKER_01: I'm not gonna spend too much time on this.
- 26:31SPEAKER_01: SFT, you guys kind of know the drill, uh, We need to warm the model up, right, for RL.
- 26:38SPEAKER_01: You'll see here that, like, our data blend is very specific.
- 26:42SPEAKER_01: People sometimes ask, like, what is Demotron good at?
- 26:44SPEAKER_01: And I'm like, you can check out the tech report.
- 26:47SPEAKER_01: And the tech report has this wheel chart.
- 26:49SPEAKER_01: And basically, the model is good at these tasks that you see data piles for.
- 26:57SPEAKER_01: You can see,
- 26:57SPEAKER_01: in fact,
- 26:57SPEAKER_01: exactly how the team thought about capabilities by looking at the data blend, right?
- 27:02SPEAKER_01: So there you go.
- 27:04SPEAKER_01: We have big pipelines.
- 27:06SPEAKER_01: We do SFT.
- 27:09SPEAKER_01: Then we get to the good part, which is RL.
- 27:11SPEAKER_01: I'm not going to go into this slide, but will you share the slides to people after this?
- 27:18SPEAKER_00: You're right.
- 27:19SPEAKER_00: I could share the slides, yeah.
- 27:20SPEAKER_01: Okay, then fuck yeah.
- 27:21SPEAKER_01: So you can look at these slides if you want to take your time through them a little bit more.
- 27:24SPEAKER_01: But the basic idea here is like...
- 27:29SPEAKER_01: We can just tell when something is right or wrong.
- 27:32SPEAKER_01: Okay, there you go.
- 27:33SPEAKER_01: So what we're going to do is we're just going to verify
- 27:39SPEAKER_01: the model did something good or did something bad,
- 27:43SPEAKER_01: and we're going to give it cookies if it did good,
- 27:45SPEAKER_01: and we're not going to give it any cookies if it did bad.
- 27:47SPEAKER_01: This is actually...
- 27:52SPEAKER_01: Just a great thing to do, and it works pretty much all the time, every time.
- 27:56SPEAKER_01: Now, this is for our LVR.
- 27:59SPEAKER_01: You'll see, though, we also...
- 28:03SPEAKER_01: We also want to have good verifiers.
- 28:07SPEAKER_01: We also want to have many verifiers.
- 28:09SPEAKER_01: So we're going to use this idea of Gerpo.
- 28:13SPEAKER_01: We use a bunch of different algorithms across all of the models,
- 28:16SPEAKER_01: which we'll talk about when we talk a little bit more about MOPD.
- 28:18SPEAKER_01: But the idea is Gerpo is good because
- 28:24SPEAKER_01: it's effectively just a great way to get...
- 28:29SPEAKER_01: Decent-ish signal quickly.
- 28:33SPEAKER_01: And what we want to do at the end of the day, again, is train models fast.
- 28:36SPEAKER_01: We really don't want to mess about.
- 28:41SPEAKER_01: Okay, so why do we use multi-environment reinforcement learning instead of just one environment?
- 28:47SPEAKER_01: The reality is, like,
- 28:49SPEAKER_01: we want to train on many,
- 28:51SPEAKER_01: many environments,
- 28:52SPEAKER_01: and doing them sequentially would just be like a silly waste of time.
- 28:56SPEAKER_01: And also, it's like...
- 28:58SPEAKER_01: So this is, you might hear this like Boogeyman, right?
- 29:00SPEAKER_01: Reward hacking, like, oh man, the model's going to like You know, do some bad stuff.
- 29:06SPEAKER_01: The reality is, like,
- 29:07SPEAKER_01: if you use many, many environments, it just becomes more complex to reward hack.
- 29:12SPEAKER_01: Like, it's not likely that the same hack is going to work across as many environments.
- 29:17SPEAKER_01: Does this mean there is no reward hacking?
- 29:19SPEAKER_01: Hell no, right?
- 29:20SPEAKER_01: Does this mean that, like, it's like this is a silver bullet and it goes away?
- 29:24SPEAKER_01: Definitely not.
- 29:25SPEAKER_01: But at least when you have a lot of environments,
- 29:28SPEAKER_01: you can be sure that the hacking that's happening has to be,
- 29:31SPEAKER_01: like, Relatively triageable.
- 29:34SPEAKER_01: So like you can identify how it's occurring and therefore mitigate it
- 29:41SPEAKER_01: a little bit easier because it has to be something that works
- 29:43SPEAKER_01: across multiple environments.
- 29:45SPEAKER_01: And the behavior that you get out of it is usually quite cursed.
- 29:51SPEAKER_01: There you go.
- 29:52SPEAKER_01: This is the idea.
- 29:52SPEAKER_01: We got some numbers if you like numbers.
- 29:55SPEAKER_01: And then this is a great slide that kind of talks about How damn challenging this is
- 30:03SPEAKER_01: in actual reality.
- 30:04SPEAKER_01: You're also going to see on this slide this little red word called CPU.
- 30:10SPEAKER_01: So I work at NVIDIA.
- 30:13SPEAKER_01: We do GPUs really well.
- 30:15SPEAKER_01: And you would think, like, surely a lot of this stuff must be, like, purely GPU-bound.
- 30:20SPEAKER_01: Hell no, dude.
- 30:21SPEAKER_01: CPU, especially when you get into these very resource-intensive tasks,
- 30:25SPEAKER_01: like SWE tasks,
- 30:27SPEAKER_01: which have a lot of execution, like, CPU is actually
- 30:34SPEAKER_01: lot of bottleneck, right?
- 30:35SPEAKER_01: Like, this is pretty well-known across the industry,
- 30:39SPEAKER_01: but I think it's important to just double-click on it.
- 30:42SPEAKER_01: Like, this is a...
- 30:44SPEAKER_01: Large-scale RL is...
- 30:48SPEAKER_01: Definitely a lot about training and understanding how training works,
- 30:52SPEAKER_01: but it is a bigger infrastructure challenge.
- 30:55SPEAKER_01: In fact,
- 30:55SPEAKER_01: I would say it's like the biggest infrastructure challenge because you have
- 31:00SPEAKER_01: so many moving parts and you need to signal as fast as possible,
- 31:04SPEAKER_01: but you don't want to sacrifice the robustness of your environment.
- 31:09SPEAKER_01: So you wind up in this world where we're in scheduling hell as long as possible.
- 31:15SPEAKER_01: Someone put a comment, it looks like on the screen, Talks about catastrophic forgetting.
- 31:21SPEAKER_01: Yeah, catastrophic forgetting is always a problem.
- 31:25SPEAKER_01: I don't have any specific information that sequential RL is more prone to
- 31:31SPEAKER_01: or less prone to it, but intuitively I can see that that makes sense.
- 31:35SPEAKER_01: There you go.
- 31:35SPEAKER_01: Latency is bad.
- 31:39SPEAKER_01: After we do the normal RL, let's call it, then we're going to do SWE RL.
- 31:45SPEAKER_01: And again, we are
- 31:50SPEAKER_01: doing this because isolating it just lets us do more at once in the long run.
- 31:57SPEAKER_01: That's it.
- 31:58SPEAKER_01: Pivot RL is something that we're really happy about.
- 32:04SPEAKER_01: So Pivot RL is basically like
- 32:08SPEAKER_01: So instead of just raw SFTing,
- 32:12SPEAKER_01: what we're going to do is we're going to find the point
- 32:14SPEAKER_01: where something got challenging, and then we're going to do RL from that point.
- 32:21SPEAKER_01: Instead of doing pure SFT the whole way,
- 32:25SPEAKER_01: which is super cheap,
- 32:27SPEAKER_01: but ineffective,
- 32:31SPEAKER_01: ineffective is the wrong term here,
- 32:33SPEAKER_01: less generalizable,
- 32:34SPEAKER_01: what we're going to do is we're going We're going to basically have traces
- 32:39SPEAKER_01: that we know where the model fucked up.
- 32:42SPEAKER_01: We're going to start from there, right?
- 32:44SPEAKER_01: So we're going to be like, hey,
- 32:44SPEAKER_01: you should learn all the parts that were good,
- 32:46SPEAKER_01: and then when we get to the part that you fucked up,
- 32:48SPEAKER_01: we're going to let you RL your way to the So this is the idea of Pivot RL.
- 32:54SPEAKER_01: It just saves a lot of tokens,
- 32:56SPEAKER_01: a lot of time,
- 32:56SPEAKER_01: and it is pound for pound more effective,
- 33:00SPEAKER_01: especially because, you know, we really want to be able to do like really...
- 33:07SPEAKER_01: Good curriculums for RL, right?
- 33:09SPEAKER_01: And Pivot RL helps us kind of control that curriculum knob very, very finely.
- 33:15SPEAKER_01: Then you're going to see a little bit about GenRM.
- 33:19SPEAKER_01: This is something I think people forgot about.
- 33:21SPEAKER_01: So RLH RLHF used to be this huge thing.
- 33:23SPEAKER_01: Everyone's RLHFing all the damn time, right?
- 33:26SPEAKER_01: It was like, oh, we need the model to be so good at talking to people.
- 33:31SPEAKER_01: And RLHF is like this magic thing that does that.
- 33:34SPEAKER_01: And we just do like a little bit of it all the time now.
- 33:37SPEAKER_01: Um, this has changed a little bit for, uh, for,
- 33:41SPEAKER_01: for how we approach things with,
- 33:43SPEAKER_01: with Ultra, but, uh,
- 33:44SPEAKER_01: the same idea is true,
- 33:46SPEAKER_01: which is like,
- 33:46SPEAKER_01: we're just going to do a little bit of,
- 33:49SPEAKER_01: of kind of RM work all the time,
- 33:51SPEAKER_01: uh, reward model, which is basically like this thing that captures, uh,
- 33:58SPEAKER_01: Things that people care about very well that you can't really verify.
- 34:03SPEAKER_01: It's very hard to look at a sentence and say,
- 34:05SPEAKER_01: oh, most people will like that sentence in a verifiable way.
- 34:09SPEAKER_01: But it's really easy to build a reward model that says,
- 34:12SPEAKER_01: this is a more likely... People are more likely to like this thing than not.
- 34:20SPEAKER_01: And then finally...
- 34:23SPEAKER_01: Wind it off, I'll see Gerpo.
- 34:25SPEAKER_01: You're going to notice we have these bars.
- 34:28SPEAKER_01: We have steps.
- 34:29SPEAKER_01: We have this sync, which is like our synchronous approach.
- 34:36SPEAKER_01: And then we have our async approach.
- 34:40SPEAKER_01: You're going to say, hey, wait a minute, man.
- 34:45SPEAKER_01: These bars look really bad for async, right?
- 34:48SPEAKER_01: It reaches where it needs to go, but it takes much longer.
- 34:52SPEAKER_01: Faked you out because this is by step.
- 34:55SPEAKER_01: Okay, so we're cheating a little bit.
- 34:57SPEAKER_01: We're fudging the numbers.
- 34:58SPEAKER_01: When you look at the training time by minutes,
- 35:00SPEAKER_01: which is basically how long is it actually taking,
- 35:04SPEAKER_01: this technique is massively beneficial.
- 35:08SPEAKER_01: It doesn't really... Again, it's not really like...
- 35:14SPEAKER_01: It's not really better in the long run, right?
- 35:18SPEAKER_01: So these things should converge, but it's faster.
- 35:21SPEAKER_01: Because it's faster, it's faster wall clock, right?
- 35:24SPEAKER_01: So because it's faster, therefore, it is better.
- 35:28SPEAKER_01: Every technique that you see us produce,
- 35:31SPEAKER_01: you're going to find somewhere deep in the heart that it's faster.
- 35:34SPEAKER_01: Then finally, of course, Reasoning Control.
- 35:36SPEAKER_01: I mean, this is just a thing you have to do.
- 35:40SPEAKER_01: People want to be able to control how many tokens the model shifts out.
- 35:44SPEAKER_01: So we do it.
- 35:45SPEAKER_01: And we do this during RL.
- 35:47SPEAKER_01: And it's pretty straightforward.
- 35:49SPEAKER_01: So this is discrete reasoning control,
- 35:53SPEAKER_01: which means that there are levels that we offer you,
- 35:55SPEAKER_01: which is reasoning off, reasoning on, and low effort reasoning.
- 36:00SPEAKER_01: For ultra, it's actually reasoning off, reasoning on, and medium effort reasoning.
- 36:07SPEAKER_01: Very small difference, but...
- 36:09SPEAKER_01: It does exist.
- 36:11SPEAKER_01: You can use runtime budget control as well with Nemotron,
- 36:14SPEAKER_01: which means you can just tell the model when you want it
- 36:19SPEAKER_01: to stop generating reasoning tokens and move on to generating normal tokens,
- 36:23SPEAKER_01: and it hasn't been trained with that behavior.
- 36:24SPEAKER_01: So the behavior is not undefined.
- 36:26SPEAKER_01: That means you can set very flexible reasoning budgets
- 36:29SPEAKER_01: if you have specific constraints that you're working within.
- 36:32SPEAKER_01: But of course, we also have the kind of typicals.
- 36:38SPEAKER_01: Then you can see big number.
- 36:40SPEAKER_01: So this is how many GPUs we used for each.
- 36:46SPEAKER_01: Lots of GPUs.
- 36:48SPEAKER_01: I think most people would look at this and think lots of GPUs,
- 36:51SPEAKER_01: but I think if you read a lot of big model tech reports,
- 36:54SPEAKER_01: you might look at this and say, that's not many GPUs.
- 36:58SPEAKER_01: I would say it's a lot of GPUs for a lot of people, right?
- 37:01SPEAKER_01: But I think for people building models like this, it's surprisingly few GPUs.
- 37:06SPEAKER_01: That's just because our teams are trapped.
- 37:08SPEAKER_01: Yeah.
- 37:09SPEAKER_01: If you want to learn more about the software, you can do it.
- 37:12SPEAKER_01: Anytime we do something,
- 37:13SPEAKER_01: we talk about something,
- 37:14SPEAKER_01: PivotRL or Async one-off,
- 37:20SPEAKER_01: there's a library that has that code and you can go do it yourself.
- 37:23SPEAKER_01: That's my favorite part.
- 37:24SPEAKER_01: So that's my presentation.
- 37:25SPEAKER_01: That's my little...
- 37:40SPEAKER_01: Spil, thanks for letting me do that.
- 37:43SPEAKER_01: This is from our post-training lead.
- 37:45SPEAKER_01: There is no wall in post-training.
- 37:48SPEAKER_01: Multi-environment RL is good,
- 37:50SPEAKER_01: and infrastructure is really what the challenge is when you're trying
- 37:53SPEAKER_01: to do this big boy style.
- 37:57SPEAKER_00: Yeah.
- 37:58SPEAKER_00: This is great.
- 37:59SPEAKER_00: Thanks for the crash course over here.
- 38:03SPEAKER_00: There's Ahmad that is saying on Twitter, thanks for sharing all this, Chris.
- 38:09SPEAKER_00: That was really cool.
- 38:11SPEAKER_00: I have a whole bunch of questions.
- 38:13SPEAKER_00: First off,
- 38:14SPEAKER_00: a realization that a lot of the problems I feel in the current era
- 38:22SPEAKER_00: of scaling these models are really mainly just infrastructure problems.
- 38:29SPEAKER_00: Like literally trying to juice up as much as possible.
- 38:32SPEAKER_00: If you think about it,
- 38:32SPEAKER_00: I think RL,
- 38:35SPEAKER_00: The point is to do the stuff fast so that you can pump in more tokens in there and
- 38:39SPEAKER_00: then learn a thing, right?
- 38:40SPEAKER_00: Because the trade-off that you're making when you're doing like off policy is
- 38:44SPEAKER_00: that you can shove the stuff,
- 38:46SPEAKER_00: like fit the stuff in your grid so that you can still use them and then jam them
- 38:54SPEAKER_00: into your model so you can learn while knowing that like most
- 38:58SPEAKER_00: of these environments are CPU bound and they're going to take a bit of time, right?
- 39:03SPEAKER_00: So if you're off by one gradient step,
- 39:06SPEAKER_00: it's not that bad and it'll give you time for the system to kind of go
- 39:10SPEAKER_00: and do their thing.
- 39:12SPEAKER_00: But a lot of the problem seems to be related to this, right?
- 39:16SPEAKER_00: Do you think that like...
- 39:18SPEAKER_00: How bad do you think it will get?
- 39:20SPEAKER_00: Because right now we have 400B,
- 39:24SPEAKER_00: which is kind of big,
- 39:25SPEAKER_00: but we're getting into the trillion and then it just keeps growing, right?
- 39:31SPEAKER_00: How do you see this infrastructure issue,
- 39:36SPEAKER_00: especially on the research side for training these models as time goes on?
- 39:41SPEAKER_01: Yeah, I mean,
- 39:41SPEAKER_01: it's like,
- 39:43SPEAKER_01: there's a reason why I think, like,
- 39:44SPEAKER_01: our models are, like,
- 39:46SPEAKER_01: especially like our Nemotrons,
- 39:47SPEAKER_01: right, are built by, like, people with engineer in their title.
- 39:51SPEAKER_01: And that's because it is a very difficult engineering problem.
- 39:57SPEAKER_01: It is, like,
- 39:58SPEAKER_01: as much time is spent in researching how to do this better from,
- 40:03SPEAKER_01: like, the engineering perspective as is spent, like, How to make the best models.
- 40:10SPEAKER_01: In a world where,
- 40:11SPEAKER_01: say, for instance, we could wave our hands and infrastructure challenges disappear.
- 40:17SPEAKER_01: We see this a lot,
- 40:19SPEAKER_01: and I think people talk about it a lot from the perspective of And some
- 40:24SPEAKER_01: of our friends across the sea who produce amazing models,
- 40:27SPEAKER_01: which is like they're operating in these very compute-bound environments.
- 40:31SPEAKER_01: They're operating in a very low-resource environment.
- 40:36SPEAKER_01: All of the effort that you see and all of the research
- 40:39SPEAKER_01: that you see coming out is like,
- 40:40SPEAKER_01: and this is how we did it with a toaster and a dream, right?
- 40:44SPEAKER_01: I think that we're...
- 40:48SPEAKER_01: We're going to continue to make a lot of leaps and bounds on
- 40:53SPEAKER_01: the infrastructure sides,
- 40:54SPEAKER_01: but also necessarily the way we train models is getting more and more complex
- 40:59SPEAKER_01: to better emulate reality, right?
- 41:02SPEAKER_01: Environments are meant to be some small chunk of reality.
- 41:05SPEAKER_01: I mean, what would be perfect is if we can train the model in the world on the world, right?
- 41:10SPEAKER_01: Like that is the environment we really want.
- 41:13SPEAKER_01: And to run that environment in a way that is accelerated enough to be useful
- 41:18SPEAKER_01: for training is – there is not a bigger infrastructure challenge.
- 41:25SPEAKER_01: So, I mean – Hopefully it answers your question.
- 41:29SPEAKER_01: The idea is like infrastructure is among the most important things to solve for right now.
- 41:35SPEAKER_01: That's why you see so many companies doing it.
- 41:37SPEAKER_01: That's why we're so happy to work closely with the people in
- 41:41SPEAKER_01: the Pneumatron Coalition who are like focused on that.
- 41:45SPEAKER_01: I think this is infrastructure is.
- 41:51SPEAKER_01: Is models, right?
- 41:53SPEAKER_01: At NVIDIA, we have like this word, we love to use it, co-design, extreme co-design, right?
- 41:59SPEAKER_01: So we're designing the models with the infrastructure,
- 42:02SPEAKER_01: with the hardware,
- 42:03SPEAKER_01: so that we can build better models that help us build better infrastructure,
- 42:06SPEAKER_01: that help us build better hardware that, you know what I mean?
- 42:09SPEAKER_01: Like this is the reality is you just cannot, you cannot skip one anymore.
- 42:15SPEAKER_00: I see it with the speed that you're trying to get and you're trying to juice up
- 42:22SPEAKER_00: the things that you have in terms of infrastructure or GPU as much as possible.
- 42:30SPEAKER_00: That's pretty good.
- 42:31SPEAKER_00: One thing I wanted to know,
- 42:32SPEAKER_00: like to kickstart a bit like this for the ultra version,
- 42:37SPEAKER_00: what would you say like there are the main like design shows that like set the,
- 42:41SPEAKER_00: like Nemontron 3, especially the ultra one, like apart from like the rest?
- 42:46SPEAKER_00: I think like you made a few key decision in there,
- 42:49SPEAKER_00: but like on your own word, like what are they?
- 42:52SPEAKER_01: Yeah.
- 42:53SPEAKER_01: So like the big ones we talked about in our blogs,
- 42:56SPEAKER_01: I just think that's like the correct, Uh, thing,
- 43:01SPEAKER_01: right, is like, so obviously the Hybrid Mamba attention, that was like a big one.
- 43:06SPEAKER_01: Uh, you know, Layton MOE is a big one.
- 43:10SPEAKER_01: Uh, NVFP4 obviously is a big one.
- 43:15SPEAKER_01: And then, uh, Kind of having MTP baked in from the start was another big one.
- 43:22SPEAKER_01: These are things that kind of like core architectural decisions
- 43:25SPEAKER_01: that wound up being incredibly impactful for like the model's performance as in
- 43:34SPEAKER_01: And, you know,
- 43:35SPEAKER_01: we're made in a way that they either didn't hurt accuracy
- 43:38SPEAKER_01: or actually aided accuracy, you know, but that's kind of like the big ones.
- 43:45SPEAKER_01: I think like the shiniest one, of course, is like hybrid attention.
- 43:49SPEAKER_01: I think that's the...
- 43:51SPEAKER_01: The classic.
- 43:52SPEAKER_01: But I think in terms of like most impactful, it's just definitely NVF before pre-training.
- 43:57SPEAKER_01: We just pre-train so fast in NVF before like that.
- 44:04SPEAKER_01: Your base model is such a limiting reagent on the quality of your final model
- 44:08SPEAKER_01: that being able to train more base models more quickly is a.
- 44:13SPEAKER_01: I cannot undersell how important that is, right?
- 44:16SPEAKER_01: So.
- 44:16SPEAKER_00: Right.
- 44:18SPEAKER_00: Right, right, yeah.
- 44:19SPEAKER_01: That makes a lot of sense.
- 44:21SPEAKER_00: I was looking at all of the different kind of Bet you guys did on the structure.
- 44:29SPEAKER_00: Some of them are pretty intense.
- 44:32SPEAKER_00: Like when you look at them just like alone.
- 44:34SPEAKER_00: And I think it's like all of these bets that are intense together that make it work so well.
- 44:41SPEAKER_00: But like the hybrid attention was very,
- 44:44SPEAKER_00: very interesting for me because like it's very, very aggressive.
- 44:48SPEAKER_00: Like you have a 108 layer And just a handful of them are not MEMBA.
- 44:54SPEAKER_00: So it's almost purely linear type of setup.
- 45:01SPEAKER_00: What was the reasoning here to go that much linear?
- 45:05SPEAKER_00: Was it to hit the one million context length for long running agentic stuff?
- 45:12SPEAKER_00: Why did you guys go that much on the linear side?
- 45:18SPEAKER_01: Yeah, the reality is attention is really good, but you just don't need a ton of it.
- 45:22SPEAKER_01: It is... We really felt...
- 45:30SPEAKER_01: From all of the work that the team does,
- 45:33SPEAKER_01: and I always have to say this to be super clear,
- 45:37SPEAKER_01: there are so many incredible researchers that work on these models.
- 45:43SPEAKER_01: I have the pleasure to talk to you today,
- 45:45SPEAKER_01: but I'm representing the work of many dozens of people.
- 45:49SPEAKER_01: The team found that
- 45:54SPEAKER_01: Really, we just need enough attention so that our long-range recall doesn't hit the dirt.
- 46:04SPEAKER_01: Because at the end of the day, that's...
- 46:06SPEAKER_01: I'm not going to say anything you don't know.
- 46:08SPEAKER_01: That's kind of what models do, right?
- 46:11SPEAKER_01: They look at their relationships across contacts, so we need some of that.
- 46:15SPEAKER_01: We can't have none,
- 46:17SPEAKER_01: but we need very little of it to have a good model or to have a model
- 46:21SPEAKER_01: that does useful things.
- 46:23SPEAKER_01: This is the kind of way that we talk about it, right?
- 46:28SPEAKER_01: But yeah, basically, if we went...
- 46:31SPEAKER_01: Worse, we would lose a lot of the long context performance.
- 46:35SPEAKER_01: And then so it's like,
- 46:36SPEAKER_01: how much can we,
- 46:38SPEAKER_01: how much do we need to keep so that when people are using the model
- 46:42SPEAKER_01: in reasonably common scenarios,
- 46:45SPEAKER_01: they aren't hitting like this, this just absolutely cursed, like no...
- 46:52SPEAKER_01: Like no recall, right?
- 46:54SPEAKER_01: Situation.
- 46:55SPEAKER_01: And so basically as few attention layers as humanly possible to avoid.
- 47:01SPEAKER_01: And then we found a ratio that worked and then just kind of kept it, right?
- 47:05SPEAKER_01: While we scaled up and we found that that did a really good job.
- 47:08SPEAKER_01: But at the end of the day, like,
- 47:11SPEAKER_01: I think if we could go more linear,
- 47:13SPEAKER_01: like we would, because it just saves time, like you have a faster model.
- 47:20SPEAKER_01: But again,
- 47:21SPEAKER_01: like at,
- 47:23SPEAKER_01: When you're dumping your code base or 75 tool calls into context,
- 47:27SPEAKER_01: you just need attention to have a model that stays coherent in turn 300 or whatever.
- 47:36SPEAKER_00: It did remind me of the design decision on this front about the Minimax family
- 47:44SPEAKER_00: where they were trying to push it to a million way back in 2025 or something like
- 47:50SPEAKER_00: that, the first big one that they put it.
- 47:54SPEAKER_00: They tried everything, the full linear one, but the ratio is almost similar.
- 48:01SPEAKER_00: I think yours is a bit more aggressive, which was the part that was interesting.
- 48:05SPEAKER_00: But on this front,
- 48:09SPEAKER_00: Realistically,
- 48:09SPEAKER_00: this thing will be used in... Well,
- 48:12SPEAKER_00: it could be used anywhere,
- 48:12SPEAKER_00: but it seems to be geared toward being into an R-ness, right?
- 48:16SPEAKER_00: Because I think you're also trained with a bunch of different R-ness agnostic settings, right?
- 48:23SPEAKER_00: But I don't know who I was talking to about this, about the...
- 48:32SPEAKER_00: The balance between having the long context baked into the model
- 48:37SPEAKER_00: or having long context baked into the harness and then have the model be trained
- 48:43SPEAKER_00: to really Use this harness to the maximum.
- 48:47SPEAKER_00: It seems that you guys just did both at the
- 48:51SPEAKER_01: same time.
- 48:55SPEAKER_00: What's the philosophy here with regard to the harness in general and the long context?
- 49:01SPEAKER_00: Do you guys plan on just keep on pushing here or now that we know
- 49:04SPEAKER_00: that these harnesses are so useful to fold a bit more to them?
- 49:09SPEAKER_00: What's the idea?
- 49:11SPEAKER_01: I mean, so there's a couple things that are true that I think are important to talk about.
- 49:16SPEAKER_01: One is that, like, harnesses do really a lot of very interesting things.
- 49:23SPEAKER_01: And they do those things better when the model understands them.
- 49:27SPEAKER_01: And so training with the idea of the harness is just necessary at this point.
- 49:33SPEAKER_01: Like, I am not going to use a model API by itself.
- 49:38SPEAKER_01: I think...
- 49:39SPEAKER_01: I might never do that in a personal setting ever again.
- 49:44SPEAKER_01: Obviously, at work, we're going to have to keep doing that.
- 49:49SPEAKER_01: But when I sit down and want to do work with a model, I'm not going to hit an API.
- 49:54SPEAKER_01: That's just the model, bro.
- 49:55SPEAKER_01: There's no way.
- 49:56SPEAKER_01: I'm going to be using some kind of harness, even if it's very lightweight.
- 50:01SPEAKER_01: So we must train in the harnesses, therefore.
- 50:03SPEAKER_01: And then it's like, okay, so if you're going to train in one harness...
- 50:06SPEAKER_01: You should be trading in all of the harnesses, right?
- 50:08SPEAKER_01: Because harness maxing is the lowest utility move you can go.
- 50:15SPEAKER_01: And I think people think harness maxing is bad because you're only good in one harness, right?
- 50:20SPEAKER_01: And that's a little bit of the reason why it's bad.
- 50:22SPEAKER_01: But it's actually bad because you will become quickly bad at even that harness.
- 50:28SPEAKER_01: The rate at which these harnesses change and modify is so fast that if we release
- 50:35SPEAKER_01: a model that's harness maxed on say like OpenCode,
- 50:40SPEAKER_01: the moment we release it,
- 50:41SPEAKER_01: in four weeks,
- 50:42SPEAKER_01: OpenCode The mode has almost four or five new features
- 50:48SPEAKER_01: that our model is certainly not well adapted to,
- 50:51SPEAKER_01: and therefore we're going to immediately take the performance.
- 50:56SPEAKER_01: So the moment you want to be good at one harness, You need to be good at all the harnesses.
- 50:59SPEAKER_01: If for no other reason, then you can generalize to all the changes they're going to make.
- 51:03SPEAKER_01: There's also like a very fun interplay where harnesses like to implement feature
- 51:07SPEAKER_01: from other harnesses, right?
- 51:08SPEAKER_01: So like when open code has a feature,
- 51:10SPEAKER_01: like py.dev will release a plugin for it or whatever like that.
- 51:14SPEAKER_01: And so if we train a model that understands that open code feature well,
- 51:18SPEAKER_01: when py.dev gets it, understands it well there as well.
- 51:21SPEAKER_01: So there's some of that, but yeah.
- 51:24SPEAKER_01: Okay, so that's the harness side.
- 51:25SPEAKER_01: So we have to.
- 51:25SPEAKER_01: There's just no escaping that models need to fit into harnesses in 2020 since.
- 51:31SPEAKER_01: And there's no escaping that models need to fit into all harnesses.
- 51:36SPEAKER_01: If for no other reason, then each harness is going to become every other harness at the limit.
- 51:41SPEAKER_01: And then on the second point,
- 51:42SPEAKER_01: which is like, Well,
- 51:44SPEAKER_01: if compaction and memory and stuff like this is kind of handled by the harness,
- 51:48SPEAKER_01: then why care about long context, right?
- 51:51SPEAKER_01: And there's two reasons.
- 51:53SPEAKER_01: One is that people will continue to push the context length no matter what.
- 52:00SPEAKER_01: Even with the greatest compaction tools,
- 52:04SPEAKER_01: people are going to wind up back in a situation where we're trying to fill up
- 52:10SPEAKER_01: as much context as possible.
- 52:12SPEAKER_01: And a second point is that We really want the model to be useful in harnesses
- 52:17SPEAKER_01: that might approach things in a new or different way that we haven't considered
- 52:20SPEAKER_01: and thought about.
- 52:22SPEAKER_01: And that still requires the model to be good at those long contexts.
- 52:25SPEAKER_01: So
- 52:27SPEAKER_01: I think the answer has to be And the reality is that it's not ultra difficult for
- 52:33SPEAKER_01: the answer to be both.
- 52:35SPEAKER_01: And so we'll just keep doing it.
- 52:37SPEAKER_01: Is it like strictly necessary with the tools that we have available
- 52:42SPEAKER_01: to use 1 million context length?
- 52:43SPEAKER_01: No.
- 52:45SPEAKER_01: You know, a lot of the harnesses offer a lot of great ways to keep your context window short.
- 52:50SPEAKER_01: And they actually do really well.
- 52:54SPEAKER_01: But like, That's the beautiful thing about the model.
- 52:57SPEAKER_01: You don't have to use the whole context window.
- 52:59SPEAKER_01: You can just use like...
- 53:01SPEAKER_01: You know, 64K or whatever.
- 53:02SPEAKER_01: You can deploy it in a configuration that just leverages 64K, right?
- 53:10SPEAKER_01: So hopefully answering your question.
- 53:11SPEAKER_01: But the idea is that I think we must continue to train with harnesses in mind.
- 53:18SPEAKER_01: And because we don't know what harnesses in 2026 September look like, we need long context.
- 53:27SPEAKER_00: Yeah, that makes a lot of sense.
- 53:29SPEAKER_00: And I don't think there's a definite answer about any of this also.
- 53:35SPEAKER_00: One thing,
- 53:35SPEAKER_00: just random fact that I found funny is that Like one million context,
- 53:40SPEAKER_00: we had it a while ago, right?
- 53:43SPEAKER_00: But it was kind of crap, like linked, right?
- 53:46SPEAKER_00: Like after some point,
- 53:48SPEAKER_00: like you could continue, like you're going to get like a lobotomized version of it.
- 53:52SPEAKER_00: At least you won't crash, but like it's, you know, it's struggling, right?
- 53:56SPEAKER_00: Yeah.
- 53:57SPEAKER_00: But what is great now is that like we see
- 54:00SPEAKER_00: that this effective context link is starting to kind of creep up, right?
- 54:03SPEAKER_00: That we have like a better window overall.
- 54:07SPEAKER_00: One thing that I find extremely fascinating is that longer and longer than
- 54:12SPEAKER_00: a million, It's very, very difficult to crack open at the moment.
- 54:19SPEAKER_00: I'm not sure if it's because generally the system just has
- 54:24SPEAKER_00: the full-blown attention all across,
- 54:28SPEAKER_00: and they just focus on that,
- 54:30SPEAKER_00: and they don't know how to weave in the linear attention.
- 54:34SPEAKER_00: But I see in the structure it is conductive to getting that expanded,
- 54:42SPEAKER_00: and especially I think I think it was a recent Kimi that kind of started
- 54:47SPEAKER_00: to drop also like the rope across and then use note,
- 54:53SPEAKER_00: which create all sorts of weird like set up with like the increasing context.
- 55:00SPEAKER_00: But I think we're getting to a point where like we're starting
- 55:02SPEAKER_00: to figure out our stuff because everything is interconnected, right?
- 55:05SPEAKER_00: But we're starting to figure out our stuff to get like Better,
- 55:09SPEAKER_00: longer context,
- 55:11SPEAKER_00: but then also leverage the R-ness to not have all this context sitting there
- 55:17SPEAKER_00: and just having that to play around.
- 55:20SPEAKER_00: So that's pretty cool.
- 55:21SPEAKER_00: Yeah, go ahead.
- 55:24SPEAKER_01: I think it's like we had this beautiful world for a moment.
- 55:26SPEAKER_01: I don't know if you remember back to the llama days, right?
- 55:29SPEAKER_01: Where it's like we were converging on like one architecture and it was like, oh man,
- 55:35SPEAKER_01: you know,
- 55:36SPEAKER_01: VLLM was so good because every model had the And now we're in this world
- 55:45SPEAKER_01: where there are so many interesting ways to solve these challenges
- 55:51SPEAKER_01: and perhaps more importantly, like you said, so many interesting trainings.
- 55:54SPEAKER_01: This is what we want the feel of the model to be.
- 55:55SPEAKER_01: And this is something that we can inspire from the architecture directly,
- 55:59SPEAKER_01: as opposed to just data blends and things like this.
- 56:01SPEAKER_01: So it's just a...
- 56:22SPEAKER_01: We're in such a fantastically creative world architecturally
- 56:27SPEAKER_01: that you can learn something from every paper these days.
- 56:33SPEAKER_01: Why the hell did they do that?
- 56:34SPEAKER_01: And you read it and you're like, what a great idea, bro.
- 56:38SPEAKER_00: And then you test the model and then you kind of feel it like, okay, right.
- 56:45SPEAKER_00: I feel the difference between this that uses this setup versus this other one,
- 56:50SPEAKER_00: which is so interesting because it's very difficult to put
- 56:54SPEAKER_00: in words what we're all looking at.
- 56:57SPEAKER_00: But when you use it, we kind of get this idea.
- 57:01SPEAKER_00: You feel it, man.
- 57:04SPEAKER_00: I think it was a team at RC where they were like sitting around
- 57:08SPEAKER_00: and like bouncing off ideas and stuff.
- 57:11SPEAKER_00: And a lot of it is like they're very well versed, right, in literature.
- 57:15SPEAKER_00: But at some point it's like their taste,
- 57:18SPEAKER_00: the vibe,
- 57:20SPEAKER_00: the direction they think that like will have a better ergonomic of some sort,
- 57:24SPEAKER_00: like in whatever shape that they're seeing in the model.
- 57:28SPEAKER_00: But it's beautiful to see it.
- 57:31SPEAKER_00: One of these shapes from your guys' side is the latent MOE.
- 57:41SPEAKER_00: It's such a great idea in terms of the compression and stuff like that.
- 57:49SPEAKER_00: Did you see any problem as you guys were implementing this on the latent MOE part
- 57:55SPEAKER_00: that you see it as a trade-off?
- 57:56SPEAKER_00: Or you see it as purely at this size with this amount of token, it's just purely free lunch.
- 58:03SPEAKER_00: We should be doing it this way.
- 58:05SPEAKER_00: Because now we're seeing more models adapt this exact technique.
- 58:09SPEAKER_00: What's your thought here?
- 58:11SPEAKER_01: Yeah, I mean,
- 58:11SPEAKER_01: it's definitely the closest thing we've got to a free lunch in the last little
- 58:15SPEAKER_01: while, I think.
- 58:17SPEAKER_01: It is...
- 58:19SPEAKER_01: It's like one of those rare
- 58:24SPEAKER_01: ways to do something where like every single component
- 58:30SPEAKER_01: of the model benefits in some way.
- 58:33SPEAKER_01: Like GPU util is benefited,
- 58:36SPEAKER_01: like throughput is benefited,
- 58:38SPEAKER_01: accuracy is benefited, like expert specialization is benefited.
- 58:45SPEAKER_01: I struggle to think of a single technology or like architectural decision
- 58:50SPEAKER_01: that like is as free of watch.
- 58:54SPEAKER_01: Like to me, the only other one that I can think of is kind of like dense to MOE, right?
- 58:59SPEAKER_01: And I think like the...
- 59:02SPEAKER_01: MOE right now, especially, is just so good.
- 59:06SPEAKER_01: It is not in every way better than dense.
- 59:10SPEAKER_01: I mean, there are ways in which dense models are still better,
- 59:12SPEAKER_01: and there are especially hardware profiles in which dense models are better.
- 59:16SPEAKER_01: But for most situations,
- 59:20SPEAKER_01: MOE is just kind of a win across the board on everything that you care about-ish.
- 59:26SPEAKER_01: Late MOE is like that, but better.
- 59:28SPEAKER_01: You know what I'm saying?
- 59:30SPEAKER_01: I don't know what else to say.
- 59:31SPEAKER_01: It is just
- 59:38SPEAKER_01: It is just a good way to do it.
- 59:40SPEAKER_01: And of course, like the main motivating factor is throughput, right?
- 59:45SPEAKER_01: So we're able to crank out more tokens per second with latent MOE.
- 59:54SPEAKER_01: This is important to us because of the faster models thing,
- 59:57SPEAKER_01: but also it wound up being a little bit of a...
- 1:00:04SPEAKER_01: Of a capability bump, which means...
- 1:00:07SPEAKER_00: This is what I find funny,
- 1:00:09SPEAKER_00: is because, like,
- 1:00:10SPEAKER_00: you guys came up with this,
- 1:00:13SPEAKER_00: and it almost felt like, oh,
- 1:00:16SPEAKER_00: we need to go faster,
- 1:00:17SPEAKER_00: so we're going to make this trade-off, but then you found there was no trade-off.
- 1:00:20SPEAKER_00: It was just bonus.
- 1:00:21SPEAKER_01: No trade-off, bro.
- 1:00:23SPEAKER_01: Yeah.
- 1:00:24SPEAKER_01: Yeah.
- 1:00:24SPEAKER_01: Why not?
- 1:00:26SPEAKER_00: I mean...
- 1:00:27SPEAKER_00: There was another point.
- 1:00:28SPEAKER_00: Yeah, go ahead.
- 1:00:29SPEAKER_01: I think one of the things
- 1:00:30SPEAKER_01: that it showcases is we weren't aggressive enough originally when we started
- 1:00:37SPEAKER_01: to think about MOE.
- 1:00:39SPEAKER_01: We could have been more aggressive.
- 1:00:42SPEAKER_01: We're just finding that that's true.
- 1:00:43SPEAKER_01: I'm excited to see how aggressive people can take something like Layton MOE and see
- 1:00:48SPEAKER_01: the level of cursed Layton MOE that's going to come out.
- 1:00:59SPEAKER_01: Where can we mess it up?
- 1:01:00SPEAKER_00: What's the limit?
- 1:01:04SPEAKER_00: Another part where you guys were surprisingly very aggressive,
- 1:01:07SPEAKER_00: I think it makes sense if it's like not everything is full attention,
- 1:01:11SPEAKER_00: is like the group query attention,
- 1:01:16SPEAKER_00: where you have like 64 query head We're just too kiwi-head.
- 1:01:20SPEAKER_00: It's like almost multi-query attention here, right?
- 1:01:23SPEAKER_00: And we know that if it's a full-on multi-query, there's some issue about that.
- 1:01:29SPEAKER_00: But how did you guys came up with this?
- 1:01:31SPEAKER_00: It was a sale again, trying to juice it up.
- 1:01:34SPEAKER_00: Did you run ablation?
- 1:01:36SPEAKER_00: How did it happen?
- 1:01:38SPEAKER_01: Yeah, I mean,
- 1:01:40SPEAKER_01: the answer is a lot of these things are ablated internally,
- 1:01:43SPEAKER_01: even if they don't make it to the tech reports.
- 1:01:45SPEAKER_01: So, yes, there are ablations.
- 1:01:49SPEAKER_01: But also, like,
- 1:01:49SPEAKER_01: I think the way to think about attention and Nemotron is,
- 1:01:55SPEAKER_01: like,
- 1:01:59SPEAKER_01: is for long context, almost purely.
- 1:02:04SPEAKER_01: Yeah.
- 1:02:07SPEAKER_01: So you
- 1:02:11SPEAKER_01: only need so few of these heads to retain performance
- 1:02:19SPEAKER_01: in long contact situations,
- 1:02:22SPEAKER_01: or more generally speaking for cross-context relationship stuff.
- 1:02:27SPEAKER_01: At the end of the day, like we're doing, Basically retrieval, right, with attention.
- 1:02:35SPEAKER_01: And so there's just enough signal through noise with two heads
- 1:02:41SPEAKER_01: to get exactly what we need from that.
- 1:02:46SPEAKER_01: So it kind of goes hand in hand.
- 1:02:48SPEAKER_01: But the team looked...
- 1:02:49SPEAKER_01: Like, we can't release every ablation,
- 1:02:52SPEAKER_01: obviously, in tech reports because they would just be so long.
- 1:02:55SPEAKER_01: But the team,
- 1:02:56SPEAKER_01: for the most part,
- 1:02:58SPEAKER_01: especially the way we approach our model families,
- 1:03:00SPEAKER_01: which is like we start with Nano and then we scale up,
- 1:03:03SPEAKER_01: the team is very aggressively running ablations to find out the best mixtures
- 1:03:08SPEAKER_01: of things.
- 1:03:09SPEAKER_01: Not everything, not literally everything is like totally empirically selected, right?
- 1:03:13SPEAKER_01: There is still some vibes and still some intuition,
- 1:03:15SPEAKER_01: but a lot of it's motivated by small studies.
- 1:03:19SPEAKER_01: And then you can say like, let's try it.
- 1:03:21SPEAKER_01: And then you try it and it works pretty good.
- 1:03:24SPEAKER_01: And then you scale up to super and you say like, well, it's not working worse, bro.
- 1:03:28SPEAKER_01: But yeah, at the end of the day, again, we just don't really need that much attention.
- 1:03:35SPEAKER_01: And because of that, we can get away with it.
- 1:03:37SPEAKER_01: All of the bits and bobs that interact with attention,
- 1:03:40SPEAKER_01: we can kind of throw them away or throw them away more aggressively than you expect.
- 1:03:46SPEAKER_00: That part is so fascinating to me because like,
- 1:03:50SPEAKER_00: I don't know,
- 1:03:51SPEAKER_00: it feels like we pushed it,
- 1:03:53SPEAKER_00: the stuff like as much as possible, like what we know was kind of important.
- 1:03:59SPEAKER_00: We didn't know how important in which measure it was, but like we just were just pushing it.
- 1:04:04SPEAKER_00: But then we're slowly realizing that maybe after a certain scale and stuff,
- 1:04:08SPEAKER_00: some of the things are maybe like training wheels.
- 1:04:11SPEAKER_00: You don't actually need that much, right?
- 1:04:14SPEAKER_00: Full-on attention is one of them.
- 1:04:16SPEAKER_00: You see more linearized model.
- 1:04:19SPEAKER_00: Of course, it's an hybrid and stuff.
- 1:04:21SPEAKER_00: So you see more of this and you see performance improving over there.
- 1:04:26SPEAKER_00: And it was kind of powered by the fact that now you can,
- 1:04:30SPEAKER_00: the reason and stuff, and it can kind of scaffold itself, things like that.
- 1:04:34SPEAKER_00: It has longer context now, so it's chill even if it's going for longer.
- 1:04:39SPEAKER_00: But then also the positional embedding.
- 1:04:42SPEAKER_00: At the start, if you don't have that, how the heck are you going to do it in the original paper?
- 1:04:47SPEAKER_00: But then now it's like, I actually don't need that much of it.
- 1:04:51SPEAKER_00: The model is smart enough to kind of figure this stuff out.
- 1:04:56SPEAKER_00: Yes, it's still a transformer and we're still kind of going with the same ideas,
- 1:05:02SPEAKER_00: but they have changed so much in shape.
- 1:05:04SPEAKER_00: And I think it's partially due to the scale of it and the profile
- 1:05:08SPEAKER_00: of how these things are used, right?
- 1:05:09SPEAKER_00: It's not just like...
- 1:05:11SPEAKER_00: A transformer that can do classification or translate and whatever.
- 1:05:14SPEAKER_00: It literally is something else now.
- 1:05:17SPEAKER_00: But it kind of emerged now from this kind of bastion of ideas that we had
- 1:05:22SPEAKER_00: and it's just shedding some of the stuff because it doesn't need it anymore.
- 1:05:27SPEAKER_01: It's a classic, like, think of how we make software, right?
- 1:05:30SPEAKER_01: Like, every time you see an initial software project,
- 1:05:32SPEAKER_01: it's just like, oh my god, look at all this shit.
- 1:05:35SPEAKER_01: And then, like, over time, like, what's a classic meme?
- 1:05:37SPEAKER_01: Like, a good PR is one that reduces lines of code, right?
- 1:05:43SPEAKER_01: And if you look at the Transformer, right, like...
- 1:05:50SPEAKER_01: And you look at kind of the absolutely cursed things that we have now.
- 1:05:55SPEAKER_01: It's the same thing.
- 1:05:55SPEAKER_01: It's just like, okay,
- 1:05:57SPEAKER_01: so we needed all of this when our data scale was really small
- 1:06:01SPEAKER_01: and our GPU scale was really small.
- 1:06:02SPEAKER_01: And now that we can actually train these It's just like you can suddenly cut out
- 1:06:07SPEAKER_01: so much fat that was only necessary because we were dealing with very small stocks.
- 1:06:20SPEAKER_01: And it's so interesting to see where the limit is,
- 1:06:26SPEAKER_01: and it's so interesting to see how we're going to think about scaling in even a year
- 1:06:31SPEAKER_01: from now compared to now.
- 1:06:34SPEAKER_01: There is no post-training wall, is what our post-training lead says.
- 1:06:39SPEAKER_01: I mean, obviously he wouldn't say that, he's post-training lead, but there is no wall, dude.
- 1:06:45SPEAKER_01: It is...
- 1:06:48SPEAKER_01: We can just cut more fat and train harder and get models that are faster and more efficient.
- 1:06:57SPEAKER_00: There was this beautiful paper from Yulu from MIT C-cell,
- 1:07:03SPEAKER_00: where he showed that
- 1:07:07SPEAKER_00: the landscape of expert around the model weights
- 1:07:14SPEAKER_00: after he was pre-trained is drastically different as you're a small model and
- 1:07:19SPEAKER_00: as you go bigger, right?
- 1:07:20SPEAKER_00: It's almost like there's this thicket of expert, right?
- 1:07:23SPEAKER_00: To the point where to get these expert, you don't even need to post-train that much.
- 1:07:28SPEAKER_00: You can just wiggle a bit.
- 1:07:30SPEAKER_00: Randomly, the way it works.
- 1:07:32SPEAKER_00: This is almost impossible at a lower scale or without zero pre-training.
- 1:07:40SPEAKER_00: It's as if as we're going and we're building these systems,
- 1:07:46SPEAKER_00: at some point they become a very different beast than what they were before.
- 1:07:49SPEAKER_00: Some of the assumptions that we had need to be always revised.
- 1:07:56SPEAKER_00: Because a lot of this stuff is driven by your ethos and your ability to experiment
- 1:08:02SPEAKER_00: and just to actually do it.
- 1:08:05SPEAKER_00: What do you think in the structure of how NVIDIA is building these models
- 1:08:12SPEAKER_00: and thinking about them, enabled to test such stuff like this?
- 1:08:16SPEAKER_00: I would argue that another lab maybe will look at this and say,
- 1:08:20SPEAKER_00: are you crazy running this with that much linearization and this
- 1:08:29SPEAKER_00: for P?
- 1:08:30SPEAKER_00: Makes no sense.
- 1:08:31SPEAKER_00: Why would we try this and get some failed run?
- 1:08:35SPEAKER_00: But you guys, in your case, are able to go and experiment with this stuff.
- 1:08:38SPEAKER_00: What enabled it within the company?
- 1:08:41SPEAKER_01: I mean, it's just like a stock answer-ish,
- 1:08:44SPEAKER_01: but I think it's an important one and the correct one,
- 1:08:46SPEAKER_01: which is that our leadership all the way to the top thinks this is the best way
- 1:08:51SPEAKER_01: to do science.
- 1:08:57SPEAKER_01: The job is to accelerate all of science, right?
- 1:09:00SPEAKER_01: That is to solve the hardest challenges.
- 1:09:03SPEAKER_01: Like that is why the company exists.
- 1:09:06SPEAKER_01: And you can't do that if your acceptance for experimentation is like low, right?
- 1:09:12SPEAKER_01: I mean, you just can't.
- 1:09:13SPEAKER_01: I mean, so like,
- 1:09:14SPEAKER_01: let's say for instance,
- 1:09:16SPEAKER_01: Something that probably most people would believe to be true,
- 1:09:19SPEAKER_01: which is like,
- 1:09:19SPEAKER_01: this is not, I mean,
- 1:09:21SPEAKER_01: Pneumatron 3 Ultra,
- 1:09:22SPEAKER_01: while it had great architecture, is not the global optimum, right?
- 1:09:27SPEAKER_01: Like, this is not the perfect architecture.
- 1:09:29SPEAKER_01: But let's also assume that,
- 1:09:31SPEAKER_01: like, the rest of the models that exist today, That is similarly true.
- 1:09:36SPEAKER_01: I
- 1:09:39SPEAKER_01: always do this every time because it's always fun to do it,
- 1:09:44SPEAKER_01: but the same way that if we just had constant learning rate for all
- 1:09:49SPEAKER_01: of our models forever,
- 1:09:50SPEAKER_01: we would never have gotten to the models we have now and they would be shit.
- 1:09:54SPEAKER_01: This is true of experimentation.
- 1:09:56SPEAKER_01: Sometimes you have to take a big step and learn was that step good or bad.
- 1:10:02SPEAKER_01: And if you work in an environment where you could never take a big step,
- 1:10:06SPEAKER_01: then you will always wind up somewhere worse than
- 1:10:09SPEAKER_01: if you occasionally do take big steps.
- 1:10:11SPEAKER_01: And so I think that's the vibe inside the company.
- 1:10:14SPEAKER_01: I mean, you got to back it with some data, right?
- 1:10:19SPEAKER_01: You can't just come out and swing it and be like,
- 1:10:21SPEAKER_00: oh, man.
- 1:10:22SPEAKER_01: Yeah, the vibes are right.
- 1:10:24SPEAKER_01: But also, if you do have that back, he's like, no, let's try it.
- 1:10:27SPEAKER_01: Why not?
- 1:10:28SPEAKER_01: Why not try it at nanoscale, right?
- 1:10:31SPEAKER_01: Like, why would we not make this bet?
- 1:10:34SPEAKER_01: And especially when our job is to understand how...
- 1:10:37SPEAKER_01: How to build the best models on Team Green hardware.
- 1:10:42SPEAKER_01: Like, why would we not make big bets on things
- 1:10:45SPEAKER_01: that tightly couple those two things together and see if it works?
- 1:10:49SPEAKER_01: And I think we just continue to find that while not everything works, a lot of it does.
- 1:10:53SPEAKER_01: And yeah,
- 1:10:56SPEAKER_01: it's just the whole atmosphere from the top to the bottom is one that embraces like
- 1:11:02SPEAKER_01: a traditional model.
- 1:11:04SPEAKER_01: Science approach, which is like, let's do some experiments.
- 1:11:07SPEAKER_01: An experiment can fail and that is okay.
- 1:11:10SPEAKER_01: That's how you learn.
- 1:11:12SPEAKER_01: We have a hypothesis, let's prove or disprove it.
- 1:11:15SPEAKER_01: I think too many people right now are obsessed with prove and not enough people care
- 1:11:21SPEAKER_01: about disprove.
- 1:11:24SPEAKER_00: Knowing that something doesn't work is fantastic because then you don't have to come back to it.
- 1:11:32SPEAKER_00: We have a question here from the audience.
- 1:11:34SPEAKER_00: Any advice Chris will give an undergrad want to work on similar thing in
- 1:11:37SPEAKER_00: the next couple of years?
- 1:11:38SPEAKER_00: What will be your advice?
- 1:11:41SPEAKER_01: I have terrible advice for this.
- 1:11:43SPEAKER_01: So here's what I did.
- 1:11:45SPEAKER_01: I loved physics a lot.
- 1:11:48SPEAKER_01: And then I learned you can teach computers.
- 1:11:51SPEAKER_01: And then I stopped caring as much about physics.
- 1:11:54SPEAKER_01: And then I stopped caring as much about my formal education.
- 1:11:57SPEAKER_01: And I started caring a lot about teaching computers how to do stuff.
- 1:12:02SPEAKER_01: And what I can say for you is that that worked out for me.
- 1:12:05SPEAKER_01: And I'm very happy that it worked out.
- 1:12:08SPEAKER_01: But maybe my advice would be, Learning is awesome.
- 1:12:14SPEAKER_01: I think that I'm talking to someone who exemplifies the advice I'm about to give.
- 1:12:21SPEAKER_01: But learning is awesome.
- 1:12:25SPEAKER_01: Reading is awesome.
- 1:12:26SPEAKER_01: But you got to be doing.
- 1:12:29SPEAKER_01: And you should be doing at the expense sometimes of your learning.
- 1:12:35SPEAKER_01: And so if you're new to undergrad, I would say...
- 1:12:40SPEAKER_01: It's important to get your formal education.
- 1:12:43SPEAKER_01: It's important to learn how structures and process of science
- 1:12:51SPEAKER_01: and society work
- 1:12:52SPEAKER_01: or whatever, but you should be spending more time building stuff.
- 1:12:56SPEAKER_01: And you should build stuff that's dumb and stupid and doesn't work.
- 1:13:00SPEAKER_01: And then you should explore why it was dumb and stupid and didn't work.
- 1:13:03SPEAKER_01: And then you should tell your friends about why your dumb, stupid thing didn't work.
- 1:13:07SPEAKER_01: And they can tell you, oh, did you try that?
- 1:13:09SPEAKER_01: You didn't think about this?
- 1:13:10SPEAKER_01: And then you can learn.
- 1:13:12SPEAKER_01: And so to distill the advice better than rambling,
- 1:13:15SPEAKER_01: what I would say is you should build more,
- 1:13:18SPEAKER_01: You should be comfortable to fail building more,
- 1:13:22SPEAKER_01: and you should find a group of people that you can build and fail with more.
- 1:13:29SPEAKER_01: And if you do those things,
- 1:13:32SPEAKER_01: I think you will find yourself in communities that will lead you to
- 1:13:36SPEAKER_01: the places you want to go.
- 1:13:39SPEAKER_01: We see this story over and over play out on...
- 1:13:44SPEAKER_01: Across the industry of people doing this until they're running the largest companies
- 1:13:49SPEAKER_01: in the world.
- 1:13:51SPEAKER_00: Yeah.
- 1:13:52SPEAKER_00: I think that's a great advice.
- 1:13:54SPEAKER_00: Just generally,
- 1:13:55SPEAKER_00: especially now,
- 1:13:56SPEAKER_00: I think it's even more relevant where we're like,
- 1:13:59SPEAKER_00: We have all these systems that can do a lot, right?
- 1:14:01SPEAKER_00: But the frontier is still the frontier.
- 1:14:03SPEAKER_00: We still have to think about that stuff.
- 1:14:05SPEAKER_00: And there's a very different skill set required to think at the frontier versus
- 1:14:09SPEAKER_00: to think at just the process that you need to go through and then run.
- 1:14:18SPEAKER_00: You have to have ideas and you have to be willing to fail in order to get
- 1:14:23SPEAKER_00: that feedback that you need to do the next iteration.
- 1:14:26SPEAKER_00: You only need to learn that by doing it.
- 1:14:28SPEAKER_00: There literally is no other way than to do that stuff.
- 1:14:34SPEAKER_00: Here's the thing, that's true.
- 1:14:36SPEAKER_01: In 2026, the time to failure has never been lower.
- 1:14:42SPEAKER_01: You can fail at something so fast today because of AI.
- 1:14:46SPEAKER_01: You can spin up that dumb idea you had, and you can learn why it was a dumb idea so quickly.
- 1:14:54SPEAKER_01: Listen, it's not exhaustive,
- 1:14:55SPEAKER_01: but after you've built something that's bad enough times,
- 1:15:00SPEAKER_01: You eventually learn,
- 1:15:02SPEAKER_01: oh wait, if I build something that's not these things, it's good.
- 1:15:06SPEAKER_01: It has never been faster to get feedback, iterative feedback on your ideas.
- 1:15:16SPEAKER_01: I'll say this,
- 1:15:16SPEAKER_01: if you sit around waiting for a great idea in 2026,
- 1:15:22SPEAKER_01: you'll be outdone by all the people who will shamelessly fail 75 times in
- 1:15:28SPEAKER_01: the time it took you to think up the good one, right?
- 1:15:30SPEAKER_01: So just,
- 1:15:32SPEAKER_01: we learn the most when we're wrong,
- 1:15:35SPEAKER_01: and so you should be aggressively wrong as much as possible.
- 1:15:40SPEAKER_00: That's fantastic.
- 1:15:41SPEAKER_00: Talking about being wrong, we can maybe talk about the shared weight MTP.
- 1:15:47SPEAKER_00: Because I really like that, because it's...
- 1:15:53SPEAKER_00: Using this draft idea and speculative decoding, slamming it inside the model as a train.
- 1:16:02SPEAKER_00: Can you just give us the broad stroke of how it works and how it's implemented in Emotron?
- 1:16:08SPEAKER_00: Because I think it's a bit different than how others are doing it.
- 1:16:13SPEAKER_01: Yeah, MT&P is a classic.
- 1:16:16SPEAKER_01: I mean, the reality is that I think what people think of when they think of
- 1:16:24SPEAKER_01: any kind
- 1:16:25SPEAKER_01: of spec deck is more your draft model spec decks,
- 1:16:30SPEAKER_01: which is this idea that there's a separate model that's going to draft tokens
- 1:16:34SPEAKER_01: for you, and then you're going to verify quickly.
- 1:16:37SPEAKER_01: And the idea is we want a high acceptance rate.
- 1:16:39SPEAKER_01: Of drafted tokens, right?
- 1:16:40SPEAKER_01: So this is what we're doing.
- 1:16:43SPEAKER_01: With MTP, we just kind of squish that into the model.
- 1:16:46SPEAKER_01: So it is trained as part of the model through pre-training, and then we keep it while we do...
- 1:16:54SPEAKER_01: All of the rest of our training.
- 1:16:57SPEAKER_01: You'll notice at the end of the pipeline that I showed earlier on,
- 1:17:00SPEAKER_01: there was this MTP healing part of the slide,
- 1:17:03SPEAKER_01: which is basically like,
- 1:17:05SPEAKER_01: we want to,
- 1:17:07SPEAKER_01: near the end of the training,
- 1:17:09SPEAKER_01: We'll lock the model weights,
- 1:17:10SPEAKER_01: and then we'll cram a bunch of data through the model
- 1:17:14SPEAKER_01: to get our acceptance rates back up.
- 1:17:17SPEAKER_01: But, I mean, it is just a head that predicts multiple tokens, and that's it.
- 1:17:24SPEAKER_01: The idea is that we will,
- 1:17:27SPEAKER_01: for every forward pass,
- 1:17:28SPEAKER_01: instead of generate one tokens,
- 1:17:30SPEAKER_01: we will generate N tokens,
- 1:17:32SPEAKER_01: and then we will have the model audit those generated tokens,
- 1:17:37SPEAKER_01: and they will either accept Or reject the drafted tokens.
- 1:17:41SPEAKER_01: And if you have a good acceptance rate,
- 1:17:43SPEAKER_01: this means that you have a fairly high rate at which the model verifies
- 1:17:52SPEAKER_01: the produced tokens.
- 1:17:54SPEAKER_01: You're going to save time because you will spend...
- 1:17:57SPEAKER_01: So basically you can think of it this way, right?
- 1:17:59SPEAKER_01: It takes one forward pass to generate end tokens or whatever it is.
- 1:18:03SPEAKER_01: And then we can then verify the tokens extremely quickly.
- 1:18:08SPEAKER_01: So if we're confident that most of the tokens will be retained,
- 1:18:13SPEAKER_01: then for every forward pass, we will save time.
- 1:18:18SPEAKER_01: That's the idea.
- 1:18:20SPEAKER_01: MTP being as part or trained natively as part of the model is
- 1:18:24SPEAKER_01: the thing that's slightly different,
- 1:18:26SPEAKER_01: but techniques like Eagle,
- 1:18:27SPEAKER_01: which we had before,
- 1:18:29SPEAKER_01: and now DeSpark,
- 1:18:30SPEAKER_01: DeFlash style methods,
- 1:18:32SPEAKER_01: which are spec tech using a separate draft model, still don't.
- 1:18:38SPEAKER_01: You can see those used across the board,
- 1:18:40SPEAKER_01: and you can see we've produced some for some models as well.
- 1:18:44SPEAKER_01: But this is the idea.
- 1:18:45SPEAKER_01: We just train MTP at the same time as the model, and we find that that does really good.
- 1:18:50SPEAKER_01: We also find that if you do that,
- 1:18:52SPEAKER_01: it could help other drafting techniques with their acceptance rates,
- 1:18:58SPEAKER_01: since the model already kind of was trained in this world where there's an idea
- 1:19:02SPEAKER_01: of draft tokens.
- 1:19:03SPEAKER_01: Yeah.
- 1:19:04SPEAKER_01: Yeah.
- 1:19:07SPEAKER_00: I like also that the core idea is just like faster speculative decoding, right?
- 1:19:15SPEAKER_00: That kind of is this line that is still going through it.
- 1:19:19SPEAKER_01: Of course.
- 1:19:20SPEAKER_01: Spectacular but fast.
- 1:19:21SPEAKER_01: Well, we're at it.
- 1:19:23SPEAKER_00: Yeah, he asked us to go fast, fast.
- 1:19:25SPEAKER_00: While we're at it,
- 1:19:26SPEAKER_00: can you also just walk us through the main idea of the multi-teacher
- 1:19:31SPEAKER_00: on policy distillation?
- 1:19:33SPEAKER_00: I found it very interesting.
- 1:19:36SPEAKER_00: I think the cool thing that I found here was there's the end cycle
- 1:19:42SPEAKER_00: that is happening here,
- 1:19:42SPEAKER_00: but it's the same model,
- 1:19:45SPEAKER_00: the student,
- 1:19:46SPEAKER_00: that is getting, I think, reused afterward, if I understood this part correctly.
- 1:19:53SPEAKER_01: So basically, MLPD is perfect model building, in my opinion.
- 1:20:00SPEAKER_01: What you do is you take a model.
- 1:20:02SPEAKER_01: Someone pre-trains a model, okay?
- 1:20:03SPEAKER_01: And then we do some kind of like warm-up phases, right?
- 1:20:10SPEAKER_01: So this is like get the model ready for what's about to come.
- 1:20:16SPEAKER_01: And so you can see here on this pipeline, you can literally see the phases.
- 1:20:22SPEAKER_01: Once we get to MLPD, what we're going to do is we're actually going to train a bunch of models.
- 1:20:26SPEAKER_01: We're going to train more than 10 models.
- 1:20:29SPEAKER_01: And each of those models is going to be trained by a team who's training it for
- 1:20:32SPEAKER_01: a specific domain or task set.
- 1:20:37SPEAKER_01: So this is like coding, STEM, Reasoning, chat, what have you.
- 1:20:44SPEAKER_01: And then what we're going to do is we're going to use OPD
- 1:20:47SPEAKER_01: to distill those teachers back into the base model.
- 1:20:53SPEAKER_01: And then what we're going to do from there What we did
- 1:20:56SPEAKER_01: with Ultra is we did one more iteration of this where we took
- 1:21:01SPEAKER_01: the now MOPD Checkpoint 1 and then from there we built new experts that were meant
- 1:21:09SPEAKER_01: to shore up.
- 1:21:09SPEAKER_01: Now,
- 1:21:10SPEAKER_01: the reason I say it's like the perfect way to trade a model or the best way to trade
- 1:21:13SPEAKER_01: a model is because each of those things,
- 1:21:24SPEAKER_01: Teams that trains the expert is themselves a pool of experts on training models
- 1:21:30SPEAKER_01: to do those things.
- 1:21:31SPEAKER_01: So it's a very concrete meta learning structure where your teams who are best
- 1:21:39SPEAKER_01: at making chat models can create the chat expert that best teaches your model
- 1:21:45SPEAKER_01: to do good at chat.
- 1:21:47SPEAKER_01: And so there's like a...
- 1:21:49SPEAKER_01: Yeah, it's a good situation to be in.
- 1:21:52SPEAKER_01: I think, too, it's why, like, I think people are going to gravitate
- 1:21:55SPEAKER_00: to it.
- 1:21:55SPEAKER_00: It's like the MOE on the MOE, right?
- 1:21:57SPEAKER_00: Like, you have the Mecha MOE running.
- 1:22:01SPEAKER_01: Yeah, so, and OPD is amazing.
- 1:22:03SPEAKER_01: I mean, OPD is amazing.
- 1:22:05SPEAKER_01: I think people learn that pretty quick.
- 1:22:07SPEAKER_01: There's a really good video by,
- 1:22:08SPEAKER_01: I think it was the Cursor guy who talked about OPD with,
- 1:22:11SPEAKER_01: like, DoorCash or something like that.
- 1:22:13SPEAKER_01: That's a really fantastic movie.
- 1:22:16SPEAKER_01: Video if you want to get an idea of why OPD is good,
- 1:22:20SPEAKER_01: but MOPD is basically the same thing,
- 1:22:22SPEAKER_01: but it's this idea that we want to fan out experts and then roll them back in.
- 1:22:29SPEAKER_01: It feels very much like better merging.
- 1:22:32SPEAKER_01: So if you remember the model merging days,
- 1:22:35SPEAKER_01: which is a year ago,
- 1:22:39SPEAKER_01: we still do a lot of merging, especially for pre-trained, but...
- 1:22:43SPEAKER_01: If you remember that, there's things like warp, right?
- 1:22:46SPEAKER_01: We would do the same thing.
- 1:22:47SPEAKER_01: You train the model to be good at a bunch of stuff,
- 1:22:50SPEAKER_01: and then you would slurp the weights back together,
- 1:22:54SPEAKER_01: and you would just keep doing that.
- 1:22:55SPEAKER_01: MLPD is like that, but just better.
- 1:22:57SPEAKER_01: Just definitively better.
- 1:23:01SPEAKER_00: And it's more stable also because the model is not crazy diverged also.
- 1:23:05SPEAKER_00: It's kind of the same base, right?
- 1:23:06SPEAKER_00: So they understand what the heck the teacher is trying to teach it,
- 1:23:11SPEAKER_00: but then you get all of this weight,
- 1:23:13SPEAKER_00: this dense kind of reward applied in a way that the model is able to understand it.
- 1:23:19SPEAKER_00: Very elegant, yeah, 100%.
- 1:23:21SPEAKER_01: And it doesn't
- 1:23:22SPEAKER_00: lead to like...
- 1:23:24SPEAKER_01: Capability collapse.
- 1:23:25SPEAKER_01: So one of the things that could happen with merging a lot is you would merge in
- 1:23:30SPEAKER_01: the stem weight so then the model would suddenly forget how to do chat tasks.
- 1:23:36SPEAKER_01: So MOPD pound for pound helps smoothen that out quite tremendously.
- 1:23:44SPEAKER_00: I think just generally,
- 1:23:46SPEAKER_00: I really like this shape because it blends well with this maybe futuristic idea
- 1:23:53SPEAKER_00: of just continuously being able to learn from everything that you're receiving from
- 1:23:59SPEAKER_00: the environment and then just applying this as a dance reward when you know.
- 1:24:03SPEAKER_00: What the actual thing was,
- 1:24:05SPEAKER_00: and you can maybe change the model without collapsing the model also.
- 1:24:09SPEAKER_00: Yeah, I just think that it fit itself well for what we're thinking should happen
- 1:24:18SPEAKER_00: with these models in the future.
- 1:24:21SPEAKER_00: Talking about the future, we have Kimiki 3 here that dropped.
- 1:24:29SPEAKER_00: Um, that is, uh, like, uh, some, like,
- 1:24:32SPEAKER_00: linear attention in there with some full attention,
- 1:24:34SPEAKER_00: latent MOE, they're doing, like, native 4-bit stuff, right?
- 1:24:39SPEAKER_00: Um, it look really like the Nematron piece that are getting scaled up, right?
- 1:24:45SPEAKER_00: But, uh, I think in their case it's, like, 2.8 trillion.
- 1:24:48SPEAKER_00: Um, is the team currently, like,
- 1:24:51SPEAKER_00: planning to just, like, yeah,
- 1:24:54SPEAKER_00: you need the super,
- 1:24:55SPEAKER_00: you have the ultra,
- 1:24:56SPEAKER_00: do you plan to kind of,
- 1:24:57SPEAKER_00: like, Can I crank up a bit more the size now that this is maybe the rest a bit?
- 1:25:04SPEAKER_01: You know what?
- 1:25:05SPEAKER_01: That's an incredible observation that other people have done that.
- 1:25:08SPEAKER_01: And if I look back at the track record of what we've done with Mnemotron,
- 1:25:12SPEAKER_01: I feel like we've followed a very similar pattern the whole time.
- 1:25:15SPEAKER_01: And so be ready for the ultimate.
- 1:25:18SPEAKER_01: This is the ultimate shithead thing to do.
- 1:25:22SPEAKER_01: I can't speak directly to the roadmap, OK?
- 1:25:26SPEAKER_01: But...
- 1:25:28SPEAKER_01: But you can expect that likely the things that we have done before,
- 1:25:33SPEAKER_01: we will continue to do in the future.
- 1:25:36SPEAKER_00: Okay, that's a great answer.
- 1:25:37SPEAKER_00: I mean, wow.
- 1:25:38SPEAKER_00: What a non-answer, eh?
- 1:25:41SPEAKER_00: That was a good non-answer.
- 1:25:44SPEAKER_00: I liked it.
- 1:25:46SPEAKER_00: This is good.
- 1:25:48SPEAKER_00: I had one last question for the GenRM stuff.
- 1:25:54SPEAKER_00: This part really,
- 1:25:55SPEAKER_00: I don't know,
- 1:25:56SPEAKER_00: I feel like Being able to
- 1:26:03SPEAKER_00: find ways of alleviating
- 1:26:05SPEAKER_00: the reward hacking is not only super important for performance,
- 1:26:10SPEAKER_00: but also just security-wise, absolutely crucial.
- 1:26:14SPEAKER_00: You have GPT 5.6, just...
- 1:26:18SPEAKER_00: Chaining a bunch of stuff in order to go in and benchmark this stuff
- 1:26:23SPEAKER_00: and reward hack itself out of the sandbox and whatever.
- 1:26:29SPEAKER_00: I feel like this is more and more important.
- 1:26:32SPEAKER_00: And then you said in the paper that it was in order to avoid
- 1:26:39SPEAKER_00: Like reward hacking if you had like a smaller model and
- 1:26:41SPEAKER_00: then you have this bigger model that is a bit more smart and is able to understand
- 1:26:44SPEAKER_00: the policy.
- 1:26:45SPEAKER_00: Like did you see any of like any weird hacking, the reward hacking happening in this?
- 1:26:51SPEAKER_00: Like did you have some story about like some of them that Demotron was able to pull out?
- 1:26:57SPEAKER_01: So I'm not on this side of the house specifically.
- 1:27:03SPEAKER_01: I can't answer.
- 1:27:04SPEAKER_01: I didn't reach out to the folks, but they didn't have a specific story to share.
- 1:27:11SPEAKER_01: But the idea is basically just like, yes,
- 1:27:20SPEAKER_01: especially for this genre piece, right?
- 1:27:22SPEAKER_01: The ability for the model to...
- 1:27:26SPEAKER_01: To learn the underlying policy of a small reward model is just super simple.
- 1:27:32SPEAKER_01: I don't know if you remember this, but we saw this early days with RLHF.
- 1:27:39SPEAKER_01: Models basically were just like, oh yeah, if I answer in three tokens, then I get good scores.
- 1:27:45SPEAKER_01: Because really what the reward model found is that humans like short responses.
- 1:27:50SPEAKER_01: It's the same thing here.
- 1:27:54SPEAKER_01: The idea is similar-ish to multi-environment,
- 1:27:56SPEAKER_01: where if we have this larger policy,
- 1:27:58SPEAKER_01: it's like less easy to suss out the exact axes upon which it's measuring.
- 1:28:05SPEAKER_01: And so we can kind of cheat out a little bit of extra performance,
- 1:28:12SPEAKER_01: as in lack of reward hacking that way.
- 1:28:15SPEAKER_01: But I don't have any specific stories on this one.
- 1:28:17SPEAKER_01: Many apologies.
- 1:28:19SPEAKER_01: The teams that do these are...
- 1:28:22SPEAKER_01: They're built different, man.
- 1:28:25SPEAKER_01: They are the most insane but incredibly smart team of people that I think I get
- 1:28:33SPEAKER_01: the chance to work with because they understand these behaviors on a level that are
- 1:28:42SPEAKER_01: beyond.
- 1:28:42SPEAKER_00: A lot of it is super surprising.
- 1:28:46SPEAKER_00: You could not have imagined that they would do that sort of stuff In order to gain
- 1:28:52SPEAKER_00: a bit more,
- 1:28:54SPEAKER_00: I think it was Florian that was talking about how in some instances,
- 1:29:01SPEAKER_00: the muggle will go out of his way just to get the information.
- 1:29:05SPEAKER_00: He knows the benchmark information is somewhere and he almost cannot resist
- 1:29:10SPEAKER_00: the temptation of getting that sweet reward.
- 1:29:14SPEAKER_00: So he's just going to go ahead and doing crime.
- 1:29:18SPEAKER_01: We train these models on internet scale levels of just go Google it,
- 1:29:22SPEAKER_01: just go search for it, just go look it up in Stack Overflow, bro.
- 1:29:28SPEAKER_01: We've taught them to be curious, you know, and fuck if they are not curious.
- 1:29:35SPEAKER_01: But a lot of like DRM stuff though is just so, I mean,
- 1:29:39SPEAKER_01: people don't understand how crucial it is For this kind of HF stuff to leak back
- 1:29:46SPEAKER_01: into the model.
- 1:29:49SPEAKER_01: When you use models that pay no attention to any of the RLHF side of
- 1:29:55SPEAKER_01: the house anymore, they are just asked to use.
- 1:30:01SPEAKER_01: At the end of the day, people use these models, so they have to be okay to use.
- 1:30:07SPEAKER_01: Our teams that give the model its sense of taste, I think they're doing a really good job.
- 1:30:13SPEAKER_01: But I don't have a specific story, but yeah.
- 1:30:16SPEAKER_00: That's fantastic.
- 1:30:17SPEAKER_00: That's good enough, man.
- 1:30:18SPEAKER_00: This was great.
- 1:30:20SPEAKER_00: A lot of good insights all across.
- 1:30:23SPEAKER_00: Well, to conclude a bit the interview,
- 1:30:26SPEAKER_00: what's next on the Need a Motron Labs roadmap here that you can share with us?
- 1:30:32SPEAKER_01: What I can share is that Nebotron 4 is on the way.
- 1:30:34SPEAKER_01: It's in the pipeline.
- 1:30:36SPEAKER_01: We've announced it.
- 1:30:37SPEAKER_01: What I can also share is that Jensen has a really great quote that I love, which is like...
- 1:30:44SPEAKER_01: I think Ahmad, the local AI guy, asked him this at some meetup or something like that.
- 1:30:49SPEAKER_01: And he...
- 1:30:51SPEAKER_01: He said, is there going to be more Nemotrons?
- 1:30:55SPEAKER_01: And Jensen's response was, we had CUDA 1, and now we have CUDA 18 or whatever.
- 1:31:03SPEAKER_01: We had Nemotron 1, do the math.
- 1:31:06SPEAKER_01: So I think, you know, the idea is like Mnemotron is here to stay.
- 1:31:13SPEAKER_01: Open science is here to stay.
- 1:31:15SPEAKER_01: These are the things that we have committed to and will continue to commit to.
- 1:31:22SPEAKER_01: And I can't stress enough how important these things are to...
- 1:31:28SPEAKER_01: So there you go.