Latest / Agentic Conversations (formally mlops.community) / The Art and Science of Training LLMs // Bandish Shah and Davis Blalock // #219

Transcript

The Art and Science of Training LLMs // Bandish Shah and Davis Blalock // #219 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey everyone, my name is Mondesi.
• 0:03
  I work on deep learning at Databricks and I take my coffee
• 0:07
  as a shot of espresso in the morning.
• 0:09
  Hey, I'm Davis. I'm a research scientist at
• 0:12
  Databricks and my caffeine of choice is green tea.
• 0:17
  Welcome, welcome. We are back with another MLM's
• 0:20
  Community podcast. I am your host and today we get
• 0:26
  2 for the price of 1. Davis and Brandish both doing
• 0:31
  some really cool stuff when it comes to training large language
• 0:35
  models at Mosaic. And if you are in to the
• 0:40
  research side of things or you are at least a little bit
• 0:45
  curious like myself, this is the episode for that.
• 0:48
  They break it down all the different places that you can
• 0:52
  fail. You know, we don't really like
• 0:54
  to talk about what is glittery and gold and all that.
• 0:58
  We're not going to tell you in these episodes that AGI is
• 1:03
  coming. Get ready.
• 1:04
  We better get universal basic income.
• 1:06
  We're talking about how these training runs can fail and what
• 1:14
  a conversation this was. There are so many different ways
• 1:18
  that you can look at training looms and the pains that these
• 1:26
  deep learning researchers go through on a day-to-day basis.
• 1:30
  Because as Davis put it, you can fail in a million different ways
• 1:37
  and he broke down just a handful of some of these ways.
• 1:41
  And a lot of times you don't even realise that your models
• 1:45
  are failing. You don't even know that the
• 1:47
  model training is not going as planned.
• 1:50
  You just see all of a sudden things start acting funny and
• 1:54
  then you have to go debug and maybe you have to debug on the
• 1:59
  hardware side. Maybe you have to debug because
• 2:02
  there's some breaking changes in PIE Torch that you didn't even
• 2:04
  realise, or there's just another type of framework that you're
• 2:08
  using and there's been a breaking change.
• 2:10
  He was mentioning how much of a headache all these breaking
• 2:13
  changes are, and how you have to keep up with that, and the act
• 2:19
  of trying to stay stable and use one version of Pytorch or
• 2:24
  whatever it is that you're leaning heavily on.
• 2:28
  It's a bit of a myth because you want the newest and baddest
• 2:34
  features that come out in the latest PIE.
• 2:36
  Which release? But guess what, Breaking
• 2:40
  changes. So that's another area.
• 2:42
  It can fail. It can fail just because you
• 2:45
  don't really understand what business problem you're going
• 2:49
  after when you're training this model.
• 2:51
  And so these guys get down and dirty with us on all the
• 2:55
  different things that they've seen in their years of
• 2:59
  experience. While training models, I also
• 3:01
  got into data quality with them and how we can ensure data
• 3:06
  quality, especially when you are training these gigantic models
• 3:10
  and you have a lot, a lot of data.
• 3:13
  It's basically the opposite of that.
• 3:15
  Big data is dead trend. This is the big data is alive
• 3:19
  and well trend, and one quote that I particularly like from
• 3:24
  the conversation is every data set is ugly in its own way.
• 3:29
  And Bandish mentioned how you got to get in there and you got
• 3:34
  to just do a little bit of manual evaluation.
• 3:37
  Look at your data set. Sometimes there's ideas that ohh
• 3:42
  yeah, a markdown file is referencing diagram 1 or table
• 3:48
  one. But because of the way that you
• 3:51
  did your data preprocessing, Table 1 does not exist in that
• 3:56
  markdown file anymore. So you'll figure it out soon
• 4:00
  enough. Just taking a few random samples
• 4:02
  of these gigantic data sets and there's always something wrong.
• 4:06
  That is what I'm coming away with.
• 4:09
  It's just a matter of if you can catch the majority of things.
• 4:14
  And these guys at Mosaic, they have been doing it long enough
• 4:17
  to where they've got some experience and they've seen a
• 4:21
  few things. They've been around the block,
• 4:23
  we could say. So let's get into this
• 4:26
  conversation. But before we do, I want to
• 4:28
  mention that I did some analysis on the YouTube page that we have
• 4:33
  and specifically the analytics side of the YouTube page.
• 4:36
  And I have some key findings for you.
• 4:38
  So if you are watching this on YouTube, there are about 30,000
• 4:45
  views that we had in the last 1028 days. 21 thousand of those
• 4:50
  views were from non subscribers. What is this?
• 4:53
  What is going on? It takes nothing.
• 4:56
  Hit that subscribe button. We would love to see you more
• 4:59
  often because the funny part of my little analysis is that 20
• 5:05
  thousand of those 30,000 views were from returning viewers.
• 5:11
  This means lots of people are coming back for seconds, but
• 5:14
  you're not hitting the subscribe button.
• 5:15
  So come on, hit that subscribe button.
• 5:19
  And I also want to give a huge shout out to everyone that
• 5:23
  reached out to me after a few podcasts ago when I was lost on
• 5:30
  a paper. I could not figure out what the
• 5:32
  paper was that I was trying to find and I knew it had something
• 5:37
  to do with data leaks and also using Open AI and all the ways
• 5:43
  that when you call the Open AI API you get data leaks and some
• 5:46
  prog researchers had written this paper.
• 5:50
  I couldn't figure it out. So many of you reached out to me
• 5:54
  and gave me that paper. I just have to say you restored
• 5:57
  my faith in humanity. It is so nice to see that and
• 6:02
  it's a little bit dangerous because now I might start
• 6:04
  leaning on you a little more than expected and I'm going to
• 6:10
  do my best to 1st Google things before I ask the masses.
• 6:15
  But I may ask you all a few more questions while we're at it.
• 6:20
  Last but not least, it is official June 25th we're having
• 6:26
  the first in person ML OPS Community Conference.
• 6:30
  It's gonna be all about AI quality.
• 6:35
  It's going to be called the AI Quality Conference June 25th in
• 6:40
  San Francisco. I really hope to see you there.
• 6:42
  We will leave a link to where you can get the early bird
• 6:45
  tickets right now. Come join us.
• 6:47
  You know we're not going to just do it like a normal conference.
• 6:50
  You can expect lots of shenanigans, that is for sure.
• 6:54
  Hopefully you did not use your learning budget up for this year
• 6:59
  yet and convince your boss that you got to go.
• 7:02
  We're going to make the tickets nice and cheap, so it's going to
• 7:04
  be an easy win. We just want to get you there.
• 7:07
  We want to have a bunch of fun friends hanging out in a room
• 7:11
  learning some stuff and maybe doing some crazy musical
• 7:17
  improvisations or some hardcore techno dance classes, you name
• 7:23
  it. I don't quite know what the
• 7:25
  craziness is going to be yet because it's still very early,
• 7:27
  but I suggest you get in on it June 25th.
• 7:30
  That is AI qualityconference.com.
• 7:34
  We've got it all laid out and before we go, I want to give a
• 7:38
  huge shout out to Databricks for sponsoring this episode.
• 7:43
  They've been doing some incredible stuff, as you will
• 7:45
  hear. And let's listen to our friends
• 7:50
  at Databricks and Mosaic in this conversation right now.
• 7:59
  Davis, I got to hit you with the question.
• 8:01
  Man, I'm a fan of your newsletter.
• 8:03
  I really appreciate when you break down all these papers that
• 8:07
  I don't necessarily under understand 100% of the time, but
• 8:11
  I feel like they're important and I feel smarter because I get
• 8:14
  your newsletter in my inbox. The only thing is, my inbox has
• 8:18
  been a little bit lonely since the beginning of the year, and
• 8:21
  I'm guessing that's because you either got access to a tonne of
• 8:26
  new GPU S and so you've been training a lot of models, or
• 8:31
  you're taking a hiatus to come back bigger and better in the
• 8:35
  second-half of of 2024. Yeah.
• 8:40
  So there are two reasons for that as an exciting one and a
• 8:43
  depressant 1. The exciting one is that we've
• 8:49
  been cooking at Mosaic. I can't say exactly what yet,
• 8:54
  but basically when the cluster costs more cash per day than you
• 8:59
  do per year, it's a little hard to justify working on other
• 9:03
  stuff. So that's kind of involved to
• 9:06
  the priority top of the priority list, to the exclusion of
• 9:08
  everything else. A little more depressing reason
• 9:12
  is, I think having to crack something out every week kinda
• 9:16
  suck the joy out of reading papers for me for a long time.
• 9:20
  Like, think about a task you don't want to do.
• 9:24
  Like the last time you had something you really dreaded,
• 9:26
  you put it off whenever you could.
• 9:28
  Now imagine that even if you do it, it just comes back again.
• 9:33
  Next week. It gets a little depressing.
• 9:37
  And now imagine that every day you put it off, it gets worse
• 9:41
  because every day more and more papers are coming in O.
• 9:44
  It used to be the case that reading papers was the one thing
• 9:47
  I could always do, no matter how sleep deprived I am, no matter
• 9:52
  what else is going on, it's always easy to sit down and
• 9:54
  focus or read a paper. But having this big extrinsic
• 9:57
  motivation, feeling like there are all these people almost
• 10:01
  breathing down my neck waiting for content, it just sucked all
• 10:06
  the joy out of it. And so I kind of reached the
• 10:10
  point where I needed, I needed to take a little bit of a break
• 10:13
  and kind of rediscover the joy of learning, of reading all this
• 10:18
  stuff, getting back to why I started doing in the 1st place.
• 10:23
  O that's what's happening. I think I would be doing it now
• 10:25
  if we weren't cooking so much, but I needed at least a few
• 10:27
  weeks to kind of reset there. For me, it's very much like when
• 10:33
  I write the newsletter for the MLS community, and especially
• 10:36
  the ones where it's more like I need to add my thought.
• 10:41
  I need to bring in new types of ideas.
• 10:44
  I want originality in it. It is literally like I I know
• 10:49
  that feeling of procrastination where each day it just builds
• 10:52
  and then you would think like more subscribers is better and
• 10:58
  you would be excited for that. But then you see like the
• 11:01
  numbers of subscribers are growing and then the
• 11:04
  procrastination is growing. And so that all just adds to the
• 11:07
  weight of it. And I would end up going nuts
• 11:10
  and then spending so much time on the newsletters and
• 11:15
  specifically like these mega OPS roundups.
• 11:19
  And I was doing those monthly. We have our weekly newsletter
• 11:23
  that goes out. But then I would do like a
• 11:26
  monthly one to kind of go over things and give my commentary on
• 11:29
  the whole scene and the ecosystem and all that fun stuff
• 11:33
  and it's hard. So I feel you.
• 11:35
  I can totally understand. A little break, hopefully you'll
• 11:38
  get back at it because my IQ thanks you.
• 11:42
  And for anyone that does not know and wants to sign up for
• 11:48
  when Davis is back in the newsletter writing phase.
• 11:53
  Hence these clusters aren't costing more than half of a team
• 11:57
  worth per day. I guess I would highly recommend
• 12:02
  subscribing to his newsletter. It's going to be in the
• 12:04
  description. It's like Davis Reed's papers, I
• 12:06
  think is the very unique name. To it and summarise this paper.
• 12:13
  There it is very clear what you're going to get on that
• 12:16
  newsletter. Davis Reed's paper, so you don't
• 12:19
  have to. Ohh.
• 12:20
  It's great. It's fantastic.
• 12:22
  And that is the value prop there.
• 12:24
  It's like you read all the, you read a whole tonne of papers and
• 12:28
  then you call out what the interesting pieces of some
• 12:33
  really good ones are. And so we're going to be talking
• 12:37
  a lot about lots of the stuff that you've been learning in
• 12:40
  those papers. But first, like maybe Davis, can
• 12:45
  you breakdown a little bit of how you got into tech in general
• 12:50
  so we can see your arc before Mosaic?
• 12:55
  Yeah. So I feel like me getting into
• 13:01
  technology was kind of inevitable in some sense.
• 13:05
  So my granddad was an electrical engineer and grew up on a farm
• 13:10
  in the middle of nowhere in Tennessee.
• 13:12
  But just always had that inclination.
• 13:16
  Ended up being the first person in his family to go to college
• 13:20
  at the University of Tennessee. Later went on to become a
• 13:22
  professor there. He had three sons.
• 13:26
  All of them taught electrical engineering degrees from the
• 13:28
  University of Tennessee. One's a professor there now.
• 13:32
  My dad's professor at UVA Other One is a practising electrical
• 13:36
  engineer. Just every male blaylock on my
• 13:42
  of my side of the family is an engineer of some sort.
• 13:45
  I'm actually a little bit of an outlier, getting a degree in
• 13:48
  computer science instead of electrical engineering.
• 13:51
  It's only electrical engineering from undergrad.
• 13:53
  And then so you you got the degree and you decided to hey,
• 13:58
  you know what, I think this ML stuff, it's pretty fun.
• 14:02
  And when did you get the bug for like starting to train and
• 14:09
  basically big GPU's making GPU's governor?
• 14:15
  Yeah, it was not what I intended to do.
• 14:19
  So back in 2013, I was in undergrad at the University of
• 14:24
  Virginia. I worked in a research group
• 14:27
  doing wearable devices. This was back in the days when
• 14:31
  Fitbit was going to change the world.
• 14:32
  We were going to revolutionise healthcare, all that stuff.
• 14:37
  And we had these little wearable devices or Lapid made record
• 14:43
  data from all sorts of different people we do.
• 14:45
  Studies identify dementia, Parkinson's, healthcare givers
• 14:51
  or whatever. And it turned out that once you
• 14:55
  had all this time series data, all this longitude and all data,
• 15:00
  it was really hard to analyse. It very hard to make sense of
• 15:04
  accelerometer, gyroscope, magnetometer signals.
• 15:07
  So I ended up just kind of backing in some machine learning
• 15:10
  and data mining. As a result of that, I was
• 15:14
  trying to work on healthcare stuff, but he just converged to
• 15:18
  the inevitable challenges of making sense of the data.
• 15:23
  So that was what I spent a fair amount of my time on in
• 15:26
  undergrad and then coming to grad school in 2014.
• 15:30
  It just seemed like machine learning was the field that was
• 15:33
  popping off. That had a lot of potential, and
• 15:37
  that was very deeply technical because I also considered doing
• 15:40
  a lot of human computer interaction stuff.
• 15:43
  There were groups that wanted me to come to HCI, but there was a
• 15:48
  lot of value it seemed, and doing something deeply technical
• 15:51
  that was emerging that had a lot of potential.
• 15:53
  So that was how I ended U signing a new machine learning
• 15:56
  when it came to MIT. And so you wrote the paper
• 16:01
  Multiplying Matrices without Multiplying, which I think
• 16:04
  gained you a bit of notoriety. Can you explain that?
• 16:08
  Yeah, that was a fun one. So the problem we're tackling
• 16:14
  there is approximate matrix multiplication, which is exactly
• 16:17
  what it sounds like. You have two matrices.
• 16:20
  You want to multiply them because matrix multiplication is
• 16:23
  of course one of the most fundamental operations in
• 16:25
  machine learning in many other fields.
• 16:28
  But you want to do it really quickly, and you're willing to
• 16:30
  tolerate some error in the output.
• 16:33
  There are a bunch of approaches to this.
• 16:36
  Under slightly different assumptions, we found a method
• 16:41
  that ends up letting you do this approximation without any
• 16:45
  multiplication. Sounds a little bit
• 16:47
  counterintuitive. Effectively, you have to know
• 16:50
  one of the matrices ahead of time.
• 16:52
  If you know one matrix ahead of time, it turns out you can
• 16:56
  process the other one in a way that doesn't involve any
• 17:00
  multiplication. I think it'll be a little hard
• 17:02
  to get into the details, but the keyword here to look for is a
• 17:05
  vector quantization. That's kind of the core of the
• 17:09
  approach, and what's cool about it is that it is extremely power
• 17:16
  efficient because you won't do vastly fewer operations.
• 17:23
  You end U doing small table lookups instead of multiplies to
• 17:28
  your place, say 4 or maybe 8 multiplies with one table.
• 17:31
  Look, that is both less work and is something you can implement
• 17:37
  in way fewer transistors. Just think about how many
• 17:41
  transistors it takes to build a multiplexer versus multiplier O.
• 17:44
  My contrarian take is that binary neural networks make
• 17:49
  basically no sense. They're strictly worse than
• 17:52
  vector quantization. They're strictly less
• 17:54
  expressive. They don't buy you any more
• 17:56
  hardware efficiency. They are the more obvious
• 17:59
  approach because they're just extrapolating taking the
• 18:03
  parameters to fewer and fewer bits, but there's effectively no
• 18:05
  advantage to them. It's all going to be vector
• 18:07
  quantization in the future. Wait, and this is been getting
• 18:13
  kind of popular recently because of the whole I I think there was
• 18:16
  a paper that recently came out that was talking about how ohh,
• 18:19
  we can do large language models in one bit, right.
• 18:22
  And so you're saying that's going to be thrown out the
• 18:26
  window. It's just a matter of time.
• 18:29
  Pretty much, yeah. There have been.
• 18:31
  There's been work on binarization of neural networks
• 18:33
  since at least 2015 ish, and there's a company called XNOR
• 18:37
  AI. I got acquired in 2016, so this
• 18:40
  is not a new thing. I think it's been riding the
• 18:44
  hype cycle up and down. Occasionally there's a paper
• 18:47
  that gets people excited, but yeah, I think it's an
• 18:50
  interesting proof of concept. But if you just look at the
• 18:55
  fundamentals, if you just look at the amount of information per
• 18:59
  stored bit you can get out of binarization versus vector
• 19:05
  quantization. You just have strictly more
• 19:08
  expressivity with vector quantization.
• 19:12
  Binarization is just a special case that kind of hamstrings
• 19:15
  you, because you can't exploit any mutual information across
• 19:20
  different parameters and every parameter has to be exactly 1
• 19:23
  bit. You're you're just really really
• 19:27
  constrained compared to a more flexible representation that
• 19:32
  looks at the distribution of multiple values together.
• 19:35
  That's really the core idea of vector quantization.
• 19:37
  People say quantization in neural networks usually in
• 19:39
  scalar quantization, which is every input floating point
• 19:43
  weight becomes 1. Lower precision weight.
• 19:46
  Vector quantization is you look at groups of weights and you
• 19:51
  effectively do K means and find structure in the distribution of
• 19:57
  your weights across different dimensions O You can exploit
• 20:00
  that structure to get vastly less quantization error.
• 20:04
  Ends. You end up with usually higher
• 20:08
  entropy in your encodings because you're saying which
• 20:10
  centroid is it? And yeah, it is just strictly
• 20:16
  more expressive while retaining the extreme low power hardware
• 20:21
  advantages. And it works today on CPUs like.
• 20:24
  You can already get a huge seed U on Ccus.
• 20:27
  Hmm. So Brandish don't wanna Don't
• 20:31
  want you to feel like we're just forgetting about you over here.
• 20:34
  Of course, I know that you were doing stuff with Matrices also.
• 20:37
  I mean, you had some time at Cisco just to catch everyone up
• 20:41
  on your journey until you got to Mosaic.
• 20:45
  You also were working at Samba Nova, so you're no stranger to
• 20:49
  this space either, and I I would love to get just the quick
• 20:54
  rundown on how you found yourself working on matrices and
• 20:58
  then jumped on to Mosaic and now you're leading teams there.
• 21:04
  Yeah, for sure. I mean, let's say so it's
• 21:06
  actually similar somewhat to, Davis said.
• 21:09
  You know, I I started out never actually thinking I'd get into
• 21:11
  computer engineering, computer science.
• 21:14
  I wanted to work on stuff that ends up in space, you know, and
• 21:18
  ended up just, yeah, I mean, I was just super like as a kid.
• 21:22
  Like I was a kid that took everything apart.
• 21:24
  I drove my parents nuts and constantly wouldn't shut up.
• 21:28
  Like, you know, my mom likes to joke that no one would babysit
• 21:31
  me because I would just keep asking questions.
• 21:33
  And it was like one of those things error I think that
• 21:37
  eventually just turned into, you know, getting into engineering
• 21:40
  school, you know and and the result a million things in
• 21:42
  between. There was astronaut, there was
• 21:44
  business. There was, you know whatever was
• 21:47
  flavour of the week kind of thing.
• 21:48
  But at the end of the day, you know, I think engineer at heart
• 21:51
  and yes I ended up getting to college, you know, declared an
• 21:55
  electrical engineering major because that's what a bunch of
• 21:58
  my uncles had done and ended up loving it and actually ended up
• 22:03
  yeah, joking. We never actually took a CS
• 22:05
  course and all of college except for like maybe the closest thing
• 22:08
  was a computer architecture and digital design course, which I
• 22:11
  took my my senior year and I got involved actually working on
• 22:16
  electrical systems for a nanosatellite project.
• 22:19
  So the Air Force research labs at the time had a nanosatellite
• 22:22
  programme. And so I ended up kind of
• 22:25
  building all the different interfaces and it was like this
• 22:27
  modular satellite that you could build very quickly.
• 22:30
  These are like nanosatellites. They're pretty common now.
• 22:32
  Like think about SpaceX or think about you know like Starlink and
• 22:36
  stuff a little smaller than that but same kind of basic principle
• 22:39
  and and yeah, I was like, OK, I'm gonna do aerospace, going to
• 22:43
  grad school for that and that will work for NASA.
• 22:44
  It's great. And.
• 22:46
  And that all changed, I think my, my serior when I, you know,
• 22:54
  decided to show up to a career fair super late, like in the
• 22:57
  last 15 minutes to like, just got you see what was out there.
• 23:00
  And I yeah. And I worked IT in college.
• 23:03
  You know, I was like my kind of school gig and ended up almost
• 23:06
  running over like a former kind of coworker that I worked with
• 23:09
  was like a year older. And he worked at Sun
• 23:11
  Microsystems at the time. And he was carrying this, like,
• 23:13
  giant, like, server motherboard that they had brought as like a
• 23:16
  prop. And I was like, that's cool.
• 23:18
  Like, how do I make that? He was like, give me your
• 23:20
  resume, you know? And then about, you know, a few
• 23:22
  weeks later interviewed at Sun Microsystems and then joined
• 23:26
  them and then about a year and a half later got bought by Oracle.
• 23:29
  So I was building silicon and I worked on like PCI Express,
• 23:33
  worked on memory ASICS, so worked on all kind of like the
• 23:37
  northbridge, Southbridge type stuff, you, you know, speak the
• 23:39
  Intel parlance. And then and then eventually
• 23:42
  started thinking about like the main processor.
• 23:44
  And so I did that for almost the better part of a decade.
• 23:47
  And actually right around the time I think in Naveen RAW was
• 23:53
  starting Nirvana and it was like, Ohh man.
• 23:56
  Like, you know, before if you had to go to, if you wanted to
• 23:57
  build trips and you wanted to work in a startup, like that was
• 24:01
  unheard of. Like everything had
• 24:02
  consolidated. Like you couldn't.
• 24:04
  You know, it's like there's just you gotta go to one of the big
• 24:06
  companies if you wanna make a career and ship design.
• 24:10
  And you know, I think a lot of part like being kind of made it
• 24:13
  cool again. Like there was new money going
• 24:14
  into chip designs, AI was picking up right.
• 24:17
  There are a lot of breakthroughs and deep learning happening.
• 24:20
  And it was like, OK, like, let's try to figure out how to build,
• 24:23
  you know, an FPGA at the time to just, like, prototype it, right.
• 24:27
  Let's do like, a quick iteration kind of hardware design on an
• 24:29
  FPGA. And so, you know, I like to joke
• 24:32
  that hammers. You kind of always see the world
• 24:34
  as nails. And so as a hardware guy, I was
• 24:36
  like, oh, like, let's clock it faster, right?
• 24:39
  Like, let's just do, you know, let's, let's take this thing
• 24:41
  that's happening in software. It's a bunch of, like, vector
• 24:43
  instructions on an X86 processor or something.
• 24:46
  And let's actually figure out how to build specialised
• 24:48
  hardware. And it was kind of along the
• 24:50
  same theme. So we started working on kind of
• 24:52
  prototyping these different things.
• 24:54
  Um, and then actually ended up, you know, a team from from
• 24:59
  Oracle at the time went off and found its Amanova.
• 25:01
  So I got involved in some Nova very early and that took a bunch
• 25:05
  of research like really just super interesting research
• 25:08
  around like building coarse grained reconfigurable type
• 25:10
  architectures and creating you know basically a deep learning
• 25:14
  accelerator from that. So then from there kind of went
• 25:16
  into performance and like really looking at how to optimise these
• 25:19
  workloads on these specialised chips, you know, probably picked
• 25:23
  up a little bit of, you know, more and more software started
• 25:26
  and moving up the stack a bit more, you know.
• 25:28
  Now I joke that I kind of manage a Python team.
• 25:31
  So it's it's you know I've come almost not not full circle, but
• 25:35
  I've definitely come from like have gone to the dark side, if
• 25:38
  you will, if you're hardware guys, so.
• 25:41
  Well, here's what I imagine both of you do while you when you go
• 25:45
  to work. And I think we're all kind of
• 25:47
  the same age. So you can remember those memes
• 25:50
  that were popular on Facebook probably like 10 years ago,
• 25:53
  where it's like what my brother thinks I do and what my mom
• 25:57
  thinks I do, what I actually do. I just, I think you guys sit
• 26:01
  there and you check in on your training runs that have been
• 26:04
  going for like the last 30 days and you're like everything
• 26:06
  looking good here. Let's see.
• 26:08
  Well then, how can I debug this? Or we need to stop training.
• 26:12
  So that's my like very naive vision of what you are doing.
• 26:17
  Give me the real breakdown on what's the day-to-day life in
• 26:21
  Davis and Ben Dishes world. Yeah, I could do that.
• 26:25
  So if you want to remember what it's like to train Ellens, you
• 26:32
  just have to remember there's a song written about it.
• 26:35
  It is 10% luck, 20% skill. It's 15% concentrated power,
• 26:42
  will 5% pleasure and 50% pain. That's pretty much the
• 26:47
  breakdown. Where's the pain coming from?
• 26:51
  Yeah. So that's a great, it's a great
• 26:53
  question. So there is pain from all over
• 26:57
  the place. The lowest level source of pain
• 27:02
  is that you get a tonne of hardware failures, at least.
• 27:06
  Supposedly this doesn't happen as much with CPUs, but at least
• 27:09
  with GPUs you're gonna be seeing hardware failures all the time.
• 27:12
  And when you get a hardware failure, it doesn't raise
• 27:16
  hardware failure exception that you can catch and handle in some
• 27:20
  nice way. You just get arbitrary bad
• 27:22
  behaviour. Maybe your job just completely
• 27:24
  crashes. Maybe some of your nose just
• 27:27
  slow down slightly. Could even just start getting
• 27:32
  subtly incorrect outputs. All sorts of terrible things
• 27:35
  happen. O you have say, thousands of
• 27:39
  GPUs, maybe just hundreds of GPUs crunching numbers and then
• 27:43
  suddenly something just kind of goes wrong.
• 27:45
  You usually what you'll see is a nickel timeout.
• 27:49
  Nickel is NVIDIA Distributed Communication library.
• 27:52
  All that means is that some node didn't respond when they were
• 27:55
  supposed to be syncing gradients or something.
• 27:57
  It's very vague, so it just means that something has gone
• 28:02
  wrong and now it's up to you to figure out what that is.
• 28:06
  You might manually run diagnostics on every single
• 28:09
  node. Try to figure out what's not
• 28:11
  responding or what's working too slowly.
• 28:14
  Maybe it's just transient. Occasionally something just
• 28:16
  randomly happens. Or maybe you don't encounter any
• 28:23
  of these issues, and actually there's some other problem that
• 28:26
  has occurred. For example, if you have enough
• 28:29
  nodes trying to write a checkpoint to an object store at
• 28:33
  once, you can basically dedos your object store and you will
• 28:38
  just see that your checkpoint saving or loading silently
• 28:42
  fails. You can get similar issues if
• 28:46
  you have, say, a network file system.
• 28:48
  You're just putting so much load on every single piece that
• 28:52
  you're touching that any piece can fail at anytime.
• 28:57
  That's the lowest level. That's the hardware and then you
• 29:01
  have the software on top of it. So because we're still in the
• 29:06
  phase of the field where we're figuring out how to do all this,
• 29:10
  and there is increasing demand for training at such large
• 29:14
  scale, the software stacks are not mature.
• 29:18
  There are breaking changes and all the NVIDIA libraries, you'll
• 29:22
  get breaking changes and pies for it.
• 29:24
  You'll get breaking phase changes in hugging face
• 29:26
  libraries, whatever abstraction layers you're using.
• 29:31
  On top of those, try to handle large scale training stuff that
• 29:34
  will experience breaking changes.
• 29:36
  I'm just loading your data, its scale is hard.
• 29:39
  You're going to have some sort of library to do that.
• 29:42
  Everything is unstable and you spend a lot of time maintaining
• 29:46
  some kind of layer to abstract it away.
• 29:48
  You're checking for different Torch version numbers and your
• 29:51
  Python code and monkey patching different functions.
• 29:54
  And like Pytorch FSP, you're in every single one of those.
• 29:59
  Is something you learned the hard way that you didn't look at
• 30:01
  the code right? Some clean unit tests and say Oh
• 30:04
  yes, well of course for Torch 221 we need the monkey patch to
• 30:08
  function in the following way. It's more like something
• 30:10
  horrible happened. He threw an engineer month at it
• 30:13
  to try to get to the root cause of the issue and eventually you
• 30:16
  found, Oh my gosh, it's this incredibly subtle thing with how
• 30:18
  they're initialising the buffers or whatever.
• 30:21
  And then you lucky patch should you figure it out and hopefully
• 30:26
  it gets better overtime. So every piece of it is
• 30:29
  difficult and that's just whether the job works.
• 30:32
  We haven't gotten to whether as a job yields anywhere near the
• 30:36
  accuracy you want. So my view is that in the worst
• 30:41
  debugging in all of computer science is debugging.
• 30:45
  Deep learning accuracy is worse than distributed systems, worse
• 30:48
  than embedded systems because it is so hard to get signal about
• 30:53
  what's happening and so expensive.
• 30:55
  If the numbers are too low, there's no exception right
• 30:59
  there. There's no concrete behaviour.
• 31:03
  You diagnose that, says ohh, well, that's null.
• 31:05
  Well, of course it's not working.
• 31:07
  No, you just get numbers that are too low with almost no
• 31:11
  indication as to what's happening and it is extremely
• 31:15
  expensive to check what is happening.
• 31:18
  O You have to be incredibly disciplined, incredibly
• 31:21
  systematic, and and have in some cases just a little bit of black
• 31:27
  magic to guess what is going wrong and identify it so that.
• 31:33
  That's like. An overview that that's a huge.
• 31:36
  Paradigm shift right like, like and I think people struggle with
• 31:40
  all the time like you in computing systems.
• 31:42
  You want a right answer, right? There's like always an expected
• 31:44
  answer and here there isn't. Like we've literally we've found
• 31:50
  issues where we're not even computing the right thing and
• 31:52
  the model still like has good accuracy.
• 31:56
  Like, you know, it's like what? You know, like that where would
• 32:00
  that fly? Right.
• 32:01
  Like you can't approximately transfer bank funds like that.
• 32:04
  That doesn't that doesn't fly right.
• 32:05
  Like that's not going to work like you, you know.
• 32:08
  So it's like here but it's like hey, like ohh yeah.
• 32:10
  We've been doing that wrong the whole time.
• 32:12
  Like you catch a bug like a year later and some code that you
• 32:14
  wrote, you know and it's like ohh.
• 32:16
  Yeah. Like something underlying change
• 32:18
  that broke the how that behaves and we never caught it because
• 32:21
  we didn't have a test or something and and it's like well
• 32:24
  when we changed it the model performed worse like and you're
• 32:27
  like wait, what What's happening right.
• 32:29
  Like and so it's you know things are things are constantly like
• 32:32
  moving there's all these moving parts things are changing and
• 32:36
  and there's no right answer. Right.
• 32:37
  It's it's very much like I mean I I, you know you were asking
• 32:41
  about what we do what's our day-to-day like I I kind of tell
• 32:44
  what like you know my my family and my friends like who aren't
• 32:47
  in this like yeah. And honestly sometimes feels
• 32:49
  like just educating someone. Right.
• 32:51
  It's like what do we do right. It's like we we have no formula
• 32:54
  for how everyone should learn. Like there's that's very clear
• 32:57
  you know and so here's a everyone.
• 32:59
  Learns. Differently.
• 33:00
  Everyone excels at different topics and it's like, what do we
• 33:04
  do? We write a bunch of tests and we
• 33:05
  say, you know take your L Sats or take your Mcats or take your
• 33:10
  whatever. And if you do well on them then
• 33:12
  you get a score and maybe that you know that's that's what this
• 33:15
  is, right? It's not there's no perfect size
• 33:18
  to it anyway should perform. There's no algorithm that it's
• 33:20
  always going to get you the right answer.
• 33:22
  It's it's like you kind of just trying different things and and
• 33:26
  seeing how the thing responds essentially.
• 33:29
  Wow. So I feel for you guys just
• 33:34
  listening to that. I can feel the amount of
• 33:38
  experience and like pain that you've had to go through.
• 33:43
  And as you mentioned, these are not mistakes that come without a
• 33:50
  price tag. Like there was something that I
• 33:55
  still quote matte your one of your colleagues at Databricks
• 34:00
  about when he came on here. He was saying how you can spend.
• 34:06
  25. 30-40 days training a model and it's not until the 40th day
• 34:13
  that you realise this model actually isn't that good and we
• 34:18
  can't use it. And so you've put all of these
• 34:21
  resources into it, whether that is the people's time or the GPU
• 34:26
  costs or just like trying to learn and make it better and
• 34:32
  you've got effectively nothing to show for it because it's not
• 34:36
  like Ohh, well we can use this model in some cases.
• 34:40
  You can't like half ass it. It's kind of all or nothing.
• 34:44
  Is this model hitting the the needs that we have or is it not?
• 34:50
  And if it doesn't been 40 days later you're like ohh, I guess
• 34:54
  we gotta try this one again and let's see if we can not mess it
• 34:57
  up next time. Yeah right.
• 34:58
  So I imagine you you felt that a few times, so that that.
• 35:02
  Definitely. So this is one of the things
• 35:04
  that we always talk to customers about, right?
• 35:06
  It's like even the problem of like knowing that the model is
• 35:10
  meeting your need or it's addressing your problem is like
• 35:13
  is pretty hard, like having good evals for your specific use
• 35:17
  case, right. Is like honestly like that's
• 35:19
  step zero. Like I think like the first
• 35:21
  question we ask is like, OK, you want to train a model with us or
• 35:24
  you want to develop a new model with us.
• 35:26
  Like what? How are you going to know that
• 35:28
  that model is doing what you want?
• 35:30
  Like you know what are you doing right?
• 35:31
  Are you going to use kind of the those leaderboard metrics?
• 35:35
  Is that reflective of your use cases?
• 35:37
  You know, are you going if you're doing code generation,
• 35:39
  are you going to use human eval? Is that good enough for you?
• 35:41
  Is that is that you know or if you have like a super specific
• 35:45
  business, right like if you you want to automate some call
• 35:50
  centre interactions or something and you have like very specific
• 35:53
  product needs like how is that going to be captured in general
• 35:56
  kind of domain stuff that you're you know getting from scraping
• 36:00
  the Internet and so like but in those cases like how are you
• 36:02
  going to evaluate this model. So I think like that's huge.
• 36:05
  Like you know we we kind of say like don't even start until you
• 36:09
  until you've answered that question And and the problem
• 36:11
  with that is sometimes you can't answer it until you have a
• 36:13
  bubble to try it out with. And so like we are, it's kind of
• 36:17
  a bit of a chicken and egg ish type scenario, Davis.
• 36:20
  I know there's there. I mean there's quite a bit of
• 36:21
  work that we also kind of do to to kind of try to understand
• 36:25
  where we'll land, you know and I think.
• 36:28
  It it is. 8 It's impossible to know like what capabilities are
• 36:34
  going to get until you have the final thing, but.
• 36:37
  In. Terms of like looking at where
• 36:39
  we're going to land, you know, like one of the things that we
• 36:41
  actually look at quite a bit is scaling laws.
• 36:43
  And so we do a lot of work to kind of try to figure out, OK,
• 36:46
  like this is how we're trending, we're running evals constantly,
• 36:50
  right? When we figure out like, OK,
• 36:52
  these are the things that are important for the models that
• 36:53
  we're training. So again, it's not 100.
• 36:56
  Percent guarantee, but it's important to like kind of track
• 36:59
  your progress as you go. Like where are you, where are
• 37:02
  you going? And that's why again having
• 37:03
  those evals are super important. Like you you want to mix like
• 37:06
  you don't want, you want all the kind of general leaderboard
• 37:08
  evals, the mmus, the GSM 8 cases like all these other things.
• 37:12
  And then on top of that you want your own evals.
• 37:14
  We do vibe tricks as well, right.
• 37:16
  Like we we we frequently asked what do you know if if model
• 37:20
  can't answer who is trained by then you know like that's that's
• 37:24
  you know something that we want to make sure that the model
• 37:26
  understands. So there's a lot of things that
• 37:28
  you kind of do along the way. And I think we we've gotten a
• 37:31
  lot. We've gotten pretty good at
• 37:32
  trying to like figure out where we're going to land.
• 37:34
  Usually it's a surprise. Sometimes you're like, wow, I
• 37:36
  can't believe it's actually this good.
• 37:39
  And so I think that's. Been a really positive trend.
• 37:41
  You know I think we have to the rinse and repeat that and see if
• 37:44
  we can consistently land there. But so far we've been we've been
• 37:47
  pretty good at kind of trying to figure out OK we can we can kind
• 37:50
  of land here and the model is going to be fairly capable so
• 37:54
  and. Does the lack of consistency
• 37:57
  come from just that it's every time you're training models for
• 38:01
  customers it's a different scenario?
• 38:04
  Or is it because it's just like you find, oh, there was a
• 38:09
  breaking change in Pytorch and we didn't recognise it and so
• 38:12
  now we got to start that again, Or a little bit of both?
• 38:16
  It's probably it's. Probably both.
• 38:18
  I mean, I think like it depends also because we're breaking it
• 38:21
  up like if we're pre training, if we're doing some more domain
• 38:24
  adaptation, fine tuning other things on top of that.
• 38:28
  But I think there is. I mean part of it is like.
• 38:33
  Being it's just I think. And it's a it's a really hard
• 38:35
  problem. Like, I can't emphasise that
• 38:37
  enough. Like when a customer has, like I
• 38:39
  have this use case like it's it's very easy for people to be
• 38:42
  like, oh, let me just throw AI in there and I'll automatically
• 38:47
  figure out everything and it's the Holy Grail and you know.
• 38:51
  I've been reading some Twitter threads that say that AI could
• 38:54
  fix this, right? Yeah, yeah, exactly.
• 38:56
  And a lot. Of a lot of folks also I will
• 38:57
  admittedly like say I think a lot of people throw in like deep
• 39:00
  learning or generative AI where you probably can just get away
• 39:04
  with like statistical level model or something much simpler,
• 39:07
  some random forest. Yeah, you don't, you know.
• 39:09
  And so I think like first of all like even just defining your
• 39:12
  problem is super super important like and then and then
• 39:16
  understanding how you know how do you measure the effect enough
• 39:18
  to solving that problem. And so I but.
• 39:22
  I but yeah. And I think if you don't, if you
• 39:23
  don't nail that down, then yeah. I think like how do you know
• 39:26
  what data you're going to really need to like gather and collect?
• 39:29
  Like how are you going to assess the quality of your data if you
• 39:32
  if you don't know that this is representative of the problem
• 39:35
  and you want a reasonable distribute.
• 39:37
  What are we, you know, like technically, I think actually we
• 39:39
  want to we want a reasonable representation.
• 39:41
  We want a good distribution of like the problems that we're
• 39:44
  trying to solve in our data set. Right.
• 39:45
  And so. So how?
• 39:47
  Do you? How do you even know?
• 39:48
  That if you don't know how you're going to e-mail.
• 39:49
  So I think that's why again, like keep harping on that, but
• 39:52
  that's so important and then yeah.
• 39:54
  Things are constantly. Changing I think, you know Davis
• 39:57
  Davis was basically describing every single day for the last, I
• 40:01
  don't know you know several months where for a long time we
• 40:05
  had these aspirations of like, you know with Pie Torch we're
• 40:07
  like, oh, we're only going to pin pie Torch versions.
• 40:10
  We're only going to do things on stable releases.
• 40:12
  We're we're enterprise, right. We we only want, we want, we
• 40:15
  want like good release cycles. We want to do all this that was
• 40:18
  like such a pipe dream. Like we had to to get like all
• 40:21
  cutting edge features, you know? Then we found this out from
• 40:23
  other people too, like, Ohh, hey, like we always run Pytorch
• 40:26
  slightly internally and it's like, well, how does it break
• 40:29
  like all the time, constantly. Like we constantly have to fix
• 40:32
  it And it's like you know there's monkey passing and so
• 40:34
  there's this all this enormous complexity that gets added on
• 40:37
  because you're you're literally like, I mean it's like building
• 40:40
  an aeroplane while it's in the air and on fire and so like and
• 40:43
  somehow it can't crash. And if it does, you know you got
• 40:46
  to put the pieces back together and and take flight again, so.
• 40:50
  So yeah, I mean. It's it's constantly and then
• 40:53
  root causing that is huge right. Like getting in there and like
• 40:56
  Ashley figuring out OK it was this one line change or it was
• 41:00
  you know this monkey patch that we had or was this version
• 41:02
  conflict like we're using older CUDA kernel because we we we
• 41:06
  have a a mismatch on the base image versus what's running in
• 41:09
  the container. And like you know it's just like
• 41:11
  all these things like we we we overarching they call the
• 41:14
  undifferentiated stuff, right. Like a lot of people I think
• 41:18
  when they want to train at scale are like ohh like let me just
• 41:21
  try to get the cheapest number like the highest number of
• 41:23
  cheapest number of GPU's that I can from AW and I'll figure out
• 41:27
  the rest and it's like you've. Only solved.
• 41:31
  Maybe 5 to 10% of the problem, right?
• 41:33
  Like you're not even, you're like get out.
• 41:35
  Getting the GPU compute has been hard because of supply
• 41:38
  constraints. But longer term, right, I don't
• 41:41
  think that's the thing that's going to really create the value
• 41:44
  creation on top of this. Yeah, even now.
• 41:47
  Man, it's so funny you mentioned that because I think in the last
• 41:50
  week and a half I've seen three or four people have either come
• 41:55
  to me or I've seen posts about how, hey, we have excess or
• 41:59
  spare GPU capacity. In case anybody wants some.
• 42:03
  We're looking to fill up our GPU's.
• 42:06
  And yeah, yeah, it is very much like it.
• 42:09
  It reminds me of that classic paper by D Scully on Ohh.
• 42:13
  The model is only one part of the problem, and then you have
• 42:16
  all the infrastructure around it.
• 42:18
  GPU's are just another part of that problem.
• 42:20
  It's not like you get the GPUs and then everything is good.
• 42:23
  You're golden. You've got to really be thinking
• 42:26
  about each specific piece and how to efficiently take on each
• 42:31
  of these boxes, right? I mean it's.
• 42:34
  Like, you know, I mean, again, I'm a hardware guy, come from
• 42:36
  building chips. Like if you can't just buy an
• 42:38
  X86 processor and be like, cool, I have a computer, right?
• 42:41
  Like there's everything else that it has to plug into and
• 42:45
  you're you're all. I mean, don't get me wrong,
• 42:46
  building a chip is super hard and it takes a lot of people and
• 42:50
  time and money and resources to do it.
• 42:52
  But again, you're only part of the way there, right?
• 42:55
  If you, you got to, you got to put everything together.
• 42:58
  You gotta get the software just right.
• 43:01
  And then not only are we doing that for one box, we're doing
• 43:04
  that for, you know, hundreds or thousands at this point.
• 43:07
  And so you know, we we, this is adorable and I wouldn't quote me
• 43:11
  on this, but you know, we roughly would assume about one
• 43:14
  failure every thousand, couple thousand GPU hours.
• 43:18
  So when you're training on 3004 thousand plus GPU's, right,
• 43:23
  you're gonna see a failure like every hour, right.
• 43:27
  So and so that's the scale. And so, so and then think about
• 43:32
  like you're moving terabytes of data around, if not more.
• 43:36
  And so like resuming state takes forever, right.
• 43:40
  And so if you're failing every hour or couple of hours, like if
• 43:43
  it takes you 30 minutes to resume, you're wasting 10s of
• 43:47
  you know, you're wasting someone's salary easily a day if
• 43:51
  if you don't, if you don't make that stuff super fast, right.
• 43:54
  And so we call it the undifferentiated stuff.
• 43:57
  But to be honest, like do you want it?
• 43:58
  You know, like that's what I keep, that's what I tell
• 44:00
  everyone is like, do you want to spend your time there?
• 44:02
  Because like it, it's it's, it is very hard to get all that
• 44:05
  stuff, right? Right.
• 44:06
  Like you just want to go and buy, like you want to just go to
• 44:08
  Dell and buy a PC. Like why?
• 44:10
  Why would you? Why would you know very few
• 44:12
  instances where you want to build all of that on your own,
• 44:15
  hence. Why Davis stopped writing his
• 44:18
  newsletter and she. Probably focused on this.
• 44:22
  Yeah. Well, was it.
• 44:23
  I mean part of it is like that's the, I mean that's it, right.
• 44:25
  That's the value that we create like we we are like and and
• 44:30
  that's what's really different. I think about what we're doing
• 44:32
  is we're not just doing this once for a model and then
• 44:36
  wrapping it up behind an API where we're actually doing this
• 44:40
  once for ourselves. That's that's literally like I
• 44:43
  mean that's kind of part of my job is like I always say like my
• 44:45
  team is like you know, two sides of the same coin.
• 44:47
  One side works very closely with Davis and the research team.
• 44:50
  The other side is working with the product team and like our
• 44:53
  customers. And I think like our job is to
• 44:55
  like basically figure out how do we take this one thing that
• 44:58
  we've done, like we've pushed the envelope, we've developed
• 45:00
  these new capabilities, but then I got to rinse and repeat that
• 45:03
  another 100 or 1000 times on the other side, right.
• 45:05
  I got to make customers just as successful, if not more than we
• 45:08
  have been training models. And so that that's what we're
• 45:12
  doing. So all that pain and suffering
• 45:14
  that Davis was just talking about, right?
• 45:16
  Like we have to package that and productize it so other people
• 45:19
  don't face it right. I don't.
• 45:20
  Other people don't have to deal with it and it's a really nice
• 45:22
  experience for them and. Getting that consistency because
• 45:26
  it's. Like.
• 45:27
  You were talking about. It's like if you're.
• 45:30
  Out there and. You're adding the value of, hey,
• 45:34
  when we do create a model, we're going to ensure that it is of
• 45:38
  quality and that if there are these problems or these failures
• 45:44
  that are happening, we can at least make sure that we're the
• 45:48
  ones that are looking into them. And you don't have to do that.
• 45:51
  I think that's a strong statement that you're saying.
• 45:53
  And I do see the value in that 100%.
• 45:57
  Yeah, another. Surprising value proposition His
• 46:01
  with that it's extremely valuable to have a complete
• 46:07
  configuration that just works. O give me your question of what
• 46:12
  happens when you spend all this time training a model and it's
• 46:16
  just not good enough at the end. That's a really big risk for
• 46:20
  people investing, often millions of dollars training a model.
• 46:25
  So one of the surprising value for oppositions we've had is the
• 46:28
  ability to just hand people a configuration from images to
• 46:32
  hyperparameters to everything and be able to say, OK, just
• 46:36
  train this just pointed at your data.
• 46:40
  Maybe you could figure it a little bit if you have, say,
• 46:43
  really strong priors about some change that needs to be made and
• 46:48
  it'll pretty reliably work, as opposed to having to discover
• 46:52
  all of that. It's like it's.
• 46:53
  Like buying insurance for those training runs?
• 46:57
  Exactly. Just the.
• 46:58
  The reliability is so huge, especially for startups like
• 47:01
  you're a startup, you raise $10 million in cash and you send
• 47:04
  6,000,000 trading. Your model.
• 47:07
  You know that kind of has. To work, you don't really get a
• 47:10
  second chance necessarily. And that's true both in terms of
• 47:15
  the MLS side like hyperparameters, and also to a
• 47:20
  lesser extent in terms of software.
• 47:22
  ANE like I can hand you 5 different code bases that all
• 47:28
  work, where all the tests ass and so on.
• 47:31
  But if you have to plug them together, the probability of
• 47:34
  your whole system actually working and not hitting a bunch
• 47:38
  of subtle subtle failures and version issues is very low O
• 47:42
  Having a happy path in terms of both hyperparameters and
• 47:47
  everything else that's necessary to make it work has proven to be
• 47:52
  surprisingly valuable for a lot of our customers.
• 47:55
  So there's another direction. That I want to go into since
• 47:58
  I've got you here and you both spend your days training models
• 48:05
  and thinking about sizes of models and architectures and
• 48:09
  models and then inference also once the model's out there And
• 48:13
  like I see you as understanding the system as opposed to just
• 48:19
  like the one piece. One thing that is very popular
• 48:23
  these days is a lot of people talking about how, OK, we've got
• 48:26
  the GT fours out there and those are great for when I want to get
• 48:31
  an MVP. But once I have something and I
• 48:35
  see that there's value, then I want to bring it in house and I
• 48:38
  want to make it faster and I want to make it more specific
• 48:43
  for that use case. I don't need the gigantic model
• 48:47
  that can do like 50 different things.
• 48:48
  I just need a much smaller model that is going to be able to do
• 48:53
  my one or two use cases, and so I think about.
• 48:59
  How people wrestle. With different questions, right.
• 49:05
  And it it's not just, I know there's a lot of interesting
• 49:08
  stuff whether it's like Ohh should we do rag or should we
• 49:11
  fine tune? And I'm not really even thinking
• 49:14
  about that. One question that I think about
• 49:16
  these days is. When do you say OK?
• 49:22
  I I want to train a model. Let's say that someone says I
• 49:25
  need to train my own model, but I just need like a 1 billion
• 49:31
  parameter model is. That even.
• 49:33
  Worth it? Are you going to see those types
• 49:36
  of like? Excel.
• 49:39
  Or are you going to see a 1 billion parameter model
• 49:42
  excelling in certain use cases if it is that very, very small
• 49:48
  model and or maybe it's that you say you guys advise, look, you
• 49:52
  know what, don't train a 1 billion parameter model from
• 49:55
  scratch. Let's take this model and let's
• 49:58
  distil it so we can make it 1 billion and it's going to be
• 50:01
  much faster. Like, there's all different ways
• 50:04
  that you can get to this end goal of how do we make it the
• 50:07
  most accurate and the fastest and the smallest as possible,
• 50:12
  but still hitting our needs. And so maybe you can, like, talk
• 50:19
  about all the different ways that you think through those
• 50:22
  problems these days, Yeah. I'll maybe I'll start high level
• 50:26
  and and then Davis, you know, you should fill in kind of the,
• 50:29
  the how to tune all of this stuff.
• 50:31
  But I think when we talk about like customers, I think what we
• 50:34
  see mostly is like, I think there's probably 4 dimensions
• 50:36
  that people really care about. And and it's really around like
• 50:39
  privacy, like privacy is number one for a lot of enterprise
• 50:42
  customers, like the most important thing.
• 50:46
  And. Privacy.
• 50:47
  It's like, yeah. I mean it's like I want to own
• 50:48
  the model. I mean I I'd love, you know not
• 50:50
  quite lumped in there but maybe said privacy and IP.
• 50:53
  Like I wanna I have unique data. I'm very good at solving this
• 50:56
  business problem. Now I fully subscribe that like
• 50:59
  hey if I'm not in this game I'm going to lose out like missing
• 51:01
  out on the Internet and you know in the late 90s or 2000s.
• 51:04
  And so I think people people kind of get that and they want
• 51:07
  to figure out how to deploy it. But privacy is like I want to
• 51:09
  you know I'm healthcare like I don't want to send by HIPAA
• 51:13
  whatever to the the the the sorry the rest APIs if you will,
• 51:17
  right the the private models because I don't know what
• 51:20
  they're going to do and and sure they can offer guarantees and
• 51:23
  all that but there's nothing like keeping it in your own
• 51:25
  house right. Like that's that's kind of like
• 51:26
  the main thing is is really important.
• 51:29
  The other is obviously quality which I think kind of you were
• 51:31
  you know kind of going and I think quality obviously
• 51:34
  especially open source like everyone cares about how do I
• 51:36
  get the highest quality at the lowest you know kind of model
• 51:39
  count. I want to run it on my laptop.
• 51:41
  I don't want an expensive GPU or something to run on and then.
• 51:44
  It's. Cost and performance and
• 51:46
  unfortunately like what it costs and performance and maybe all
• 51:48
  three of all four of these are not like completely independent
• 51:51
  variables, right. They're all kind of related and
• 51:53
  and you can kind of tune this entire space.
• 51:57
  But I think one of the other. Things that I think we really
• 51:59
  look at is. You know, if you look at like
• 52:02
  kind of. Like scaling laws, Chinchilla,
• 52:04
  right? That's good at telling you
• 52:05
  that's really focused on training, right?
• 52:07
  So for a like a given quality or or some some metric of quality,
• 52:12
  you know how much, What's the minimum?
• 52:14
  Amount of cost or. Compute that I have to spend to
• 52:18
  get there. But what those things?
• 52:21
  Said. That leave out and and actually
• 52:22
  like I had someone as part of the Mosaic team actually trying
• 52:26
  to bridge these two things together is, is the cost of
• 52:30
  inference, right. And so being able to like you
• 52:33
  know this is better to train as well.
• 52:34
  Like if you look at the difference between a 20 billion
• 52:37
  parameter model and a 30 billion parameter model, I mean it's,
• 52:40
  it's huge, it's it's actually over like 2X the cost and
• 52:44
  compute right, because it's, it's a quadratic relationship.
• 52:47
  And so I mean just think about a 1 billion versus 10 billion
• 52:50
  parameter model and so, so at the end of the day like you can
• 52:56
  over train a model and make it much cheaper to actually deploy.
• 53:00
  And in order to do that, you have to understand kind of all
• 53:03
  of it, right? You have to understand like OK,
• 53:05
  like what is the cost of deployment?
• 53:08
  If I, if I, if I train a 500 billion parameter model, like
• 53:13
  you know, it's going to be super expensive to serve.
• 53:16
  You know, if I need a lot of throughput, if I need a lot of
• 53:18
  users, do I need the generalizability that a 500
• 53:22
  billion parameter model, you know, if trained well can offer
• 53:25
  versus you know, some Can I get away with something that's a 30
• 53:28
  billion parameter model? And so I think you know, but and
• 53:32
  then I think the other part is the architecture of the model
• 53:34
  kind of helps here, right. This is why people start getting
• 53:36
  into like things like mixture of experts, where you draw out the
• 53:39
  kind of this compromise between model like parameter count, but
• 53:43
  also the inference cost goes down quite a bit.
• 53:47
  There's a lot of different. Jobs that you can tune during
• 53:50
  that you know and then also looking at like the amount of
• 53:52
  data that you have to collect. So, but yeah, I mean Dave is
• 53:55
  feel free to dive in. I just wanted to kind of say
• 53:57
  these are the dimensions, you know, that I think we generally
• 53:59
  see. Yeah, definitely.
• 54:03
  I think when making these choices, the basic framework is
• 54:08
  the same As for rolling out any sort of software or any sort of
• 54:12
  roduct feature, which is you should start with the simplest
• 54:15
  thing possible, and only if that's not good enough do you
• 54:20
  proceed to something more sophisticated.
• 54:23
  So if you can just query a third party API and that's good enough
• 54:28
  for your use case then you should probably just do that.
• 54:32
  Then if you have say privacy constraints or cost constraints
• 54:37
  or maybe latency constraints that maybe you deploy your own
• 54:40
  pre trade model. Let's take a generic mixed role
• 54:45
  or lawn or whatever it is and just put that behind an API and
• 54:51
  that. If that's not quite good enough,
• 54:52
  OK, maybe curate some in context examples to give it a better
• 54:57
  prompt. Try to squeeze more quality out
• 55:01
  of it, and if that's not good enough, maybe hook it up to a
• 55:04
  retrieval system, depending on the latency you have and how
• 55:08
  much complexity it adds to get that set up.
• 55:11
  And if that's not good enough, maybe fine tune it on a little
• 55:14
  bit of data and that's not good enough.
• 55:17
  Swap out a bigger bottle, maybe pre train your own model to a
• 55:22
  much larger scale, fine tuning or even continued retraining.
• 55:27
  The size here is a little bit of an open question, but there
• 55:29
  definitely seems to be alpha in doing some sort of domain
• 55:33
  adaptation with a retraining regural next token reduction.
• 55:39
  Retraining objective could be followed by fine tuning or
• 55:42
  something like ROHFF. So you're keep adding layers as
• 55:47
  needed rather than necessarily saying here a priori is the
• 55:53
  exact set of features or the exact pipeline we're going to
• 55:58
  need to hit this target. Unless maybe somehow you can
• 56:01
  forecast that this is so. This is such a good point and it
• 56:04
  reflects something that Sam who we had on the podcast probably
• 56:09
  like 2 weeks ago was talking about.
• 56:11
  And I mentioned it to you guys before we hit record.
• 56:13
  But it was along the lines of I haven't met a lot of people who
• 56:18
  said damn, I wish I would have architected a more complicated
• 56:22
  system. And this comes back to that
• 56:25
  exact point where it's just like if you can get away with just
• 56:29
  calling an API, do it. Don't even worry about like
• 56:34
  setting up any link chain or lum index or any of that.
• 56:37
  Like it's just the straight call to the API and you can get value
• 56:41
  out of that. Yeah, do that and try and go as
• 56:44
  simple as possible. And then I really like this idea
• 56:48
  of breaking down the levels of complexity that you've seen.
• 56:53
  And so you're like, alright, well yeah, there's API, there's
• 56:56
  alright then you can potentially bring it in house, you can set
• 57:01
  up a rag if you want. You can set up some fine tuning.
• 57:06
  You can try and go a bigger model if that's going to, if
• 57:10
  that's where you need to see like everything just to try and
• 57:14
  hit a little bit better accuracy on that.
• 57:19
  And we can't forget this idea that you're talking about
• 57:23
  Bendish is like. All of that's great.
• 57:27
  If you understand what the problem you're trying to solve
• 57:31
  for is, because otherwise you're just shooting for something that
• 57:36
  you don't know if it's actually that good or not, right?
• 57:39
  You're just saying, well, we could probably do better, but
• 57:42
  better against what? What is it that you're trying to
• 57:44
  solve for? Because you could always do
• 57:47
  better, right? Yeah, and.
• 57:49
  Some of this, by the way, translates to, I think, like
• 57:51
  almost organisational complexity and sophistication.
• 57:53
  Like, I think, you know, a lot of times there's a lot, there's
• 57:57
  so much information out there right now about this.
• 57:59
  And I think that it takes time to like, build and foster this,
• 58:03
  this type of expertise, right. Like we.
• 58:07
  We we spent a lot. Of time thinking about like the
• 58:09
  audience, right? So like we internally were like
• 58:12
  hey like when I when I talk to my team it's like Davis and the
• 58:15
  research team is customer 0. But what our internal research
• 58:20
  team does is like vastly different, right.
• 58:23
  Like this is you know like what the labs are doing the
• 58:27
  state-of-the-art all of that when we expose these prop these
• 58:31
  things as products to customers they have to be significantly
• 58:34
  simplified and streamlined for people.
• 58:36
  You know like we were talking about defaults earlier like I
• 58:39
  cannot like like like undervalue like the the it's just the value
• 58:45
  of a good default is massive right.
• 58:47
  If I if I actually like probably.
• 58:48
  And I I would guess if I looked at all of the configs that we
• 58:51
  look at and I just dipped them all and like looked at how many
• 58:55
  changes there were. Like there's a few like you
• 58:57
  could have hundreds of lines of a config and just probably see
• 58:59
  maybe a dozen lines change because people don't change that
• 59:03
  much stuff right. Like and and I, I frequently
• 59:06
  hear when I share with customers like, hey, like we just, we we
• 59:09
  did all these sweeps. It was like super expensive.
• 59:13
  But then we just use your defaults and they were like the
• 59:16
  best convergence that we were able to get.
• 59:18
  And it's like, yeah, because we like did that for you, right?
• 59:21
  Like we did all these sweeps and we've tried like to find good
• 59:25
  defaults, right. And so I think, but that's
• 59:27
  another way of breaking down that complexity.
• 59:30
  But I I would say like exactly what you're kind of saying,
• 59:33
  right? Is like if I'm just prototyping.
• 59:35
  If I'm trying to understand my problem, don't.
• 59:38
  Don't train A. Model like just start with
• 59:40
  something out there, right and and and only do it if like you
• 59:43
  think that solving the problem necessitates that customization.
• 59:48
  And I think what we see with most customers is they start
• 59:51
  small like the the ones that are the most successful like and you
• 59:55
  know we've written about like the reports we've done like you
• 59:57
  know several different customer stories out there like they
• 1:00:00
  start small, they start out like 1 billion, three billion
• 1:00:02
  parameter models. They work their way up
• 1:00:05
  internally. We do that too.
• 1:00:06
  That's how we manage complexity. We do like a bunch of small
• 1:00:09
  gating runs to like get things in a place where, OK, we can now
• 1:00:13
  like predictably understand that hey, if we were to like double
• 1:00:16
  the parameter count. Where would we go?
• 1:00:18
  Right. And I think as you continue to
• 1:00:20
  build that expertise as a team, you can take on more of these
• 1:00:25
  challenges, right. And then I hope you know also
• 1:00:27
  from the complexity standpoint, the tooling gets better.
• 1:00:30
  We're building different techniques that people can start
• 1:00:33
  using and deploying at scale, right, to actually improve the
• 1:00:36
  quality and several of those dimensions that we were kind of
• 1:00:38
  talking about. But I think yeah, I mean.
• 1:00:41
  It's just this all comes down to managing complexity.
• 1:00:44
  You know, you don't have to go train a 100 billion parameter
• 1:00:47
  model like on day one. You know, like.
• 1:00:50
  Don't. Don't be that person who goes to
• 1:00:51
  your like CTO or CIO or whatever it is like, ohh, yeah, you know,
• 1:00:55
  we're going to do this because like if you don't, if you
• 1:00:57
  understand it, you're just going to waste millions of dollars
• 1:00:59
  basically, yeah. Yeah.
• 1:01:02
  And you're not going to be in that job for too much longer,
• 1:01:04
  that's for sure. The, the one thing that I also
• 1:01:09
  wanted to dig into is the complexity on the data side and
• 1:01:14
  just like how hard it can be with the data pipelines because
• 1:01:19
  we all know and we've heard it like a million times, Ohh data
• 1:01:23
  is your mote, right. And so you still have a lot to
• 1:01:27
  do on the data side just in general I think and a lot of
• 1:01:33
  times that can be overlooked. So maybe like Davis, have you
• 1:01:37
  seen good ways of data pipelines being efficiently thought out?
• 1:01:43
  Or where are there parts, bottlenecks, areas that fail all
• 1:01:49
  of that fun stuff too? Yeah, there.
• 1:01:51
  Are so many pitfalls and potential failures here.
• 1:01:57
  So. A lot of it is just true of all
• 1:02:01
  of machine learning and data science.
• 1:02:03
  And data engineering is hard. Getting good data is hard,
• 1:02:06
  cleaning data is hard. And a lot of this holds
• 1:02:10
  independent of whether you're training LMS or doing any sort
• 1:02:13
  of large scale machine learning. In.
• 1:02:17
  The context of alms. You do hit a few extra problems
• 1:02:24
  that you may not have seen if you've been doing more
• 1:02:28
  traditional data science. One that sticks out.
• 1:02:31
  Is tokenization O? We don't train models on the raw
• 1:02:37
  UTF 8 or whatever encoding you have in your documents.
• 1:02:41
  We split the documents U into what we call tokens O, like the
• 1:02:46
  suffix Tihon might be a token, or those probably a token,
• 1:02:51
  right? It's this algorithm for
• 1:02:53
  splitting text into unique identifiers, and that process
• 1:03:00
  his has many subtle pitfalls, as many tokenizers, for example
• 1:03:07
  allow you to generate just completely invalid bytes.
• 1:03:13
  So you might have some amount of outputs that are just not UTF 8
• 1:03:16
  at all. You will also get weird
• 1:03:19
  coalescing of stuff like white space with other characters, so
• 1:03:23
  maybe like colon isn't a token. Colon space is a token.
• 1:03:27
  So we've had issues, for example, when trying to get the
• 1:03:31
  Falcon models to do math where we just could not get them to
• 1:03:36
  produce anything useful. It's just garbage.
• 1:03:38
  And then we realised, oh wait, we need a space at the end of
• 1:03:42
  the prompt. And then everything started
• 1:03:45
  working. There's.
• 1:03:47
  And that's not even. That uncommon?
• 1:03:49
  That's not some weird quirk of the Falcon models.
• 1:03:51
  We've had evals that work this way.
• 1:03:54
  We've had bug fixes in our evaluation programmes with that
• 1:04:00
  say add 10 or 20% accuracy once we fix them so.
• 1:04:04
  There's. All sorts of pitfalls there.
• 1:04:09
  There are also a bunch. Of challenges associated with
• 1:04:12
  even loading your data at scale. So if you've only operated A
• 1:04:19
  relatively small scale, you might be used to just loading
• 1:04:22
  your CSV file or whatever. Everything just runs on your
• 1:04:25
  laptop. You just point your spark
• 1:04:29
  programme and have ark file in 1/3 bucket, whatever.
• 1:04:34
  But when you have a sufficient number of nodes loading a
• 1:04:38
  sufficient amount of data, you hit a whole bunch of weird
• 1:04:42
  requirements around deduplication.
• 1:04:45
  Like you can't have different nodes request the same data.
• 1:04:48
  You're going to be requesting the same data 2000 times, and
• 1:04:51
  then your egress fees might actually matter.
• 1:04:53
  If you're in a remote object store or it's just incredibly
• 1:04:56
  slow. You also need to shuffling, or
• 1:04:59
  it turns out you get subtle convergence issues if you ever
• 1:05:02
  see your loss curves going up and down, probably you have bad
• 1:05:06
  shuffling. We call those runs away, boys.
• 1:05:10
  There's. Also, issues with resumption.
• 1:05:15
  If you want to load data at scale, like your job crashes and
• 1:05:19
  then you want to restart it, well OK are you going to
• 1:05:22
  redownload your data set from scratch?
• 1:05:24
  What happens if you want to on a different number of nodes that
• 1:05:27
  completely screw up your shuffling?
• 1:05:29
  There are also whole bunch of issues along those lines that
• 1:05:33
  make it such that you kind of need to use some library for
• 1:05:37
  this. We have one.
• 1:05:39
  I know other people have attempted to write them, but you
• 1:05:42
  just said a whole bunch of subtle pitfalls here that you
• 1:05:44
  may not be used to thinking about and how.
• 1:05:47
  Much of these like translate over to Alright cool, I want to
• 1:05:50
  fine tune a model. So a lot of it.
• 1:05:55
  Is a matter of scale rather than whether you're pretraining
• 1:06:00
  versus fine tuning, because fundamentally pre training
• 1:06:03
  versus fine tuning is kind of just a different loss function.
• 1:06:06
  Unless you're doing AHF, then you're using O.
• 1:06:11
  Things get a little more strange because there's a reward model,
• 1:06:14
  but I think very few people are actually doing RHF at this
• 1:06:18
  point, at least on on their own. So.
• 1:06:24
  It's more a matter of scale. Like if you're training
• 1:06:26
  something on 2 nodes and you have redundant downloads and
• 1:06:30
  your data set isn't that big and you can just redownload it every
• 1:06:32
  time the job crashes, then you can dodge a lot of this
• 1:06:35
  complexity. You don't have to worry about a
• 1:06:38
  30 or 60 minute ramp up time when your cluster doesn't cost
• 1:06:42
  1000 cost thousands of dollars per hour.
• 1:06:45
  Yeah, but yeah, you'll hit a a bunch of barriers as you scale
• 1:06:49
  up, You know it. I'll put this.
• 1:06:51
  Another like it was interesting I think for me, and I was I
• 1:06:55
  think blissfully naive about, you know, data in general until.
• 1:07:01
  First of all, we scaled. And and and and I just don't
• 1:07:04
  mean scaled in terms of computing infrastructure but
• 1:07:06
  like we had to scale our token count to train you know pre
• 1:07:09
  trained useful models. But.
• 1:07:11
  Also you know then this little company called Databricks
• 1:07:15
  acquired Mosaic and new type of. Skill.
• 1:07:19
  Yeah, new new type of scale. But I mean, you know like
• 1:07:21
  managing data at scale and doing AI at scale, there's
• 1:07:25
  similarities, there's differences.
• 1:07:27
  But what I what the, the the kind of thing that drove at home
• 1:07:30
  for me was we had these data pipelines that you know do
• 1:07:33
  things like deduplication and filtering.
• 1:07:35
  And some of these things would take like hours, maybe days
• 1:07:38
  depending on like what we were, you know, kind of doing when we
• 1:07:42
  brought him in. And kind of you know basically
• 1:07:45
  we just went to the horde of like you know field engineering
• 1:07:47
  and kind of spark experts at Databricks and we're like help
• 1:07:51
  us how do we make these faster. And we got it down to like you
• 1:07:54
  know something that took days down to like a couple of hours.
• 1:07:57
  You're like, wow, like you know we're we're not you know we
• 1:08:01
  didn't start this data engineers you know and and it was like one
• 1:08:04
  of these things we're like, OK if you use the right tool for
• 1:08:06
  the job, if you if you use the system that's built to process
• 1:08:10
  data at scale. That's incredibly.
• 1:08:13
  Important and and I think when I you know driving it back like a
• 1:08:18
  couple. Enterprise.
• 1:08:18
  Is actually like a lot of customers like have a lot of
• 1:08:21
  data and so that whole pipeline, that whole ETL processing
• 1:08:26
  pipeline that you have to do, like there's this whole problem.
• 1:08:30
  Of. Like getting that data and then
• 1:08:32
  like whittling it down to like the Super high quality data in
• 1:08:35
  the right format that you actually need to do different
• 1:08:39
  types of training right. Like retraining kind of just
• 1:08:41
  takes just feed in all the text you can get right.
• 1:08:43
  But then you have things like instruction fine tuning.
• 1:08:46
  We're now you need to craft like proper response pairs.
• 1:08:49
  You want to add instructions on top of things.
• 1:08:51
  You might want to augment your data.
• 1:08:52
  You might want to like create synthetic data to add additional
• 1:08:57
  data and get your token counts up.
• 1:08:59
  So. There's like this whole.
• 1:09:00
  Sophisticated pipeline that I think is emerging that also has
• 1:09:04
  to happen at the scales that we're talking about.
• 1:09:07
  To actually be able to. Train like a really high quality
• 1:09:10
  model, right. So that's that's why this is
• 1:09:12
  like this is like this was such a big, you know, like a match
• 1:09:15
  made in heaven, right. It's like there's there's this
• 1:09:17
  gravity to data and this this gravity because of the scale of
• 1:09:22
  data that you know it was very interesting to look at like OK
• 1:09:26
  like is data going to kind of migrate to to training computer
• 1:09:30
  is training computing to migrate to data and kind of turns out
• 1:09:33
  like we just have to build a bridge between the two like the
• 1:09:36
  the gravity of both is very massive and and and that bridge
• 1:09:39
  that's what we that's what Davis was alluding to.
• 1:09:41
  Like we have this kind of data loader that now we're working on
• 1:09:44
  like really, really super optimising to kind of like move
• 1:09:46
  between these sort of these data processing compute clusters and
• 1:09:51
  as well as like our training clusters, right.
• 1:09:52
  So I think getting all of that right again is like one of that.
• 1:09:55
  It's like, you know, we say maybe it's the undifferentiated
• 1:09:57
  stuff, but it's the Super hard stuff at the end of the day and
• 1:10:00
  when it works, it's like magic, so.
• 1:10:03
  And you guys are lucky. I think you have probably some
• 1:10:05
  of the best Spark engineers in the business that you can lean
• 1:10:09
  on. Yeah.
• 1:10:10
  So it's like I imagine a lot of people wish that they had that
• 1:10:15
  type of person that they can just go and ping for questions.
• 1:10:19
  On the data engineering side, yeah, I'm.
• 1:10:22
  I'm also thinking about though, like when it comes to data
• 1:10:26
  quality, we hear that term being thrown around so much and.
• 1:10:33
  When you're. At this scale that you're
• 1:10:36
  talking about, like, how do you even deal with data quality?
• 1:10:40
  How can you ensure that the data quality is like the highest
• 1:10:44
  purest form? It's super hard.
• 1:10:49
  I think the core of the answer is look at your data.
• 1:10:54
  Sadly, there's there's really no substitute for that.
• 1:10:58
  You'll find all sorts of horrific things happening, like
• 1:11:02
  you might find that all of the tables and all of your markdown
• 1:11:05
  files got stripped away by whatever preprocessing you have
• 1:11:09
  O the textual reference table one and there just is no table
• 1:11:11
  one. I You'll see all sorts of
• 1:11:15
  horrific things like that, and it's also hard to manually.
• 1:11:19
  Looking at your data to some extent.
• 1:11:23
  If you're randomly sampling, there, at least initially, is
• 1:11:27
  probably enough broken stuff that you will find errors if you
• 1:11:32
  sit and look at your data. There is also some amount of
• 1:11:35
  stuff you can do at scale, but automating it is awfully
• 1:11:40
  difficult because every data set is ugly in its own way.
• 1:11:43
  Every data set needs to be cleaned in its own way, and what
• 1:11:47
  good data even means is going to depend on your exact problem.
• 1:11:52
  So there's some stuff. You can do like you put in some
• 1:11:55
  filtering heuristic and that gives you a subset of your data,
• 1:11:59
  and then you're training a simple model on it.
• 1:12:02
  And maybe if the number is higher than it was with the
• 1:12:05
  different subset of your data, you conclude that your filtering
• 1:12:08
  heuristic was helpful. But even that can be very
• 1:12:11
  difficult because one of the things we've seen and other
• 1:12:15
  groups have seen is that you don't get that much signal and
• 1:12:18
  data quality until you're operating a pretty large scale.
• 1:12:21
  You need pretty good models in many cases to.
• 1:12:26
  Have data. Quality and or quantity become
• 1:12:29
  the bottleneck. So it's not something where you
• 1:12:32
  run it on your laptop in 10 minutes like ohh, the SVM got
• 1:12:36
  93% accuracy instead of 91. And so we trust that as a
• 1:12:39
  reliable signal, more like, well OK, that was a GPU week of time
• 1:12:47
  on 7 billion parameter model and it kind of seems like it did
• 1:12:50
  better. And if you throw enough
• 1:12:53
  computer, enough experiments at it, you kind of get a feel for
• 1:12:57
  what's working. But the over stuff you can be
• 1:12:59
  confident in is usually when there's just an outright problem
• 1:13:02
  that you can programmatically fix, like all the tables getting
• 1:13:04
  dropped. Yeah.
• 1:13:05
  There's there's talk about the horrific stuff that happens like
• 1:13:09
  when you're doing some processing, there's also just
• 1:13:11
  horrific things and data sets. Like, you know, I mean, I can't
• 1:13:15
  something like, please like pay your data engineers more because
• 1:13:19
  they have to go through this stuff and sometimes you just
• 1:13:21
  have to like like imagine like getting into the depths of the
• 1:13:24
  Internet, right. It's like it's there's just some
• 1:13:27
  stuff in there that like just the depths of.
• 1:13:29
  Reddit. Yeah, these poor folks have.
• 1:13:31
  To go through and and get you know, figure out what to do with
• 1:13:34
  it. So it's so.
• 1:13:36
  Funny, we need to create the Data Engineer Appreciation Day
• 1:13:40
  and we cry need two of those a year.
• 1:13:43
  It's not enough to just have one.
• 1:13:45
  They have really gotta have to fellas, this has been so
• 1:13:50
  awesome. I really appreciate the both of
• 1:13:52
  you coming on here and schooling me on this and just like taking
• 1:13:58
  the time to break this down, it was fascinating from my side.
• 1:14:04
  I love what you are doing and Mosaic.
• 1:14:06
  I know that you've got all kinds of fun stuff lined up that
• 1:14:10
  Denny, our mutual friend who also is working at Databricks
• 1:14:14
  project and he is. He keeps dropping bread crumbs
• 1:14:18
  about how you're coming out with some really fun stuff really
• 1:14:22
  soon, but he doesn't share anything else which is killing
• 1:14:24
  me. So hopefully we will be able to
• 1:14:28
  show off what you all create in the near future.
• 1:14:31
  Yeah, we can't. Wait to share it with you and it
• 1:14:33
  was thanks for having us this was this was so much fun.
• 1:14:35
  Thanks, tonne of. Fun.
• 1:14:39
  Yep. And Davis, I'll.
• 1:14:40
  Be patiently waiting for when the GPU training jobs go down
• 1:14:44
  and you get back on to that newsletter.
• 1:14:48
  Yeah, definitely. That's how you know something
• 1:14:50
  big is coming. Yeah, yeah, Davis puts that puts
• 1:14:53
  it out. You're like Ohh man mosaics
• 1:14:55
  gonna, you know mosaic database gonna drop something here is it?
• 1:14:59
  He got a rest just. Ohh Classic.
• 1:15:03
  Well, thanks fellas.