Latest / Agentic Conversations (formally mlops.community) / Performance Optimization and Software/Hardware Co-design across PyTorch, CUDA, and NVIDIA GPUs

Transcript

Performance Optimization and Software/Hardware Co-design across PyTorch, CUDA, and NVIDIA GPUs · Agentic Conversations (formally mlops.community)

• 0:00
  And there's entire products like Sagemaker Hyperpod was built
• 0:04
  specifically to have a standby, like a warm standby of GPU's
• 0:13
  that have been pre pre warmed up that can be swapped in.
• 0:18
  That's the whole premise of the product.
• 0:28
  First of all, Congrats on the book.
• 0:30
  I think that's a huge accomplishment and I want to
• 0:33
  dive all into everything about creating it and what you've
• 0:37
  learned along the way, what you've been up to since there is
• 0:41
  a GitHub repo that comes along with the book, which is super
• 0:44
  cool. So if folks want to get hands
• 0:46
  on, we're also going to go fairly low level.
• 0:50
  You know, this is something that I'm excited about because of the
• 0:57
  depth at which you bring and then being able to kind of
• 1:01
  traverse the whole stack up and down from the kernel to the
• 1:05
  applications. And you had asked me right
• 1:08
  before we hit record, what do you think about the whole
• 1:12
  narrative of not wanting software engineers?
• 1:15
  So maybe we start with that because I have some interesting
• 1:18
  takes, which is more along the lines of the applications right
• 1:23
  now. A lot of them are becoming very
• 1:27
  throwaway. And you can like think of it as
• 1:30
  akin to fast food apps where you just you build it, you take some
• 1:38
  time, maybe it's 20 minutes, maybe it's two hours.
• 1:40
  It's for a very specific you use case and you never have to
• 1:47
  productionize that. So you don't have to think about
• 1:49
  the million things that come with production because it's
• 1:51
  just built for you. Yeah, I guess, you know, coming
• 1:55
  from a large scale application, you know, background places like
• 2:01
  Netflix and you know, Databricks and then of course most recently
• 2:06
  AWS and, you know, various startups in between.
• 2:10
  I think about statements like that.
• 2:12
  And the first thing that I, I think of is this person has no
• 2:16
  idea what it takes to maintain and run an application.
• 2:21
  Now, if you're, if you're limiting it to just yourself or
• 2:25
  maybe 5 or 6 people, right? But actually I'm curious, you
• 2:29
  know, continuing your your thoughts there, where would you
• 2:34
  deploy it just out of curiosity, like a versal or like a?
• 2:38
  Yeah, Versal Railway, but even a lot of the stuff that I'm
• 2:41
  running, I'm just running locally.
• 2:43
  Yeah, yeah. So there is no deploy in that
• 2:47
  sense. Right.
• 2:47
  And then you're still using a database though, right?
• 2:50
  I assume like a supervise or something.
• 2:53
  Yeah, like I'm as hacky as just using Notion for my database is.
• 2:58
  It's sounds horrible. You know, it definitely defies
• 3:02
  all laws of everything that you would think.
• 3:06
  But it's like, Oh, well, if I just want like, for example,
• 3:10
  I'll give you the exact toy that I was playing with and building
• 3:13
  today. I kept getting sick of creating
• 3:17
  deep research reports and then wanting more information on just
• 3:21
  a piece of the report. And so I created a little bot
• 3:26
  that I can bring into the Google Docs and then I can tag at AI
• 3:33
  and say ask it questions or tell it to like, note this down and
• 3:37
  put it in my Notion database, whatever.
• 3:40
  And it does that. And that's just for me.
• 3:43
  Like I don't want anybody else to have access to that.
• 3:47
  And as your workflows change and as your frustrations mount, you
• 3:53
  will chip away at that. Exactly.
• 3:57
  And so being being a little bit old school in, you know,
• 4:02
  traditional software scaling distributed systems, I guess, I
• 4:06
  guess that that doesn't really matter, right.
• 4:09
  Like, yeah, some of those things don't matter.
• 4:11
  I mean, there's like backups. And I was watching, you know,
• 4:16
  podcaster Matt Berman, who I'm sure you're, you know, like
• 4:19
  familiar with. He was describing his whole
• 4:22
  workflow and, and, you know, and one of the things I noticed is
• 4:27
  he was backing everything up to GitHub.
• 4:29
  And then he he had these Cron jobs on top of Cron jobs that
• 4:32
  were doing backups on top of backups.
• 4:35
  And it's like, well, what if the one backup Cron job fails and it
• 4:38
  doesn't run till the next day? Now you're.
• 4:40
  So there's a lot of hackiness going on, of course.
• 4:43
  But. If it works for you and you can
• 4:46
  move quickly too and and I guess you're not waiting on someone
• 4:50
  else to release the feature or Google Docs to implement it or.
• 4:54
  That's the beauty. That is really the beauty of it.
• 4:57
  And so going back to the original idea of like, oh, are
• 5:00
  we going to need software engineers?
• 5:02
  I still am 100% convinced that we're going to need software
• 5:06
  engineers because me doing that is nothing like me taking
• 5:13
  something and bringing it to production and I understand that
• 5:17
  gap. What I think is just different
• 5:19
  is now how custom we are able to quickly spin things up.
• 5:24
  And also what you were talking about too with like maintaining
• 5:28
  software is kind of a pain. 100% agree.
• 5:31
  And I don't want to now have 50 or 100 or 200 different
• 5:37
  personalized apps that I have to maintain.
• 5:40
  I get the feeling that it's probably faster for me to just
• 5:44
  start from scratch again next time, then have to maintain it.
• 5:49
  Yeah, yeah, 100%. I was working on a project over
• 5:54
  the holidays. I, you know, picked up for like
• 5:56
  a buddy of mine and he's ACEO and runs a, you know, pretty
• 6:02
  tight ship and he's got like really good revenues and he
• 6:05
  keeps his expenses low. And, and he was mortified when
• 6:11
  during one of our stand ups, I said, yeah, I just threw
• 6:14
  everything away that I've been working on for the last two
• 6:16
  weeks. And I, and I started over and it
• 6:19
  actually, we had to talk about it offline.
• 6:24
  Like, I didn't realize he, you know, he, he, he wanted me to
• 6:28
  also kind of share information with his team on my workflows.
• 6:35
  And I had already gotten to the point that you just mentioned,
• 6:38
  which is I could start over and it's actually faster and cleaner
• 6:42
  And it doesn't have to maintain all the old tests.
• 6:44
  It's just going to write new tests, you know.
• 6:46
  And so I think they're, we're, we're in the shift.
• 6:50
  Yeah. Curious.
• 6:51
  Do you read all of the code? That's being, That's being.
• 6:54
  General, I don't read any of. It to be honest.
• 6:57
  I mean, occasionally I'll jump in and try to read and figure
• 7:02
  out what's going on. But honestly, if I have a
• 7:05
  problem or if I'm debugging stuff, there's First of all,
• 7:08
  there's some awesome skills that are just open repos on GitHub
• 7:13
  and one of them is like a debugger skill Agents by ODA.
• 7:22
  I just I was checking it out a few days ago and I got it and
• 7:26
  since then I haven't had to use my other trick which is using
• 7:31
  the playground skill to create a visual diagram of what's going
• 7:36
  on with the code and then help me debug it.
• 7:40
  Interesting. Yeah, that was that was awesome.
• 7:43
  And that was my go to because then I could isolate like what's
• 7:48
  the problem, what is happening and kind of explain it in a
• 7:52
  diagram and then the prompt would just be there in the
• 7:55
  playground. So my thing was I would say
• 7:59
  create a playground skill to help me debug the data flow of
• 8:04
  this app or something like that, or all the possible scenarios of
• 8:08
  why this isn't working. And then it would have the the
• 8:13
  debug prompt that I could just copy paste back into when I
• 8:17
  would isolate the issue. It's like, oh this part of the
• 8:20
  app is not rendering this part, this button is not clicking
• 8:24
  through or whatever. Then boom, it just helps me.
• 8:27
  Yeah, transfer that. I actually didn't even know
• 8:29
  about that. I I was having it generate
• 8:32
  mermaid diagrams and generate markdown.
• 8:35
  But all right, I have to go look into these and these are skills
• 8:38
  for like codecs or for Claude or?
• 8:41
  Yep and it's so nice. I feel like the playground skill
• 8:46
  is a skill that keeps on giving. Anything I throw at it, I can
• 8:51
  try and get it to do stuff. Like the other day I was
• 8:55
  creating a Horsey Ranch game with my daughter to try and
• 9:00
  teach her a little bit of geometry and we wanted to change
• 9:04
  the appearance of some of the game characters.
• 9:08
  And I would say like the playground skill is really
• 9:11
  something for when you want a little more fine grained ability
• 9:17
  than words can offer you. So that means you don't want to
• 9:22
  have to explain things with more intensity in words.
• 9:27
  You just want to have a slider that you can Jack up the
• 9:31
  intensity or, or bring it down with, you know.
• 9:34
  And so with the appearance of these different game characters,
• 9:38
  we were able to change the colors and change the size of
• 9:41
  their clothes or the size of their eyes, that type of stuff.
• 9:46
  And it works with any it. It just figures it out.
• 9:50
  It just works, huh? It's amazing how well it figures
• 9:53
  it out too, like throwing some random stuff at it and it is
• 9:57
  wow. Yeah, impressive.
• 10:00
  Yeah. Any tips for educational things
• 10:04
  for like, I have two nephews and all they do is play like Roblox
• 10:08
  all day. And they they haven't made the,
• 10:10
  you know, the transition. My sister doesn't let me force
• 10:15
  them to learn, you know, computer science and and things.
• 10:18
  So I'm trying to do subtle ways. But like I think I've heard one
• 10:22
  of the tips is like you can, you know, have Chachi BT or Claude
• 10:28
  like ask questions back. So when when they ask a question
• 10:33
  about something to research coming back, have had it ask
• 10:36
  questions. Any other thing like that?
• 10:38
  What I've started doing with my daughter on the way to school
• 10:42
  sometimes is we will put it on voice mode and then I'll say,
• 10:51
  you know, hey, we're trying to learn about geometry and I'm
• 10:56
  with my 7 year old daughter and she really likes horses.
• 10:59
  Can we, can you tell us a story that like incorporates the
• 11:04
  diameter and the radius and all that?
• 11:08
  So that's fun. Another one is trying to, yeah,
• 11:11
  create this horsey ranch game, which she loves.
• 11:16
  And we plugged in a voice agent too, so she can talk back and
• 11:20
  forth to it. And that is really cool 'cause
• 11:23
  it's just so easy. It's ridiculously easy to build
• 11:28
  these things. But then sometimes I'm so out of
• 11:31
  my depth and it's like, oh, we hit a bug and this isn't
• 11:35
  rendering and I don't know what's the problem.
• 11:38
  And so then I have to sit there for 30 minutes trying to debug
• 11:42
  and my daughter gets bored and she's long gone.
• 11:44
  You. Know she's gone hilarious.
• 11:48
  But dude, talk to me about this book because there's that Venn
• 11:53
  diagram or that triangle of skills that you now need, and
• 11:57
  maybe give an overview of the book that you wrote and what
• 12:02
  kind of inspired you to write it, baby.
• 12:04
  Yeah, I mean, this is the third book I've written with the
• 12:08
  O'Reilly folks and each of them, the common theme was I was
• 12:13
  trying to, you know, learn as much as I could about a
• 12:19
  particular domain. So I joined yeah, I joined
• 12:22
  Amazon, what was it, 2019? And I I was, you know, came
• 12:27
  through an acquisition. I had a startup that was doing
• 12:30
  inference, you know, very much like a base 10 today or like a
• 12:35
  together AI or a fireworks before.
• 12:38
  Your time a little. Before you it was, you know,
• 12:41
  before like LMS, so like you can imagine it was Tensorflow
• 12:45
  models. Pie Torch wasn't even big, you
• 12:47
  know, back in 2016, like we had one pie Torch customer and they
• 12:51
  were the annoying one that we didn't know anything about.
• 12:53
  Like we would help them but they had to do everything on their
• 12:56
  own. We just gave them our, you know,
• 12:57
  product and they can fit it in. But yeah, the startup was
• 13:02
  interesting. So I had originally started at
• 13:05
  Netflix when I moved to the Bay Area.
• 13:07
  I'm, I'm from Chicago, but moved to the Bay Area to get a lot of
• 13:10
  scale. My buddy recruited me there.
• 13:13
  Very quickly realized that my little financial apps and toy
• 13:16
  apps that I was building for companies in Chicago are very
• 13:20
  different than, you know, 300 million customer, you know,
• 13:25
  hundreds of thousands of streaming servers.
• 13:28
  So I joined the the first streaming team.
• 13:30
  I was on the streaming team from the beginning.
• 13:33
  That's when when they split off DVD.
• 13:35
  And so I never had anything to do with the DVD side.
• 13:39
  I was always streaming and then did that.
• 13:43
  And then I, I joined Databricks and and I, I got to see data at
• 13:48
  scale, you know, that's where I met Holding Corral, who I think
• 13:52
  you know, And yeah, so she's she she kind of turned me on to the,
• 13:59
  the, the book bug writing. Yeah, yeah.
• 14:03
  And I had always wanted to write a book.
• 14:06
  I, I submitted about 7 or 8 proposals to the O'Reilly folks.
• 14:13
  I always got rejected. Yeah, even with, you know,
• 14:17
  Holden's help and with other people's help.
• 14:19
  So they're, they're very selective.
• 14:21
  What's interesting about the book proposal process, it's very
• 14:25
  much like when you're pitching your startup to VCs.
• 14:31
  And so it, it was not coincidence that I had to pitch
• 14:37
  to VCs with my startup and and then, you know, eventually sold
• 14:41
  to like Amazon. And then when I went to write my
• 14:46
  the most successful book proposal, I basically followed
• 14:49
  the 1213 slide Sequoia, you know, pitch deck, which I had
• 14:53
  used for all my pitches. And because what they want to
• 14:57
  know is, you know, why now, right?
• 14:59
  Like, why is this book relevant now?
• 15:01
  Why, you know, yeah, why me like, why am I the right, like
• 15:06
  author for it? And they want to know
• 15:09
  competitors and they want to know the Tam Yeah, like what's
• 15:14
  the addressable market for this book?
• 15:15
  Right. And so yeah.
• 15:19
  And if you ever do want to write a book, you know, for you or for
• 15:23
  your like audience with O'Reilly, they, you know, give
• 15:26
  you the the proposal when you have to kind of fill it out.
• 15:28
  And it's super annoying, just like filling out a pit.
• 15:32
  You know, the pitch deck gets and then you have to pitch it to
• 15:34
  them, right? So by the 3rd book, they were
• 15:37
  like, OK, yeah. You know, just just like, give
• 15:39
  us everything because I had done it so many times.
• 15:42
  But oh, back to this. So there's always this thing of
• 15:49
  there's so much material and I'm getting it from 100 different
• 15:52
  sources. Like the most recent book, the
• 15:56
  AI Systems Performance Engineering book, I was pulling
• 16:00
  tips off of Twitter, you know, yeah, X and I was getting tips
• 16:05
  out of, you know, Discord channels, which like Discord
• 16:11
  scares me. I don't like Discord.
• 16:12
  Like it doesn't make any sense to me, right?
• 16:14
  So, so and I can't copy and paste or something out of
• 16:17
  there's there's something weird. So I have to do screenshots and
• 16:20
  you know, it's very strange, strange land over there.
• 16:24
  But. And so once you, you realize
• 16:27
  this and Nvidia's not really known for their documentation,
• 16:32
  right? And it's, it is really, really
• 16:35
  bad. And when you finally land on
• 16:38
  something that explains the hardware, it's in the context of
• 16:42
  some super obscure, you know, fluid dynamic algorithm that
• 16:49
  that kind of has some similarities to it what an LLM
• 16:51
  could do, but really it has has nothing to do.
• 16:54
  So so this was my attempt to really bring together Pie Torch,
• 16:59
  you know, framework CUDA, which is always kind of an elite type
• 17:03
  of technology and and framework to learn and then the hardware
• 17:09
  and, you know, the algorithms. So really it's it's hardware,
• 17:13
  it's the software and then it's algorithms.
• 17:16
  And that's called Co design. So those three together.
• 17:19
  Yeah, that's what this book focuses on it.
• 17:22
  Feels like you're doing God's work because there's like 100
• 17:26
  people in the world that understand those three things
• 17:30
  that you just mapped out. Yeah.
• 17:32
  And my goal was to make that, you know, 100,001 one million
• 17:37
  people, right? And and do it in a, in, in a, in
• 17:43
  a domain that is relevant today. Because I like I start reading
• 17:49
  these CUDA books and by the end of it I can do some crazy, you
• 17:53
  know, numerical, yeah, optimization, you know, thing
• 17:57
  that that I can't, I can't make that connection into the pie
• 18:02
  torch world and stuff. And so.
• 18:04
  I see where you're going. Yeah, because it's just so far
• 18:09
  in left field. They're like, how does this
• 18:11
  relate to the whole world of LLMS and machine learning?
• 18:16
  Yeah, and I I while while writing the book, I recognized
• 18:23
  quickly that it's a cultural thing within NVIDIA and hot
• 18:29
  take, no one from NVIDIA was allowed to be a formal reviewer
• 18:38
  on my book like right, like let alone co-author.
• 18:42
  But that, that was already rolled out quickly when I, I had
• 18:45
  approached a couple friends of mine, they were not willing to
• 18:48
  give quotes. They were not willing to be
• 18:50
  associated. Now, that said, they were super
• 18:54
  curious and they helped me quite a bit.
• 18:57
  And they were able to, you know, unlock a few concepts that to me
• 19:01
  weren't, weren't driving that I couldn't get from the forums.
• 19:03
  And I unfortunately, I can't thank those people publicly.
• 19:07
  So I, you know, I've had to thank them privately.
• 19:10
  And and that's, yeah, that's a bummer.
• 19:13
  Yeah. That's.
• 19:13
  Right. Yeah, It's funny you mention
• 19:15
  that because my buddy and past Co host of this here podcast,
• 19:21
  David Aponte, he's been working on a lot of real small.
• 19:27
  He's trying to figure out all of the different ships, not just
• 19:30
  the NVIDIA ones. But one thing he said to me in
• 19:33
  passing that I, it has stuck with me since is he was like,
• 19:37
  yeah, So, you know, NVIDIA chips, they don't come with like
• 19:41
  a user manual on how they are. Built right, right?
• 19:44
  Nobody knows what's going on behind the veil.
• 19:48
  Yeah, and that was the DeepSeek moment, right?
• 19:52
  That was one of the, and my book starts off that way, like
• 19:55
  chapter 1. I, I kind of explained, you
• 19:58
  know, like why now? Basically like why did did and
• 20:03
  there's a lot of DeepSeek references throughout.
• 20:07
  Right, like knowing that there's going to be the next version of
• 20:09
  DeepSeek and all that. So like I'm not trying to be on
• 20:12
  on the cutting edge, right? This book took a year and there
• 20:15
  was like 9 models that that surpassed DeepSeek in that time.
• 20:19
  But what was interesting about the DeepSeek moment is it made
• 20:23
  people realize that even with hardware constraint, you know,
• 20:26
  Chinese export like restrictions, DeepSeek was
• 20:31
  somehow able to basically reverse engineer the some lesser
• 20:38
  documented. Yeah, right.
• 20:41
  Like the NVIDIA folks are like very, you know, careful to say,
• 20:43
  no, this was a documented feature.
• 20:45
  It's just we didn't make it easy for you to find the docs
• 20:48
  basically. Right.
• 20:49
  And so they like, yeah, one of the many like innovations out of
• 20:55
  DeepSeek was that they, they found instructions that were not
• 20:58
  well documented that could, you know, bypass the L2 cash under
• 21:01
  certain circumstances. And of course, the NVIDIA folks
• 21:05
  that are optimizing for other use cases, they would never
• 21:08
  think to do this. And so that's why it's not like
• 21:10
  a first class like CUDA API. But if you know what you're
• 21:15
  doing, you can drop down basically to the assembly and
• 21:18
  you know, it's called PTX or it's it's it's PTX for the
• 21:24
  acronym, but they call it PTX. Yeah, the NVIDIA folks call it P
• 21:28
  techs. So you have to be able to like
• 21:29
  translate that. So yeah, we have GTC coming up.
• 21:32
  Are you coming to that that conference?
• 21:33
  No, but oh man, that's my favorite one of the year.
• 21:37
  Yeah, that is a wild one. I've heard very good things
• 21:40
  about it. We have some half off, I think
• 21:46
  tickets for the community and one thing that is pretty much
• 21:51
  impossible to get is free tickets to that.
• 21:55
  Oh yeah, no. Not unless you're speaking
• 21:58
  right. Yeah, yeah, yeah.
• 22:00
  I spoke a bunch of years ago. Oh, nice.
• 22:03
  I was going to ask you more about just like diving into the
• 22:07
  book and other sections of it. What were some pieces that for
• 22:12
  you, you feel like, OK, this was, as you said, challenging to
• 22:18
  put two and two together to figure out how they coexist.
• 22:22
  But now that you have it, it's like shining a light on
• 22:27
  something that was very in the dark.
• 22:31
  Yeah, and the book takes very much like a bottoms up approach,
• 22:37
  right. So there's a sort of intro
• 22:38
  chapter to explain kind of the context of the book and what
• 22:43
  we're going to, you know, like some of the key terms, like one,
• 22:46
  one term that is related to the hardware software algorithms
• 22:50
  trifecta, if you will, Venn diagram, if you will, is the
• 22:54
  term mechanical sympathy. And this is a term that
• 22:58
  actually, I think came about from the Java world, even where
• 23:02
  it and it, it was a term that references the fact that if
• 23:08
  you're going to be a really good developer, you have to
• 23:12
  understand like the environments.
• 23:15
  And in the Java context, that's really understanding the JVM
• 23:18
  right in, in, you know, C++ CUDA context, that's understanding
• 23:24
  the hardware, the, the, you know, GPU hardware, because
• 23:28
  there's every generation is, is more than just an incremental or
• 23:34
  even like a doubling of power. There's, you know, one thing I
• 23:38
  realized when I was diving deep is that the NVIDIA folks are
• 23:42
  really good about working with the researchers to figure out
• 23:47
  what they can abstract away into the hardware.
• 23:52
  And of course they're thinking about it at a much, much deeper
• 23:55
  level. You know, then, then when you
• 23:57
  abstract things at a software level, right, you know,
• 24:00
  software, we're, we're trying to abstract like the database,
• 24:03
  we're trying to abstract like the, like distributed systems,
• 24:07
  you know, at companies like Temporal, for example, they,
• 24:10
  they're trying to abstract away retries and you know, all the
• 24:15
  distributed systems pain. But at the hardware level,
• 24:19
  implementing operations that are common for the transformer,
• 24:24
  right, like softmax, you know, and so when they work with the
• 24:33
  researchers, hopefully they're able to get down to the level
• 24:37
  of, OK, you're trying to do this within Pytorch.
• 24:40
  What can we do to the hardware to make it better for you for
• 24:43
  the next generation? That was a big unlock for me
• 24:46
  because I always in my head, I still picture NVIDIA as like the
• 24:51
  gaming company that just got lucky, right.
• 24:54
  But you know, Jensen's is is a smart guy and Jensen is works
• 25:00
  very closely with these, you know, researchers.
• 25:02
  If you remember when like open AI first launched and Jensen
• 25:06
  showed up with that, you know, big, big massive DG Xbox and
• 25:11
  yeah, there was Elon was there and everyone, right got back
• 25:13
  when things were were good and stable for, for open AI.
• 25:18
  So that was a real eye opener. Other things you, you start to
• 25:26
  learn some of the what's called Jensen math, right?
• 25:29
  So Jensen's the CEO of like NVIDIA and they, they do these
• 25:35
  tricks. It it's sort of related to
• 25:38
  marketing math. You know, it's like, OK, well
• 25:40
  we're going to talk about the number of flops for this chip in
• 25:47
  in and and talk about the maximum flops even though the
• 25:51
  transformer doesn't actually use that that type of of like
• 25:55
  algorithm, right? Like sparsity, yeah, let's say
• 25:57
  24 sparsity or something, right. It's not as common in like LLM
• 26:01
  land as it is maybe in other domains, but we're going to talk
• 26:05
  about it as though the chip can do you know, yeah, whatever 180
• 26:10
  like teraflops or something so. It's like, we just blew that
• 26:14
  number out of the water. Can you believe how good this is
• 26:17
  now? And then the 2% or .02% of
• 26:23
  people that actually know what's going on, they're like, So what?
• 26:27
  And what's what's bad about that is you go out and buy this chip
• 26:34
  and you give them the 70% margin and you know, whatever, and you
• 26:38
  get this thing installed and you're like, where's my
• 26:41
  teraflops? You know what's going on here?
• 26:43
  And then they're like, Oh yeah, that's only under certain
• 26:45
  condition. The other little sneaky part
• 26:48
  that like NVIDIA has done a couple times in their history,
• 26:52
  including Blackwell, is they actually end up putting 2 chips
• 26:56
  onto 1. And you know, so I think it's
• 26:59
  called MCM multi chip module something.
• 27:03
  And so really Blackwell is 2 smaller Blackwells and Grace
• 27:07
  Blackwell is 2 smaller Blackwells with a Grace CPU.
• 27:10
  So it's got this like 2 eyes and a little nose and they have, you
• 27:14
  know, super fast interconnects and they can keep the memory.
• 27:19
  The memory has to stay coherence.
• 27:23
  Grace Blackwell, the reason why people either love it or hate it
• 27:27
  is because the CPU has so much memory now you basically have a
• 27:34
  gigantic pool like relative to the smaller GPU memory right now
• 27:39
  you can access the CPU memory yeah, it's slower, but I could
• 27:43
  load more, you know like parameters.
• 27:46
  I, I can load larger things. I, I can do things.
• 27:49
  And so when you're in fact, I, I just got done working with a
• 27:53
  company doing performance benchmarks on a Grace Blackwell
• 27:57
  system GB 200. And the calculus, the, the way
• 28:05
  that the kernels are written has to change.
• 28:08
  And so it's, it's not enough to just take your same PIE torch
• 28:13
  code and run it on AGB 200. Yeah, you have to think about
• 28:16
  other things and you have to and there's, there's, there's one
• 28:21
  other concept and it's the arithmetic intensity.
• 28:25
  And that's a property of an algorithm.
• 28:28
  So that's a property of like the attention, like algorithm
• 28:32
  within, you know, Transformers, which is really the most
• 28:34
  critical part where tokens come in, you know, words come in and
• 28:40
  they're, they're, they're then sort of related to each other
• 28:43
  and like how much attention are they paying to each other?
• 28:45
  It's a very expensive computation.
• 28:47
  It's, you know, O n ^2 roughly. And, and so if if you're not
• 28:56
  paying attention and, and you so that term, the arithmetic
• 29:03
  intensity is how much like data movement has to happen per
• 29:09
  compute. And so, and you know, data
• 29:13
  movement is actually the slowest part, right?
• 29:16
  Like moving it from memory into the chip like once, once the
• 29:22
  data moves into the chip and into the registers and then it
• 29:26
  can get the full T flops, right? The, the full compute and like
• 29:29
  compute keeps going up like crazy.
• 29:30
  It's just memory bandwidth is really the limiting factor.
• 29:34
  So there's a lot more. Yeah.
• 29:37
  That leads nicely into something that you were mentioning before
• 29:40
  we hit record, which is all of these things that we now have to
• 29:44
  think about when building platforms for AI, which we
• 29:50
  didn't necessarily have to when we were just thinking about the
• 29:53
  CPU's. These like these GPU's are just
• 29:58
  little side cars, you know, to the CPU, right?
• 30:02
  I mean, that's why they're called accelerators.
• 30:04
  And that gets to another point where like the more of these
• 30:08
  accelerators you have now you've got like bigger problems.
• 30:12
  And, and you know, there's the the NVL 72 Grace Blackwell,
• 30:20
  which is what a lot of the neo clouds are starting to deploy
• 30:23
  now. The land, the labs of the, you
• 30:25
  know, Verda out of Finland and cor weave and stuff.
• 30:29
  They and that's got, you know, 9 trays of, you know, eight of
• 30:35
  these like GPU's. So that's the 72 and these are
• 30:40
  all interconnected in a single rack.
• 30:43
  So that's the beauty of it is it's a single rack uses a crazy
• 30:47
  amount of power, weighs like, you know, 3000 lbs, which I
• 30:51
  don't know how many kilograms that is 1500, yeah.
• 30:56
  So, yeah, pretty heavy, requires massive cooling, you know, so,
• 31:00
  and I guess they're relatively efficient in, you know, compared
• 31:05
  to if you have the same kind of of stack or set of, you know,
• 31:10
  servers up and running. But they're they're beasts that
• 31:14
  that have to be very carefully interconnected.
• 31:18
  So now we have something that's called the NV link, right, the
• 31:21
  NVIDIA link. And one little secret by the
• 31:24
  way, that NVIDIA doesn't talk about much probably because of
• 31:27
  regulatory concerns is they, they acquired a, a company in
• 31:33
  2019, I think it closed in, in 2020 called Mellanox.
• 31:40
  And that's what converted right like that company into not just
• 31:47
  a chip company, right? So now we've got like NVIDIA
• 31:50
  started off just building chips. Now they're an AI systems
• 31:53
  company. So they've got the switches that
• 31:56
  the, you know, networking switches, they've got like the
• 31:59
  really fast interconnects and they just brought in a lot of
• 32:03
  expertise that company also that they acquired.
• 32:06
  What were the creators or the most most commonly or yeah, I
• 32:13
  think the, the, the most popular, like creators of
• 32:19
  InfiniBand, right, which is like sort of next generation
• 32:23
  Ethernet. And so while there's still a lot
• 32:26
  of data centers out there that are using Ethernet and you know,
• 32:29
  you, you pretty much want to stay away from those by the way,
• 32:31
  but you, you want to stick with InfiniBand, you know, and then
• 32:36
  if you want, we, we can kind of get in at some point to some of
• 32:40
  the cloud providers have their own networking and so they don't
• 32:43
  use InfiniBand. Yes, AWS being a very big one
• 32:47
  and it it always caused problems with us with customers because
• 32:55
  there was an extra layer where we had to adapt, you know, like
• 32:58
  the NVIDIA libraries had to then work with this custom layer that
• 33:02
  was called EFA, which required Amazon to then have backward,
• 33:08
  you know, forks of pie Torch and it was just mass after mass
• 33:12
  after mass and and that's that's still with the NVIDIA GPU's
• 33:15
  that's not even getting into Tranium and.
• 33:18
  That's just changing one piece of the puzzle.
• 33:21
  Yeah, yeah. And when you don't buy into the
• 33:25
  NVIDIA like sort of reference architecture, they call it,
• 33:30
  which includes all of that, you know, networking, there's a lot
• 33:35
  of things that you have to do on your own that like NVIDIA is
• 33:38
  just like, sorry, like we can't help you.
• 33:40
  So do. They void the warranty.
• 33:43
  Yeah, right. That's.
• 33:47
  And that's something else too, actually.
• 33:49
  Good. Good segue.
• 33:50
  These chips, I don't know how familiar you are with the
• 33:53
  failure rates of these GP US. I mean, yeah, yeah, these things
• 33:57
  run. I wanted to ask you about the
• 33:59
  reliability in that regard because I've heard from all
• 34:03
  different areas on how unreliable they are, just the,
• 34:08
  the chips themselves, but then also everything that comes
• 34:12
  around the data center and not having power to 1/4 of the whole
• 34:18
  data center and it's going out. And so you have to have all of
• 34:22
  these extra fail safe modes because you basically have to
• 34:26
  expect that things are going to get switched off.
• 34:30
  And so the the chips themselves are very unreliable.
• 34:34
  And then you're talking about like just chips blowing up or
• 34:37
  chips basically being rendered not usable.
• 34:41
  Yeah, Every single cluster that I get on, like a new cluster,
• 34:46
  the first thing I do is, is put it through sort of AII, just,
• 34:51
  you know, put the pedal to the metal, yeah, as they say, and
• 34:56
  figure out like which ones are on the verge of, of burning out.
• 35:00
  And I'll burn out 2% just within a couple, you know, minutes.
• 35:04
  Because the, you know, they're purposely designed to run hot,
• 35:09
  which is why they need a lot of cooling even.
• 35:12
  Yeah, I don't have one here. But those little small DGX
• 35:14
  sparks that they came out with, if you've ever run like
• 35:18
  something on there, like, I mean, that thing can, can burn a
• 35:22
  child, right? It's not safe to have your home.
• 35:25
  Like, I don't think it's going to burn down a house or
• 35:27
  anything, but it's going to burn your skin.
• 35:29
  It's and and that's one, you know, Blackwell, it's a it's a
• 35:33
  Grace Blackwell. But and that's also why that
• 35:38
  particular machine is is a little bit constrained.
• 35:42
  They don't have certain instructions for some of the
• 35:45
  tensor cores and things because they they physically can't keep
• 35:49
  it cool enough in a consumer device, right.
• 35:51
  So it's not really the best device to learn like real CUDA
• 35:54
  programming on, but yeah, they'll, they'll, they'll burn
• 35:58
  out their, they throttle super quickly.
• 36:01
  So when you're doing any kind of benchmarking, what you're
• 36:04
  supposed to do is lock the, the clock speed, you know, the like
• 36:08
  GPU speed, so it doesn't get up above a certain temperature.
• 36:12
  Because yeah, otherwise at yeah, any given point in time they're
• 36:16
  they're being throttled so that they don't burn out.
• 36:19
  And you're talking even the new ones, right?
• 36:21
  Yeah. Oh yeah.
• 36:22
  Especially the new ones, Yeah. Oh yeah.
• 36:24
  Because I think I heard somewhere where there's like a
• 36:27
  warm up period that you have to go through in order to know like
• 36:31
  is this going to go or not? For sure, yeah, yeah, that's
• 36:35
  what I was referencing to, to see because they all you know,
• 36:42
  that's another thing that you, you realize when you start
• 36:45
  thinking about hardware and and reading about the failure rates
• 36:48
  and the the fabs and how they're designed.
• 36:51
  Like at the end of the day of, of the day, they're they're
• 36:56
  still manufacturing like defects, right, that are
• 36:59
  happening. And of course, they're
• 37:01
  generating millions of these, these chips.
• 37:04
  They're, you know, building millions of of these per year.
• 37:07
  And they have to be at a specific temperature, you know,
• 37:10
  which is why the Arizona fabs have always been a little bit
• 37:12
  controversial because so hot. Yeah, it's hot.
• 37:17
  There's no water. I mean, you're in the desert,
• 37:19
  right. And even if you do like get
• 37:22
  through all that, even even the ambient air being different than
• 37:27
  what like Taiwan Semiconductor has in, you know, like Taiwan,
• 37:32
  it's just a different climate, just a different thing.
• 37:34
  And if any of those are different than what they
• 37:38
  expected, they have to then go through and and, you know, add
• 37:42
  extra filters or do something with with the air.
• 37:45
  So, yeah, there's a lot going on.
• 37:47
  Why with these things? Did you uncover why they're so
• 37:51
  finicky? Like why can't they make them so
• 37:55
  it's less precious and just like it'll hardened?
• 37:59
  Yeah, well part of it is right. Like, do you remember when you
• 38:06
  maybe built your first computer or your friend built, built his
• 38:09
  first computer and you, you found out you can over clock
• 38:13
  the, you know, GPU or the CPU or something?
• 38:16
  And you're like, why the hell would I not over clock this?
• 38:18
  This is obviously I'm going to do this, right?
• 38:20
  And then you do it and then you realize, OK, it burned out.
• 38:24
  Now I got to go back to Fry's or Micro Center or whatever Best
• 38:28
  Buy and get a new, you know, chip.
• 38:31
  So part of it is they're optimizing for performance,
• 38:37
  right? And they but even suffered from
• 38:41
  that. There's just at scale, there's
• 38:45
  just going to be like, you know, like molecular issues, right?
• 38:50
  And so the best thing you can do, because I'm given like new
• 38:53
  clusters all the time to say, hey, you know, help us tune this
• 38:55
  thing. And so that's the warm up period
• 38:58
  is like really figuring out and in a cloud type of environments,
• 39:04
  I hammer those things because I'm like, you guys should have
• 39:07
  already warmed this up. Now it's my turn from the sort
• 39:11
  of application level to make sure that that my application,
• 39:15
  you know, that, that my workload.
• 39:17
  And then it I would say probably 60% of the time, 60 to 70% of
• 39:23
  the time they have to spin up a new cluster because something
• 39:26
  went wrong. Yeah.
• 39:28
  And there's entire products like Sagemaker Hyperpod was built
• 39:32
  specifically to have a standby, like a a warm standby of GPU's
• 39:41
  that have been pre pre warmed up that can be swapped in.
• 39:46
  That's the whole premise of the product.
• 39:49
  Well you were also talking about a an AB D type of product, but
• 39:54
  for GPU's that from the. Yeah, Tell.
• 39:58
  Me more about that. Yeah.
• 39:59
  So for for listeners that might not be familiar, AppDynamics is
• 40:03
  a company, they're, you know, relatively old right now.
• 40:07
  They were new when I joined Netflix and that was I think
• 40:09
  2011 and App Dynamics was, you know, think of it like an agent.
• 40:15
  Not, not the current definition, you know, yeah, of an agent, but
• 40:18
  like, think of it like, right, like a demon thread or a process
• 40:24
  that runs on all of your servers.
• 40:26
  So back then, you know, Netflix, it was all Java based streaming
• 40:30
  servers. And, you know, they, they could,
• 40:36
  each of those processes that were running were, were, were
• 40:40
  very, very like lightweight, but they could, you know, track all
• 40:45
  of the, the like, network traffic.
• 40:47
  So we knew which of our, you know, 'cause like we had
• 40:50
  hundreds of these micro services.
• 40:51
  It was, you know, very much inspired by like Amazon and
• 40:55
  Bezos and the, you know, micro service.
• 40:58
  And so just trying to even get an understanding of how our, our
• 41:02
  services, we're all connecting to each other and the health,
• 41:05
  that was a big win. But it also tracked the memory.
• 41:08
  It tracked any sort of oddities going on.
• 41:12
  You know, this was all CPU based.
• 41:14
  And so there's this company that I met actually because of the
• 41:17
  book that's called Zim Trace, Zym Trace.
• 41:20
  They're actually based out of London and they, they are X like
• 41:24
  app dynamics people, right? Which is why I was very
• 41:27
  interested in them. And they remember when, you
• 41:30
  know, Netflix was their first customer.
• 41:32
  And so add in all of the similar things that AppDynamics did, but
• 41:39
  then also now add on all this complexity about the like GPU
• 41:45
  power, the GPU temperature, any like they can detect when a GPU
• 41:50
  is being underutilized and they can make suggestions at at the,
• 41:54
  the full system level, you know, and say, hey, maybe.
• 41:58
  And they're very much like, right, like LLM aware, yeah,
• 42:03
  this product. And so it can actually look at
• 42:05
  your training workload or your inference workload and say, hey,
• 42:08
  it looks like you need to add a few more like experts in your,
• 42:13
  you know, mixture of experts model over here and, and on the
• 42:16
  servers underutilized or the networking is not, not optimized
• 42:20
  for. Yeah, for this particular setup.
• 42:22
  Yep. Going back to when you're
• 42:24
  warming up these GPU clusters, this is primarily with you
• 42:30
  running inference loads, right? It's not for training.
• 42:33
  Yeah, I do both. I mean it, it depends on and you
• 42:38
  know, keep in mind RL, right. This is kind of the The Dirty
• 42:41
  little secret is that it's both, right.
• 42:44
  It's doing inference and it's it's doing, you know, back prop
• 42:47
  and updating weights and things. So, but interesting that you say
• 42:52
  that because Hyperpod was actually built for training and
• 42:58
  the ability to if if a GPU or, or like an instance dies, we
• 43:03
  have this warm standby that can come in and the job can then
• 43:06
  continue without this like major, you know, job restart
• 43:11
  where you might lose the last couple hours because your last
• 43:13
  checkpoint was a couple hours ago and things so and then very
• 43:18
  quickly customers were like, Hey, we actually want to do like
• 43:21
  Riley inference on this thing either.
• 43:23
  We're, we're generating synthetic data for the next
• 43:26
  training run or we're doing RL where we have to do a bunch of
• 43:32
  inference and then, you know, compare it to the OR like verify
• 43:36
  it right in an RL, in an RLVR, right where you're doing
• 43:41
  reinforcement learning with verifiable rewards where you're,
• 43:44
  you know, either like executing code or you're doing math
• 43:47
  equations or you're running some kind of experiment and then
• 43:51
  verifying it. So you undoubtedly have looked
• 43:56
  at or have a good pulse of what's on the market for whether
• 44:02
  that is the Lambda Labs of the world where you can get down
• 44:06
  into the bare metal and kind of control everything from what I
• 44:10
  understand, or it's the base 10s where it is straight like, hey,
• 44:14
  just throw whatever you want and we'll give you this API.
• 44:19
  Are there any that or do you have any commentary on what's
• 44:24
  out there right now? Yeah, sure.
• 44:28
  Yeah. And obviously any, as we've seen
• 44:34
  over the years, all of these companies end up looking like
• 44:37
  the, you know, yeah, over the years they all start to build
• 44:40
  each other's features. So I would argue, yeah, well,
• 44:45
  you, you start off with one set of skills, get a little success,
• 44:50
  and then customers start to ask for more and more and more.
• 44:53
  Perfect, perfect example here would be Lambda Labs.
• 44:59
  Actually Lambda was primarily training.
• 45:04
  They, they started to dabble a bit into inference.
• 45:09
  They even had their own chat, like Lambda chat or something at
• 45:12
  some point, which which I didn't really use that much.
• 45:17
  But it's it's very obvious when you see their road map about
• 45:24
  which kind of customers that they're now starting to get like
• 45:28
  the, the the first initial tranche, you know, might have
• 45:30
  been like the big labs, right? I don't know for sure, but and
• 45:35
  and then and then to contrast folks like base 10 and together
• 45:39
  and fireworks, you know, like those folks, they were just
• 45:42
  trying to get the open source models out like behind an
• 45:45
  endpoint, maybe supporting fine tuning so that you can customize
• 45:49
  the open source models. Very different model, right, The
• 45:52
  the labs or or sorry, you know, places like Lambda labs and you
• 45:57
  know, core weave. They're there to customize and
• 46:00
  pre train and do the full full training life cycle.
• 46:03
  But they're realizing that because inference is like a
• 46:08
  major, you know, the these apps are are running 24/7, right?
• 46:12
  People are are like asking questions 24/7 that that they
• 46:16
  have to get there. They're like inference act
• 46:18
  together. Yep.
• 46:20
  Yeah, it's, it's fascinating to me to think about how that idea
• 46:26
  of everyone converging. And I do think that we'll see
• 46:30
  that. It's cool to see some of the,
• 46:34
  the different ones stay in their lane or just say like, hey,
• 46:38
  we're, we're just going to do this and that's all that
• 46:40
  matters. And then others are trying to go
• 46:43
  and be the whole platform. And I, I do appreciate that you
• 46:49
  have so much variety right now in the way that like the value
• 46:55
  prop of modal is you can get these GPU's going really fast.
• 47:00
  You've got that cold start or like gets going and you're good.
• 47:05
  One thing that is wild to me though, is that they're all
• 47:08
  running on top of other people's hardware.
• 47:13
  They're just renting out GPUs from the other people that are
• 47:18
  potentially competitors. Right.
• 47:22
  And they're, they're able to aggregate that demands, you
• 47:26
  know, 'cause yeah, I don't know when the last time you've tried
• 47:29
  to go in and, and like rents AB200 from, from AWS, but you,
• 47:35
  you can't do it. It's impossible.
• 47:37
  And core we've laughed at me. I like I literally during the
• 47:40
  development of the book, I said, hey, hey guys, I'm, you know,
• 47:43
  writing a, a book. I'm a third time author and, you
• 47:46
  know, would love to, to mention you guys in the book, which I
• 47:50
  still do. I'm not like a Dick or whatever,
• 47:53
  but they're like, we literally have no capacity and the minimum
• 47:58
  spend is blah, blah, blah. And you know, so like they
• 48:02
  didn't laugh at me. They were laughing with me, you
• 48:04
  know, cuz I was like, yeah, I, you know, came from Amazon.
• 48:07
  I get it. But I just, you know, yeah, I
• 48:09
  thought I would try. And you know, they, they're,
• 48:14
  they're booked out. Like, yeah, when I spoke to them
• 48:17
  over the summer, they were booked out for like 2 years and
• 48:20
  they just can't even get enough. And and then there's the issue
• 48:22
  of even if they could get enough, where are they going to
• 48:25
  get the power? And how do they, you know, all
• 48:27
  these turbines and the, I guess the turbine blades are the most
• 48:33
  precious. Like you can build a.
• 48:34
  Turbine, I heard that recently on the Stripe podcast.
• 48:39
  Oh, interesting. It it was exactly that, talking
• 48:42
  about how the blades are not being made fast enough.
• 48:46
  Yeah, that's interesting. They're coming from, I'm in
• 48:49
  Germany right now and I think I remember hearing that it was
• 48:52
  coming from Germany. And it's like, hey, somebody
• 48:54
  needs to put some, you know, light a fire in the German.
• 48:58
  Gas fuel on the fire. Yeah, what's going on?
• 49:04
  It is wild. And how much did you research or
• 49:09
  decide not to research on the alternatives?
• 49:12
  Cause like AWS has Tranium, right?
• 49:15
  And Inferencia and then you've got TP us and you've got AMD.
• 49:19
  There's lots out there, but like a very small subset of the users
• 49:26
  actually use that. Yep.
• 49:30
  And that's the answer to the question right there, right, Is,
• 49:34
  you know, this book was supposed to be 300 pages.
• 49:38
  And when I got as deep as I did, I said it will be a disservice
• 49:44
  to the folks that read this if I don't cover like all of CUDA,
• 49:51
  right? That, that makes sense for this
• 49:54
  domain. There's still so much more to
• 49:56
  CUDA and there's still, you know, so much more.
• 49:58
  And, and all the, the texture renderers and things that are
• 50:02
  actually, you know, still used as, as different types of caches
• 50:06
  and, and stuff that, that have wonky names that don't really
• 50:09
  translate to the LLM world. So, but there's so many patterns
• 50:14
  that have been built up that are.
• 50:16
  And if you take something like flash attention, you know, that
• 50:22
  was built by the chief research scientist over at together AI,
• 50:27
  right? He's a famous guy.
• 50:28
  Tree. Yeah.
• 50:29
  Tree John. Yeah, yeah, yeah, he's pretty
• 50:32
  bad ass. He he really brought light to
• 50:38
  this like mechanical sympathy type of thing.
• 50:40
  And the the big revelation was is let's use the fastest memory
• 50:45
  for the things that that need to move, you know, the, the like
• 50:49
  memory that's closest to the chip.
• 50:52
  Let's put things that are constantly moving in and out.
• 50:55
  Let's keep it there so that doesn't have to travel across
• 50:58
  all the other slower, you know, bandwidth like interconnects
• 51:02
  and, and just just that one small change made huge
• 51:06
  performance gains. And that's why with every
• 51:09
  generation Flash attention has to get like has to get rebuilt
• 51:14
  and reoptimized because it's not fully taking advantage.
• 51:18
  And, you know, there's a flash attention 4 or maybe it's up to
• 51:21
  five. I think 4 is the Blackwell
• 51:23
  version, basically. So the, the short answer to your
• 51:27
  question was, is there's so many patterns that fit the goal of my
• 51:34
  book, which was this Co design just within like the NVIDIA
• 51:39
  ecosystem and it and it really, really covered everything that I
• 51:44
  wanted to cover. So not only is it the most
• 51:47
  widely used chip, but it's also the most widely like supported.
• 51:53
  There's a lot of community support.
• 51:55
  There's people that have figured out the dot like, like you can
• 51:57
  infer the crappy documentation from all of the examples that
• 52:03
  are out there. And that's not the case with
• 52:05
  AMD, that that's not the case with Tranium.
• 52:07
  It's really hard to understand these chips without having
• 52:12
  someone from AMD or someone from Amazon right next to your your
• 52:16
  terminal. Yeah, helping you.
• 52:18
  You code it so. And it just would have exploded
• 52:21
  the book. I mean, then the book went from
• 52:22
  300 pages to 1000. And then.
• 52:24
  But during reviews, people kept asking, you know, why aren't you
• 52:27
  talking about Tranium? And yeah, my funny answer was
• 52:32
  yeah, yeah. And TP is actually, I probably
• 52:34
  would have talked about TP use even over Tranium just because
• 52:38
  it's a different type of, you know, architecture and systolic
• 52:41
  arrays and, and, but you know, we, we didn't have a lot of
• 52:48
  customers using Tranium back then.
• 52:51
  Yeah, obviously Anthropic was basically forced to use because
• 52:56
  of the partnership. And there will, yeah.
• 53:01
  Yeah, and well, no, I mean, they've been pretty successful.
• 53:04
  It, it just took a lot of work and price performance, you know,
• 53:09
  was good and they're they're, you know, trying to keep cost
• 53:12
  low and stuff and just having other alternatives.
• 53:16
  I mean, I saw this back at I like Netflix too with, with the,
• 53:20
  the cloud vendor. You know, Netflix famously chose
• 53:23
  Amazon, right, like AWS as the cloud vendor, but we always
• 53:29
  teased them to, to say that we were going to go to Azure or to
• 53:33
  GCP when at least back then there was no way in hell we were
• 53:38
  switching off. It was, it was going to take too
• 53:40
  much effort, you know, and like we had features to build right.
• 53:42
  Like these days, Netflix is in kind of a maintenance mode is
• 53:46
  what I've heard. I, I haven't worked there in 15
• 53:49
  years, but, you know, so, so now they could take on projects like
• 53:54
  that. But yeah, back then, you know,
• 53:56
  we were building streaming while we were, you know, scaling onto,
• 54:00
  yeah, onto AWS. So just just having that sort of
• 54:03
  threat of being able to switch, which is what Sam Altman's doing
• 54:06
  right now, right? He's got Cerebrus and he's got.
• 54:10
  How much have you tried to play with coding agents like at this
• 54:18
  level? Yeah, 100%.
• 54:20
  I'm actually tuning a a kernel right here, right behind your
• 54:24
  your your screen. It's very different than
• 54:29
  generating a Next JS app. Or my local app that I was
• 54:34
  telling you about in the beginning, right?
• 54:38
  Yeah, single use bespoke throwaway app.
• 54:41
  It's you know, and halfway through writing my book, the
• 54:46
  other reason why I insisted on it being as big as it is, is I
• 54:52
  selfishly wanted to use it with my coding assistant.
• 54:56
  So about July last year, I was asked to to cut a lot of this
• 55:01
  book because it just got massive.
• 55:04
  We were either going to split it into three books.
• 55:07
  It's really three books in one. It's like kind of a systems like
• 55:12
  like, you know, OS Kubernetes, like optimization for AI and
• 55:19
  storage and and then it's like CUDA and all the, you know,
• 55:24
  crazy things there to support the LMS and then it's inference
• 55:28
  and training and it's kind of systems wide.
• 55:32
  And they were. So we were going to do a
• 55:34
  three-part thing and then, you know, I just couldn't take it
• 55:37
  anymore. I just had to get these these
• 55:39
  words out and you know, I had to.
• 55:41
  Stay in. My old producer, when I was
• 55:45
  making music would tell me he was like, you don't finish an
• 55:49
  album, you abandon it. No, that's right.
• 55:53
  Yeah, Yeah. So that was that was what you
• 55:55
  hit. But sorry, I cut you off.
• 55:57
  You were saying that there was that moment.
• 56:00
  There was a moment when I was like, like, this book is going
• 56:06
  to be good for the readers, but this book is going to be even
• 56:11
  more valuable to the like, to the agents right, to the, you
• 56:18
  know, coding assistants. And it has saved me so many
• 56:21
  times because these models that, you know, lack in certain
• 56:28
  domains, right? There's just not enough training
• 56:31
  data, for example, on how to tune kernels and you know, how
• 56:35
  to optimize CUDA. It's in a lot of engineers heads
• 56:39
  that that work in, you know, Santa Clara at the NVIDIA
• 56:43
  office, But it's not, you know, because the codes not open
• 56:46
  source, right? There's a lot of code within
• 56:48
  their, their ecosystem that's hidden and it's all C++ and it's
• 56:53
  all, you know, shipped as binaries and assembly.
• 56:55
  And you know, yeah, good luck trying to reverse engineer some
• 56:58
  of that stuff, right? So my goal was to, if I was
• 57:04
  going to continue down this route and, and let you know
• 57:08
  others explore this route, I really wanted to make the book
• 57:12
  more of a flywheel. And that's why the GitHub repo
• 57:15
  is, is so critical. And I've built a bunch of MCP
• 57:19
  tools that are of course very friendly.
• 57:22
  I now have open claw, you know, yeah, I had to jump on, on open
• 57:25
  claw. So I've got like a GPU claw or
• 57:31
  yeah, like a yeah, I keep like renaming it right.
• 57:35
  Just like open claw. Yeah, I've like renamed it about
• 57:37
  3 or 4 times. But really it's a sort of AI
• 57:39
  systems optimizer skill and it can go out there and, and run
• 57:45
  all the cluster checks and, you know, try out the different
• 57:48
  things to make sure all the tensor cores are working as they
• 57:51
  should be and stuff, right? Because that's the key in, in
• 57:55
  CPU lands, we've got instruction pipelines, right?
• 57:59
  And some of them are instructions to load memory,
• 58:02
  instructions to do, but they're like fairly general and basic on
• 58:06
  a GPU. There's pipelines to do crazy,
• 58:09
  crazy shit. You know, there's, there's even
• 58:14
  special memory that's called tensor memory that works with
• 58:17
  the tensor cores. So tensor cores are like
• 58:21
  segmented off separate from the CUDA cores that sort of generic,
• 58:26
  you know, CUDA cores. And they're typically designed
• 58:30
  for specific precisions like floating .16 or FP4.
• 58:35
  You know, And so when you're, when you're writing a kernel,
• 58:40
  you, you have to know that there are separate pipelines for these
• 58:46
  things so that you can write them in, you know, parallel and
• 58:49
  say, OK, while, while you're, you know, computing the results
• 58:54
  in FP8, I want you to go do the multiplication or, you know,
• 58:59
  for, for the first instruct because you're like constantly
• 59:03
  overlapping, right? That way you can hide the
• 59:07
  relatively long time that it takes to move the data in from
• 59:10
  from memory in. And so now it's it's a trick of
• 59:14
  trying to take advantage of those parallel pipelines.
• 59:17
  And yeah, Blackwell did something sneaky by the way,
• 59:20
  when they move from Hopper to Blackwell that like not a lot of
• 59:23
  people know about where they got rid of of 1 pipeline and put it
• 59:29
  towards another, you know, yeah, basically said, OK, it seems
• 59:33
  like these days no one's really using, you know, int int date
• 59:37
  or, or something or FP64. So we're going to actually take
• 59:43
  transistors away, you know, resources from that, that
• 59:47
  particular use case and put them towards.
• 59:49
  And if you don't know that and you've got code that's that's
• 59:52
  relying on that you're going to run on on Blackwell and see a
• 59:55
  huge performance hit and be like, you know, yeah, what the F
• 59:58
  so. Yeah, I thought this was
• 1:00:00
  supposed to be better. Yeah, yeah.
• 1:00:03
  Oh, man. And so have you just created,
• 1:00:08
  like you said, skills for them that you've thrown on the GitHub
• 1:00:12
  repo or is it that you're using some kind of a rag on the book?
• 1:00:20
  Right. I of course, back when when rag
• 1:00:23
  was big, that was in my head. These days I just rely on
• 1:00:27
  cursors like grep and R grep. And so yeah, I've converted the
• 1:00:31
  book into markdown and it's by chapter and I've got all of the
• 1:00:35
  examples also by chapter. And so I can usually hint, you
• 1:00:42
  know, to cursor and like put, put those the, the chapter that
• 1:00:45
  I know it is, you know, relevant to this particular task that I'm
• 1:00:49
  working on just to kind of minimize it.
• 1:00:51
  So it's a little bit faster. But and this gets in this, you
• 1:00:56
  know, might be a good segue to the controversial codecs versus
• 1:01:00
  Claude. So yeah, curious what what your
• 1:01:04
  go to is these days? Is it Opus 46?
• 1:01:08
  So I use Opus 46. I have heard from friends I just
• 1:01:11
  haven't given codecs enough time.
• 1:01:14
  And I will also use Gemini and then anti gravity which
• 1:01:20
  sometimes completely bricks out my computer.
• 1:01:24
  So like it's so heavy. It's just like what is going on
• 1:01:28
  here? But that's a whole nother story.
• 1:01:31
  And I I haven't given codecs enough time.
• 1:01:34
  I have friends that have said they prefer it, it's the best
• 1:01:39
  because it doesn't take as many shortcuts, all of that stuff.
• 1:01:42
  But I'm like, dude, I'm already paying anthropics so.
• 1:01:46
  Much. Every month that I'm cool, just
• 1:01:50
  sticking with it, it's good enough.
• 1:01:52
  It's good enough. Yeah, yeah, Yeah.
• 1:01:54
  I was once like you. Yeah, for sure.
• 1:01:56
  You've you've changed. I, yes, I, I used Claude code
• 1:02:03
  like initially because of the hype and because I just, and,
• 1:02:07
  you know, trying like the codex is relatively new, you know,
• 1:02:11
  compared to Claude code. Maybe it came out a couple
• 1:02:14
  months later. So obviously the the, but what
• 1:02:17
  what I noticed, especially with larger code bases, you know,
• 1:02:20
  there's, there's almost 1000 samples in my, you know, GitHub
• 1:02:25
  repo for the book and I've noticed yeah, I think shortcuts
• 1:02:30
  is probably a good way to put it where it's not reading all the
• 1:02:34
  files. Claude, it, you know, like
• 1:02:38
  codecs is way slower. Yeah, obviously way slower,
• 1:02:41
  which is why I was excited about the cerebrous version, the, you
• 1:02:45
  know, codec Spark. I've thoughts on on that.
• 1:02:48
  I I still stick with the slower one.
• 1:02:52
  Yeah, but I I can step up and and walk away and go to the gym
• 1:03:01
  or go do something to the store, come back and I know Codex will
• 1:03:06
  have completed the task with Claude.
• 1:03:09
  I feel like it's designed for a different type of user.
• 1:03:13
  Who is sitting there waiting and wants to see and wants to be
• 1:03:18
  part of the of the solution? Yeah.
• 1:03:20
  Do you feel that it's a? Little more back and forth.
• 1:03:22
  I will notice that, yeah, it hardly ever gets it the first
• 1:03:27
  time. And so you have to go and you
• 1:03:29
  have to manually validate and then be like, oh, I'm getting
• 1:03:32
  this error as something this isn't working.
• 1:03:35
  Check that and I can see that. All right, well, shoot, like I
• 1:03:42
  needed another freaking subscription in my life.
• 1:03:44
  I guess I'm going to have to go now and do it.
• 1:03:47
  I mean, I think that's where the Ralph Wiggum stuff comes from,
• 1:03:51
  right? Like putting in a loop.
• 1:03:52
  Like, I just thought that that was so ridiculous.
• 1:03:54
  You know, I'm like, yes, I love Ralph.
• 1:03:57
  He's he's, yeah, obviously my favorite character on that show.
• 1:04:02
  But I don't need him in my, you know, development cycle.
• 1:04:06
  I just want the model to do it. Now the other thing too is that
• 1:04:09
  it it appears that the way that codex has been post trained, it
• 1:04:14
  is doing more frequent like compactions and it's doing it in
• 1:04:20
  in the model where it seems like anthropic post training has it
• 1:04:27
  kind of only when it gets full. Whereas, you know, codecs kind
• 1:04:32
  of does it when it determines that it's a good time that it
• 1:04:35
  can compact things and not not lose too much, you know, versus
• 1:04:39
  a huge, huge context. OK.
• 1:04:41
  Now I'm at 70% or 65% and I'm going to compact so.
• 1:04:45
  Yeah, actually, a lot of times the best performance gains you
• 1:04:51
  can get are just by clearing that context window.
• 1:04:54
  And then it's like, wow, the problem that I was having, it
• 1:04:58
  just miraculously disappeared. Yeah, it's counter intuitive
• 1:05:04
  because you want it to like remember all the mistakes that
• 1:05:07
  it made or what not. But yeah, if you give it.
• 1:05:10
  Out for the skills piece, you know what other one that I've
• 1:05:13
  been loving? It's called, I think it's called
• 1:05:16
  like Claude Ception. Have you heard of that one where
• 1:05:19
  it will it creates skills When it goes through and it does
• 1:05:24
  something, it will then reflect back and be like, could I have
• 1:05:27
  made a skill out of this? And then it will exactly what
• 1:05:30
  I've been looking for. Oh, that's crazy.
• 1:05:33
  OK 'cause I do that manually. Yeah, I figured when you were
• 1:05:37
  saying Oh well, yeah, I I will go and create some skills if if
• 1:05:41
  it's there and I see that it's needed.
• 1:05:43
  But yeah, try cloudception out, see if it works.
• 1:05:46
  And there's there's other ones that have been very useful to my
• 1:05:51
  friend. Raul was telling me he just
• 1:05:55
  creates one skill and he spends a lot of time teaching that one
• 1:06:01
  skill how to then create all the other skills.
• 1:06:04
  So if he ever says OK go make a skill out of this, it knows what
• 1:06:08
  to do. Cool.
• 1:06:11
  OK, Yeah, Yeah, that's a cool pattern.
• 1:06:14
  Yes. I love these discussions because
• 1:06:16
  everyone has their own workflow and and.
• 1:06:19
  That's what we started doing because there's just basically
• 1:06:22
  there's 100,202 million. I don't know how many people now
• 1:06:27
  are their own little R&D department and we've all been
• 1:06:32
  empowered with hey, just go try stuff.
• 1:06:35
  And so we, we get together on Fridays now and have little
• 1:06:39
  lunch and learns and be like, what are you doing with it?
• 1:06:41
  Well, oh really? Oh, that's interesting.
• 1:06:43
  And we've had some folks come with ideas of, well, in human
• 1:06:48
  psychology, we can't really understand 7 layers deep.
• 1:06:53
  Oh, I love it. We.
• 1:06:54
  Can't go past 7 layers so anytime we get I create a file
• 1:06:58
  structure. I never want to go past 7 layers
• 1:07:01
  for the LLM and it's like dude I have no idea if that works but
• 1:07:06
  just the the fact that you're trying it is really cool to me.
• 1:07:11
  Yeah, when do you run these? This is a formal.
• 1:07:13
  Fridays, yeah, yeah, I'll, I'll invite you to them on Fridays,
• 1:07:17
  please. Lunch time in California, so
• 1:07:21
  around 11:00 AM, yeah. Nice.
• 1:07:24
  Yeah, man, that's super fun. Yeah.
• 1:07:27
  In fact, do you know Swix? Yeah, of course.
• 1:07:31
  Last I I was chatting with him at the the like Claw conference.
• 1:07:38
  Yeah. Do you remember the the claw con
• 1:07:39
  that happened? The claw con, Yeah.
• 1:07:41
  Dude, that was a shit show. It was an absolute shit show,
• 1:07:44
  yeah. Oh man.
• 1:07:47
  In like a good way, you know, like a San Francisco's back,
• 1:07:50
  like, kind of right. People have like, lobster.
• 1:07:53
  They were serving like crab legs and just chaos.
• 1:08:00
  Yeah, but I was talking with Swix and we were chatting a
• 1:08:05
  little bit about making that like a first class track in his
• 1:08:11
  like upcoming AI engineer summits because and just just
• 1:08:15
  having people, yeah, even if it's informal, maybe like like a
• 1:08:19
  lunch where everyone comes, gets together and shares.
• 1:08:21
  But I think that's super powerful.
• 1:08:22
  Yeah. Yeah.
• 1:08:23
  When did you start that? Just recently because we did a
• 1:08:27
  reading group in January, our first reading group of the year.
• 1:08:30
  It was on Vibe Coding and then we read the paper, which was
• 1:08:34
  cool for the 1st 15 minutes and then it just devolved into
• 1:08:38
  everybody sharing. They've been getting a little
• 1:08:41
  bit more gain. And that has, that was like,
• 1:08:45
  wow, we'll, we should do this again, Let's make it a thing.
• 1:08:48
  And so we started doing it on Fridays and it, it's been so
• 1:08:52
  useful to me and especially because maybe I'll know about
• 1:08:56
  this, but then somebody else knows about that.
• 1:08:59
  And I'll say, Oh yeah. But did you try Claude Ception?
• 1:09:01
  And they're like, well, no, actually, I like, you know, this
• 1:09:04
  one better. And and it's the community Slack
• 1:09:07
  is great for that too. But when you are actually
• 1:09:11
  prodding and probing a little bit more than you get different
• 1:09:16
  ideas out of folks. Totally.
• 1:09:19
  Yeah, great idea. I'm that's going to be, I'm
• 1:09:22
  going to clear my my calendar for that one.
• 1:09:24
  There you go. Well, yeah, and, and actually
• 1:09:26
  it's now that you say that we're doing the coding agent
• 1:09:29
  conference on March 3rd, right. And one idea that I had for that
• 1:09:33
  was exactly what Swix is thinking on.
• 1:09:36
  We need some kind of unconference Y way to let people
• 1:09:41
  just give their best tips. And I landed upon, we're going
• 1:09:47
  to do a session where we'll just have a Slido up and people go to
• 1:09:52
  the Slido and then they put in Slido what their best trick is.
• 1:09:57
  And then whoever up like what gets up voted the most, that
• 1:10:02
  person gets to come up on stage and then demo.
• 1:10:05
  Demo. And showcase it.
• 1:10:07
  So that's one of the things that we're doing.
• 1:10:09
  Yeah. And then I'm also going to do
• 1:10:11
  that kind of thing with hot takes, like what's your hottest
• 1:10:14
  take? And then whichever ones get up
• 1:10:16
  voted the most, they come on stage and have to defend their
• 1:10:19
  hot take. You know, I think the the
• 1:10:23
  biggest hot take from the Peter Steinberger, the open claw guy,
• 1:10:28
  right the. Whole journey.
• 1:10:30
  The journey, yeah, he was it it was very clear from his Lex
• 1:10:36
  Friedman interview from, you know, he was at the the claw
• 1:10:40
  con, of course yeah. I mean, he was talking trash
• 1:10:44
  about everything. Yeah, he said something and, you
• 1:10:46
  know, this is all, all public. You can go watch the video.
• 1:10:48
  But he was joking that he, he was reached out by so many
• 1:10:52
  folks. He got an e-mail from NVIDIA,
• 1:10:57
  like not, you know, Jensen directly, but some whatever,
• 1:11:02
  like Lieutenant of, of Jensen's. And they sent him a Microsoft
• 1:11:06
  Teams invite and he just rejected it because of that and
• 1:11:11
  said I'm not going to work with you guys because you use Teams.
• 1:11:14
  No way. Yeah.
• 1:11:16
  So I mean, he's a, you know, he's a really interesting
• 1:11:18
  character in that. Like he just doesn't give
• 1:11:20
  enough, you know, He's. Got enough money to not care?
• 1:11:25
  Right. So, yeah, which I didn't really
• 1:11:28
  like. I knew nothing about this guy
• 1:11:30
  until me either. Open claw.
• 1:11:31
  Yeah. Yeah, but yeah, I think a lot of
• 1:11:34
  folks had no idea who he was or what he was doing.
• 1:11:37
  I just love the fact that he shipped like 40 products before
• 1:11:40
  Open Claw took. Off totally.
• 1:11:42
  Oh, right. That's that's awesome to see.
• 1:11:45
  I also just found out that he is absolutely jacked and he might
• 1:11:50
  be on steroids. Yeah.
• 1:11:54
  I mean, yeah, yeah, he's a big dude, I guess in person.
• 1:11:59
  I mean, he's, he's like Austrian, so.
• 1:12:01
  He's got those. So is it steroids or is it
• 1:12:03
  Austrian? I don't know younger.
• 1:12:06
  Yeah, he's got the Schwarzenegger.
• 1:12:10
  That's hilarious and. Yeah, did you listen to the
• 1:12:14
  Friedman podcast? Is it worth?
• 1:12:17
  Is it worth it? It's like 9 hours or 4 hours or
• 1:12:20
  something. But Friedman's good now about
• 1:12:23
  doing clips and so you could see.
• 1:12:25
  But yeah, Friedman kept pushing him on.
• 1:12:28
  Which direction are you going? Oh yeah.
• 1:12:30
  And then this way. I was going to say I was
• 1:12:32
  surprised. He's he's pretty anti anthropic.
• 1:12:36
  Everything from I mean, besides the little mini cease and desist
• 1:12:40
  on the name. Yeah, he was not impressed It it
• 1:12:44
  didn't make the top 2. The top two that he told Lex
• 1:12:49
  about were either Meta, because Zuckerberg actually pinged him
• 1:12:53
  on on WhatsApp directly and was asking him questions about the
• 1:12:57
  API and how it was designed and said he was actually building
• 1:13:01
  something and he wanted to talk to him.
• 1:13:02
  And so any he liked Sam Altman, he he made a comment and said,
• 1:13:08
  you know, a lot of people vilify those, you know those guys, but
• 1:13:12
  you know, honestly, they're, they're hackers like us, right.
• 1:13:15
  So just just in, you know, different leadership types of
• 1:13:19
  positions. But yeah, anthropic and
• 1:13:21
  basically he's a codex user. So, and he, he was describing
• 1:13:27
  very similar things to what what you and I are mentioning, where
• 1:13:31
  it's like there's, there's too much back and forth.
• 1:13:33
  There's there's too many decisions that I need to make
• 1:13:36
  that I would expect the model to make.
• 1:13:39
  And I just want to fire off 15 of these things and know that
• 1:13:42
  it's going to get it done. Yeah.
• 1:13:45
  Oh man, no, I got to go try it. No, I got to go have some fun
• 1:13:49
  with code. 200 bucks. Yeah, I'm going to go and do
• 1:13:53
  that right now. Shoot.
• 1:13:55
  Yeah. What's your biggest like monthly
• 1:13:57
  bill for Claude or for any of these?
• 1:14:00
  Probably. So I am relatively, again, I'm
• 1:14:04
  just building for myself, right? So I'm relatively calm, but I
• 1:14:08
  probably spent like 300, three, 140, which can be a lot or it
• 1:14:15
  can be nothing depending on who I'm talking to.
• 1:14:17
  That's nothing. Yeah, Yeah.
• 1:14:20
  That's just me getting my, you know.
• 1:14:22
  Warming up those clusters. Totally.
• 1:14:27
  No, They go by quickly. I think from what I hear at at
• 1:14:31
  meetups, I think 1500 to 2000 per month seems to be pretty
• 1:14:36
  standard. Yeah, some people hit at 25
• 1:14:39
  three, but yeah, I can burn through like 2000 without even
• 1:14:42
  thinking about it. And then I have to, you know,
• 1:14:45
  set up alerts and be like, oh crap, you have to make sure I
• 1:14:47
  can pay my, my rent so. Oh, that's classic.
• 1:14:54
  And and so talk to me how you're using these for the kernel
• 1:14:58
  exactly. I mean you're doing the skills
• 1:15:00
  like you mentioned, but what other stuff are you doing
• 1:15:03
  spending all this money? One of the the issues if I get a
• 1:15:08
  cluster with a new chip. So let's take the AMD or the
• 1:15:14
  newest versions of, you know, Tranium, which I didn't have any
• 1:15:17
  exposure to before I left Amazon.
• 1:15:19
  So they're, you know, pretty new to me.
• 1:15:23
  You have to like really dig deep.
• 1:15:26
  You, you, you have to look at like take a really fast library
• 1:15:32
  like in, you know, the NVIDIA world, that would be Kublas,
• 1:15:35
  right, which is like highly tuned by the Kuda ninjas.
• 1:15:38
  And it's like this, this badge of honor that you can get it to.
• 1:15:42
  And so doing something like that for these, these other chips and
• 1:15:49
  looking at the assembly and these models are not good with
• 1:15:54
  assembly, but they're good at reading text.
• 1:15:58
  I mean, if it's like genomic text or strings and you know,
• 1:16:02
  and can build patterns. And so I've, I've developed some
• 1:16:06
  ways to like reverse engineer and like this is all brand new
• 1:16:13
  stuff that I've been working on since I finished a book and, you
• 1:16:16
  know, finally had time to explore.
• 1:16:19
  But really that's outside of having one of these CUDA
• 1:16:24
  engineers next to you or these, you know, hardware folks next to
• 1:16:27
  you. That's really the only other way
• 1:16:28
  to do it is to take these non public, you know, frameworks,
• 1:16:36
  have them compile the code and then just just try to decompile
• 1:16:39
  and and yeah, just basically like reverse engineer and then
• 1:16:42
  try to apply it to other use cases.
• 1:16:45
  Wow. Yeah.
• 1:16:48
  And that that then leads to different types of like
• 1:16:52
  innovations with flash attention and you know, the DeepSeek
• 1:16:55
  moment where they found something that was in the
• 1:16:57
  assembly and then they had to reverse.
• 1:17:00
  And, and fortunately, you know, the instructions kind of give
• 1:17:06
  you a hint as to what they are like if it's something dot L2
• 1:17:11
  like it, it's doing something or bypassing or something with the
• 1:17:14
  L2 cache. And so you can play with that
• 1:17:17
  and you can pull it out and you can put it in.
• 1:17:19
  You could double it. You could, you know, do it
• 1:17:21
  twice. You can do it and see what's
• 1:17:23
  actually going on. So it's right now it's a game of
• 1:17:26
  just let's just try to figure these things out because even,
• 1:17:30
  you know, the NVIDIA folks aren't always optimizing for
• 1:17:34
  your for your use case. Yeah, I mean, fortunately they
• 1:17:39
  have a lot of resources now on on right, like LMS and and
• 1:17:44
  things. But even even like like one or
• 1:17:47
  two years ago, yeah, it wasn't this case.
• 1:17:49
  But there's not enough training data.
• 1:17:50
  And think of I think when these labs go to do their next round
• 1:17:58
  of, of RL or their their next round of, of pre training.
• 1:18:02
  Let's start with pre training, you know, trying to get in these
• 1:18:05
  long reasoning chains which I'm generating right now.
• 1:18:08
  Like it, it's trying different things.
• 1:18:10
  It's like exploring. So actually this like leads to
• 1:18:16
  kind of a more general pain point that I see when I'm I'm
• 1:18:20
  using these models is yes, I could read snippets of the
• 1:18:23
  reasoning chains, the thinking. But you can't see all of it.
• 1:18:27
  Yeah, and I feel like heck, we're we're paying for it.
• 1:18:31
  I mean, that's 90% of my, you know, like $2000 bill is the
• 1:18:36
  stuff that I can't see, right. And so I actually want to get
• 1:18:39
  away to to, you know, pull that in.
• 1:18:42
  So I think you can. I think with with the API I
• 1:18:46
  think you can get more. Yeah, You know, I, I also had
• 1:18:54
  this guy Rohit on here probably 7-8 months ago now and he said
• 1:19:01
  something beautiful that I still can't do.
• 1:19:05
  But you can kind of hack it together, which is he's like,
• 1:19:09
  why can't we have different situations where if I want to go
• 1:19:18
  through and figure something out during that reasoning process, I
• 1:19:22
  can nudge it one way or the other.
• 1:19:24
  And so now I think you can kind of do that with just
• 1:19:29
  interrupting it and then saying go here, go there.
• 1:19:32
  But the problem is, if you don't see the reasoning happening as
• 1:19:38
  much as you would like, then you can't steer it.
• 1:19:41
  Yeah, yeah, or it's, it's mid steer already.
• 1:19:47
  And like you're, yeah, 'cause that actually happened to me
• 1:19:50
  last night. I was trying to steer it and,
• 1:19:53
  and, or by the time I see it, it's already too late, too late,
• 1:19:58
  right. And so, yeah, yeah, a little
• 1:20:01
  confusion around that kind of stuff.
• 1:20:03
  But you know, I mean, I'm thinking of this in the fact
• 1:20:06
  that I'm, I'm burning tokens. To get to an answer that I want
• 1:20:11
  to make sure I get the answer a lot faster next time, right.
• 1:20:15
  And it's like, how do I do that? I can do the the thing, you know
• 1:20:18
  where it's going to create a skill like you were saying, kind
• 1:20:20
  of dynamically or I can have it update the agent's MD or the
• 1:20:24
  Claude MD to remember something about that.
• 1:20:27
  But I still think there's a case when, oh, and so the point that
• 1:20:33
  I was going to make was I also met this other company, actually
• 1:20:39
  several companies, but one company in particular out of New
• 1:20:44
  York that is building their own sort of reasoning chains for
• 1:20:50
  CUDA kernel optimizations. And that got the attention, of
• 1:20:54
  course, of like of of the big laps, right?
• 1:20:59
  They like very much, you know, have an interest in making their
• 1:21:03
  models better at doing this because it's going to make their
• 1:21:05
  jobs, you know, much easier. Also too, because open AI, for
• 1:21:10
  example, hasn't quite made the shift to Blackwell or, or, you
• 1:21:14
  know, last I heard they were still and like, you can tell
• 1:21:17
  because they created this language called Triton.
• 1:21:21
  So Triton is like a little bit. Yeah.
• 1:21:24
  So I cover Triton in the book, I think chapters 12 and 13 or
• 1:21:27
  something. Triton is like abstracts you
• 1:21:31
  from having to write CUDA and it's in Python And it's it's own
• 1:21:35
  DSL, you know, Python based DSL and it gives you access to CUDA
• 1:21:39
  like constructs, but that's, you know, kind of a higher level of
• 1:21:42
  abstraction. And those libraries are not
• 1:21:45
  optimized for Blackwell. And it's and it's because like
• 1:21:50
  Open AI hasn't gotten there yet. So they're not, you know,
• 1:21:53
  spending any resources so you can kind of back into where
• 1:21:57
  they're at in their hardware stack by how much they're
• 1:21:59
  supporting the software that they have have open sourced.
• 1:22:03
  I mean, they might have an internal fork that they're not
• 1:22:05
  sharing publicly, but yeah, yeah, so, but and, and oh so
• 1:22:11
  these these companies that are tuning and tuning all day and
• 1:22:15
  have huge, huge clusters and trying out different experiments
• 1:22:19
  to to get the best speed under the under sort of LLM, you know,
• 1:22:24
  like token generation use cases. They're, they're, they're
• 1:22:30
  sharing this data with the labs, which when, when I talk to them,
• 1:22:36
  I say, don't do that. Like, why are you doing that?
• 1:22:38
  Right? Because that's, well, that's.
• 1:22:42
  Sell the data, don't share her. Yeah, and you know, quite
• 1:22:46
  honestly, maybe they are selling it and they're not telling me or
• 1:22:50
  something. But it, it sound, it sounded to
• 1:22:53
  me like they're trying to get in with the labs, so they're just
• 1:22:57
  going to give it to them for free.
• 1:22:59
  And I'm like, dude, like, that's your, that's, that's your, your
• 1:23:04
  Moat, you know? Yeah, exactly.
• 1:23:05
  That's basically you're shooting yourself in the foot there.
• 1:23:08
  Yeah, yeah, potentially. So not unless there's some other
• 1:23:11
  arrangements. Yeah, they're also doing it with
• 1:23:13
  like AMD, they're also doing it with these sort of newer chip
• 1:23:18
  companies, Yeah. Have you thought about open
• 1:23:22
  sourcing your different conversations or reasoning pads
• 1:23:27
  for stuff that you've been doing?
• 1:23:29
  Because it could potentially be training data there too, right?
• 1:23:33
  Yeah, but like that's what I'm saying is I have to get this
• 1:23:37
  data and I don't know how to get it from Open AI, right?
• 1:23:42
  Yeah, if if I was using the API maybe, but I use my
• 1:23:47
  subscription. You know, there's like two
• 1:23:50
  different ways to authenticate. You could use the API key or you
• 1:23:53
  could use but but yeah, that's exactly what I'm saying is like
• 1:23:57
  I think, I don't know if like anyone else has like thought of
• 1:24:00
  this, but we, we should have access to these tokens.
• 1:24:04
  Especially if you're we're paying such high valuable data.
• 1:24:09
  Yeah, right. Well, open AI very likely has
• 1:24:14
  access to these. So.
• 1:24:17
  So now it's. I wonder what the chances are of
• 1:24:21
  them actually right knowing that this is being created and this
• 1:24:27
  is valuable. It reminds me of back in the
• 1:24:30
  day. I heard a story that if you
• 1:24:32
  googled something about some really fancy algorithm, then you
• 1:24:36
  would get a call and it would be like, hey, come work for us
• 1:24:40
  because there was only. Like yeah, from.
• 1:24:44
  Google it was something around, I think it was a search
• 1:24:47
  algorithm or something. Where you were.
• 1:24:49
  Like oh, if you know about this, you've gotten this far, then
• 1:24:54
  you're legit and you should come and work for.
• 1:24:57
  Us hilarious. So hilarious.
• 1:24:59
  It's it's like that. Are they?
• 1:25:02
  Do they know that the data that the data chain that you have
• 1:25:07
  going on, is that valuable, I wonder?
• 1:25:10
  I mean, they certainly can. They can find it.
• 1:25:15
  Yeah, yeah, of course. We're telling them right now,
• 1:25:18
  we're making it very clear if anybody at the labs is
• 1:25:24
  listening, go and look at if anybody at opening eyes
• 1:25:28
  typically, yeah, it's. Like 938 in the morning.
• 1:25:31
  Yeah, you're tuning kernels. That's some sweet stuff.