Latest / Agentic Conversations (formally mlops.community) / Boosting LLM/RAG Workflows & Scheduling w/ Composable Memory and Checkpointing // Bernie Wu // #270

Transcript

Boosting LLM/RAG Workflows & Scheduling w/ Composable Memory and Checkpointing // Bernie Wu // #270 · Agentic Conversations (formally mlops.community)

• 0:00
  My name is Bertie Wu. I'm the VP of Strategic
• 0:03
  Partnerships and Business Development for Memberge, and
• 0:06
  I'm a fairly simple guy. Take my coffee black, but I take
• 0:09
  a lot of it. Welcome back to the ML OPS
• 0:12
  Community podcast. I am your host to Dimitrios and
• 0:18
  you know, normally I like to take my vitamins and I generally
• 0:24
  take quite a few vitamins. I do, you know an array of
• 0:28
  vitamins, vitamin D, vitamin B, vitamin C, vitamin A, all that
• 0:32
  good stuff. I also have been known to take
• 0:36
  some mushrooms, not the hallucinogenic kind, the kind
• 0:39
  that help your brain function. And Oh my, I wish I would have
• 0:45
  remembered to take them today because I did not.
• 0:48
  And this conversation I was hanging on by a thread.
• 0:52
  Bernie brought some serious learnings to me.
• 0:56
  He helped me see things in a different way, specifically
• 1:00
  around memory. Memory allocation, elasticity of
• 1:05
  memory. And it just so happens, funny
• 1:09
  enough, ironically, that I should have taken these pills to
• 1:15
  help with my own memory. Not the hardware memory, my own.
• 1:21
  If anything Bernie says piques your interest, feel free to
• 1:23
  reach out to him. He is very cool and very open to
• 1:28
  chatting with new people. Here we go.
• 1:37
  One thing that most of the time we'll snark at when people say
• 1:42
  this is the word or this expression of oh, you just have
• 1:47
  to think about it by first principles.
• 1:50
  You have to apply first principles thinking to it and I
• 1:54
  know it is great, but it's almost been overused since it
• 1:57
  came into fashion and you are must start using it.
• 2:00
  And now when you hear people saying it, I don't really trust
• 2:04
  that they actually a even know what that means when B actually
• 2:10
  do it. But in the pre conversation that
• 2:12
  we were just having, the only thing that I could think about
• 2:16
  is your vision right now that I want to get into with not not
• 2:23
  looking at GPU shortage in the market being the problem,
• 2:27
  looking at memory shortage in the market being the problem,
• 2:30
  that is what I would consider taking things to 1st principles.
• 2:37
  Yeah, I know. I I agree.
• 2:39
  This is funny you mentioned first principles because long
• 2:43
  time ago I went to UC Berkeley and the emphasis you'd at the
• 2:46
  undergraduate levels get to get rid of down to 1st principles.
• 2:48
  You know, if you don't understand the first principles,
• 2:51
  you're not going to graduate from this place.
• 2:53
  But I, I do think that in this, this industry, I mean, I'll
• 2:58
  start off with this little factoid.
• 3:01
  I mean it the, you know, the bulk of the industry is going to
• 3:04
  be delivery on production. I mean, that's where, that's
• 3:09
  where you get an ROI on it. All this AI investment that's
• 3:11
  going on in the industry. And, and the transformer model
• 3:16
  is, you know, kind of taking over the world with a
• 3:17
  transformer model and foundation models and things like that.
• 3:21
  They, they, I believe the numbers around .1 flop per, per
• 3:28
  byte. So, or in other words, there's
• 3:30
  10 bytes of, of, of memory movement or loading into
• 3:34
  registers for every floating point operation on average.
• 3:37
  And so it's a very data intensive area.
• 3:40
  And on the flip side, you know, in the news there is this, the,
• 3:45
  all the stories about GPU shortages and things like that.
• 3:49
  But if you actually look at how people size their purchases of
• 3:52
  GPU is based on, oh, OK, here my model is this, but big.
• 3:56
  And therefore I need this much memory and OK, each, each GPU
• 4:01
  has this much HPM for, for, for GPU.
• 4:04
  Therefore I buy X number of GPU. That's how it was done.
• 4:07
  And you find that out that a lot of times the, the GPU are then
• 4:14
  purchase that way and often they're underutilised because
• 4:19
  they're memory bound. And so some of these models that
• 4:22
  were trained, I know at least at scale, they, they, their
• 4:26
  utilisation was down in the 30% level.
• 4:30
  So if you could actually get more utilisation, obviously you
• 4:32
  wouldn't need 20 GPU's, but they are, they are memory bout.
• 4:35
  So we are working to solve that. I mean, it's, it's, it's a, it's
• 4:40
  a complex problem takes, it's going to take some new memory
• 4:44
  technologies, new software, which is what my company's
• 4:47
  working on. And, and it's going to percolate
• 4:51
  all the way up through the stack.
• 4:52
  I start seeing signs of that already there new kinds of
• 4:55
  schedulers are going to have to be developed and things like
• 4:57
  that to handle a more composable memory architecture.
• 5:01
  Yeah. Well yeah, talk to me about how
• 5:04
  you see memory. Right now it almost feels like
• 5:07
  we are memory scarce. How would a memory abundant
• 5:16
  world to look? Yeah, on the memory abundant
• 5:20
  world, well, I I would say that you you would have, you know,
• 5:25
  very common in in things like Kubernetes and AI environments
• 5:31
  that you have these things where you run out of memory and then
• 5:33
  you start spilling to disc. And when you start spilling to
• 5:36
  disc also your performance grinds to, you know, it runs
• 5:41
  4500 percent slower, that kind of thing.
• 5:43
  It starts swapping and things like that, or in some cases,
• 5:46
  because you don't even have the option to swap the disc, these
• 5:49
  things just get killed the other their their own killed or
• 5:53
  whatever. But you know, so, so as an
• 5:56
  example, in that kind of use case, we're working on a
• 5:59
  situation where we have this elastic memory, we have this
• 6:02
  surge capacity of memory in a separate pool.
• 6:04
  And then what we detect that something is running out of
• 6:07
  memory, particular compute instance, we just give it a shot
• 6:11
  of memory and until the problem goes away, then we reclaim it at
• 6:15
  a site of somewhere else. So I, I think part of the
• 6:18
  overall solution, kind of like what people are doing with GPU's
• 6:22
  is the is to pull things together and, and you can get
• 6:25
  higher efficiency. And then also you can address
• 6:28
  fluctuations in demand as workloads, you know, go through
• 6:32
  different compute instances and, and add either more GPU's or add
• 6:36
  more memory. You know, there's a more in
• 6:39
  other words, the the infrastructure needs to get a
• 6:41
  more elastic and and composable, you know.
• 6:45
  Well, it's, it's interesting that you talk about how a lot of
• 6:50
  times you're seeing 30% utilisation rate, right?
• 6:54
  But then there are those spikes that will kill the job.
• 6:58
  And so you're trying to help for that spillover when you get that
• 7:03
  spike so that you don't lose everything, right?
• 7:06
  Right. Yeah.
• 7:07
  So there's that's another interesting area.
• 7:09
  There's a lot of actually because the AIML areas is a
• 7:14
  relatively new industry that and and because you need to preserve
• 7:18
  these models, they they take a long time to run.
• 7:20
  There's been a lot of work done at the framework level, the
• 7:24
  software, you know, the, the pie Torx level, etcetera, to, to
• 7:28
  take checkpoints of things and save those states so that
• 7:32
  between training FX, you know, something crashes, you can at
• 7:36
  least restore or if you overshoot the model training,
• 7:38
  you can rollback, etcetera. But and, and those are, those
• 7:42
  are interesting technologies, but they, they, I don't think
• 7:44
  they can address all the different use cases for, for
• 7:49
  example, when, when a, when a system just overheats or memory
• 7:53
  failures occur. That and especially a scale.
• 7:55
  I think I was at a conference and they said that some of these
• 7:59
  larger models I get up to, they're planning to build models
• 8:02
  with a million, you know, GP us. And if that's kind of scale, you
• 8:07
  know, you're going to have 30-40 fifty GPU failures per hour.
• 8:11
  So, so a lot more resilience has to be built into the
• 8:15
  architectures. And if you know anything about
• 8:17
  memory, as soon as the power or something goes off, you lose it.
• 8:20
  So, so we're also working on what we call checkpointing
• 8:24
  technology at the at the memory level, so the ability to save
• 8:28
  machine state and memory state at that level as well as the
• 8:32
  framework level. And I think between the
• 8:33
  combination of those two will get much higher resilience that
• 8:37
  be able to run all these long running training jobs or long
• 8:39
  running inferencing jobs. Now they're actually going to
• 8:41
  become inferencing services. So they need to be much more
• 8:45
  resilient than they are right now.
• 8:47
  What do you mean by inferencing? Services, well, I think yeah,
• 8:50
  things go into production. We, we, we go from what
• 8:53
  fundamentally from a scheduling operation standpoint is more of
• 8:56
  a batch operation, batch to train this batch to train that
• 9:00
  to a service where we're constantly, you know, dealing
• 9:04
  with maybe hundreds of thousands of consumers.
• 9:06
  And this thing has to run like any other web service, you know,
• 9:10
  so it, it's resilience, it's ability to be migrated to new
• 9:15
  infrastructure as old infrastructure needs to be
• 9:17
  upgraded. All those kinds of tools that,
• 9:21
  you know, I'm sure in the lofts community is very, very familiar
• 9:24
  with they're, they're going to be needed.
• 9:25
  And so we're, we're, we're interested in collaborating with
• 9:28
  that, with your community to help bring those tools, the
• 9:31
  right tools with to handle this new onslaught of demand for,
• 9:36
  for, for memory, no. Well, it's funny you you
• 9:39
  mentioned how it's fairly common for GPUs to go offline.
• 9:46
  I think the best way I've heard it was when we had our AI
• 9:51
  quality conference, Todd Underwood, who was head of the
• 9:55
  research platform at Open AI, he said you can just look at a GPU
• 10:01
  the wrong way and it'll go offline.
• 10:03
  And so he was just obviously taking it to the extreme and
• 10:08
  making jokes about how sensitive GPUs are and how finicky they
• 10:12
  are and how difficult it is for people to actually get that.
• 10:17
  And then in that same talk, he talked about how the other part
• 10:21
  that is complicated to really get right is knowing when to
• 10:26
  checkpoint because he's like, yeah, maybe you say, OK, I'm
• 10:30
  going to checkpoint every second.
• 10:32
  But what happens if it doesn't go offline or it on average GPUs
• 10:37
  go offline every 100 seconds? Then you should be OK to do it
• 10:41
  every 50 seconds or every 25 seconds, right?
• 10:45
  But what happens if it goes offline every 3 seconds?
• 10:48
  Then you really have to be figuring those things out and
• 10:51
  then doing the calculations and recognising it.
• 10:54
  And another person that we had come on here from Nebius Cloud
• 10:59
  was talking about how these checkpoints are just so.
• 11:02
  Big when he's. Training these models so you
• 11:06
  can't be that sparing with how often you're checkpointing
• 11:11
  because it adds up quickly. You, I think he's if I remember
• 11:14
  correctly, it was like 3 terabyte checkpoints every time.
• 11:18
  Yep. Yep, some of the training models
• 11:19
  are getting up to that size and writing that all to a file
• 11:23
  system, even a high performance file system is is is becoming a
• 11:27
  bottleneck and then also restoring it.
• 11:30
  So, so 11 area where some of this memory technology that I'm
• 11:35
  working on is relevant is to create a memory pool that can
• 11:40
  add a connect as a cache to basically dump all that stuff
• 11:44
  very quickly into memory, which is much faster.
• 11:46
  Obviously it's about memory speeds are about you know, you
• 11:50
  know, the latency and all that kind of stuff, but it could be
• 11:54
  up to 10X difference by dumping it into memory and then
• 11:57
  asynchronously bleeding it off into the file system.
• 12:00
  That way you, you minimise the interruption period of the
• 12:03
  checkpoint because that that's that is a problem.
• 12:06
  The checkpoint overhead is. It's not surprising in a lot of
• 12:09
  training models to be 2530% of the total training.
• 12:12
  A life cycle is spent on just doing the checkpoints.
• 12:16
  Dealing, yeah, checkpoint management.
• 12:18
  So basically you're saying, hey, what if we just had a Redis type
• 12:21
  of thing where you dump it into memory, it's cached, and then
• 12:26
  slowly it gets offloaded from that memory.
• 12:29
  So you don't have to be sitting there taking the time to offload
• 12:33
  the checkpoint and then come back online.
• 12:36
  Some of that's been applied in other industries, like the high
• 12:39
  performance computing industry. They've had these giant clusters
• 12:42
  for for decades and they've, you know, they're, they're running a
• 12:45
  model that runs for a whole month, you know, simulation of a
• 12:48
  nuclear explosion or whatever they're doing.
• 12:50
  And yeah, they have to they, they, they use this kind of of
• 12:53
  checkpointing approach. So I think some of those
• 12:56
  techniques will have to come into play here.
• 12:59
  And then, you know, there's other, there's other needs for
• 13:02
  memory caches. So for example, the on the
• 13:07
  inferencing side, what's happening is that the, the in
• 13:10
  order to scale it, the models are getting so big, like the
• 13:12
  latest Llama four O 5 be a 405 billion parameter model now
• 13:18
  doesn't fit in most GPU nodes and now needs to be distributed.
• 13:22
  And so now people are not only figuring out how to, they've
• 13:25
  already figured out kind of how to distribute training with 3D
• 13:28
  parallelism and all this kind of stuff.
• 13:30
  And now they're trying to figure out how to distribute
• 13:33
  inferencing. And one of the first things that
• 13:35
  people realise that, hey, this is a 2 stage process.
• 13:39
  There's a, a pre this kind of a, a pre prompt phase, which is CPU
• 13:46
  intensive. And then there's this decode
• 13:48
  phase, which is really purely, you know, memory, memory bound.
• 13:53
  And so they're starting to split the architecture, but the
• 13:56
  prompts are also getting so long that the, the, the KV weights
• 14:01
  and these training models is that goes up quadratically with
• 14:04
  a link to the prompt. And so they, most those things
• 14:07
  don't fit in memory. So now they have to start
• 14:08
  distributing the memory. So we have a distributed GPU and
• 14:13
  a distributed memory problem. And, and anytime you deal with
• 14:16
  memory, the first thing that happens is it gets fragmented,
• 14:19
  you know, And so, so how do you manage all that stuff?
• 14:22
  So, so I don't think there'll be any shortage of work for, for
• 14:26
  companies like ours and the, the rest of the memory industry and
• 14:28
  trying to solve some of these problems.
• 14:30
  And they're, they're really cool problems because, you know, like
• 14:33
  I said, this revolution, I feel like compared it to the big data
• 14:36
  revolution, there's a lot of things that the infrastructure
• 14:40
  area can do it to innovate and improve this overall stack, you
• 14:45
  know? Yeah, it's really attacking that
• 14:48
  hardware side of things, which, yeah, the big data revolution, I
• 14:53
  think you mentioned before how it it was a revolution, but not
• 14:59
  as hard on the whole stack. Same processors, just actually
• 15:03
  just using lower grade servers, you know you didn't need to have
• 15:06
  your high end server. The whole trick was scale out,
• 15:08
  not up here. I think I would contrast that
• 15:12
  because these workloads are going to get more right now most
• 15:16
  of this memory problem is what I would characterise as capacity
• 15:20
  bound or, and then secondarily bandwidth bound.
• 15:24
  But I think latency bound, which is the other, the other concept
• 15:28
  in when you talk about what do you mean by memory bound, It's
• 15:31
  either capacity latency or, or, or, or bandwidth or ball, all of
• 15:35
  them. I think the latency bound issues
• 15:37
  are going to become more and more critical because the
• 15:41
  pipelines are getting more and more complicated.
• 15:43
  And so it's the end to end time is, is becoming a factor.
• 15:47
  You know, if, if, if you're sitting at your computer, you
• 15:49
  type something in and you get an answer back at 200 millisecond,
• 15:51
  life is good right now these computers are going to go off
• 15:55
  and start reasoning and, and do lots of, or do some sort of, you
• 15:59
  know, multi agentic kind of rag LLM of pipeline.
• 16:03
  And so there's a big difference between waiting 10 seconds for
• 16:06
  the answer and maybe waiting for, you know, 10 minutes,
• 16:08
  right? And so getting faster devices,
• 16:12
  which is memory basically, that's where memory really tops,
• 16:16
  you know, using conventional, you know, flash storage, using
• 16:20
  more memory more aggressively to help compress those pipelines,
• 16:24
  increase the the utilisation and throughput, it's going to be
• 16:27
  key, yeah. Well, talk to me about this
• 16:30
  distributed memory problem and when you go into that type of a
• 16:36
  problem, can you breakdown how you think about solving it?
• 16:40
  I think, well, I think it's going to be sort of a, you know,
• 16:44
  at the lowest level, things like models like the VLL model was
• 16:49
  first to pioneer, but they, they built a virtual memory operating
• 16:52
  system inside the basically inside the HBM memory space.
• 16:57
  So they could page and, and, and have uniform size blocks and,
• 17:00
  and manage memory that way. Because every prompt that comes
• 17:04
  in creates a different size KV set of weights, right?
• 17:07
  So you're constantly fluctuating in.
• 17:09
  So you had to have some order to that allocation of that memory.
• 17:13
  Now, now these things don't even fit.
• 17:16
  So, so now you're, you have this same a problem of allocating
• 17:20
  memory and but now it spans now multiple nodes, multiple GPU's
• 17:25
  and even multiple nodes. So it's going to have, it's
• 17:28
  going to take some kind of of scheme, kind of a scheduling
• 17:31
  scheme, which is something else that we're we're starting to
• 17:35
  work on at our company is how do we schedule these resources?
• 17:40
  And it's not just scheduling GPU resources, but I think also
• 17:44
  we're going to need to figure out how to schedule memory
• 17:46
  resources and and then leverage a composable memory, which is a
• 17:50
  or a virtualized memory pool. That makes it easier to, you
• 17:53
  know, move memory around between compute nodes as well as within
• 17:58
  a, within a GPU itself. And then we have to figure out
• 18:01
  how to migrate things, right. So a lot of times you classic
• 18:04
  memory, at some point you have to start, you know, garbage
• 18:07
  collecting, defragmenting things or compacting, been packing
• 18:10
  things, Same, same exercise. So a lot of it's interesting,
• 18:13
  the industry kind of goes of a spiral.
• 18:16
  All these concepts have existed for decades, but now we got to
• 18:19
  figure out how to apply them to this new set of circumstances.
• 18:22
  Yeah, but some of those basic principles are still going to
• 18:24
  need to be applied in this area. I'm not sure I fully understood
• 18:30
  when you said the composable memory there and the virtual
• 18:35
  memory sharing it around. Yeah, I think I get it.
• 18:39
  But yeah, not fully. Yeah, yeah, yeah.
• 18:42
  No, I. Actually, I kind of jumped
• 18:44
  ahead. I didn't explain that concept.
• 18:46
  So up until now DRAM has been, I call it change to the processor.
• 18:51
  It's either chained to the GPU as HPM memory or it's chained to
• 18:56
  a a CPU and it's the system every on the DDR bus.
• 19:00
  And so what's going on in the in this memory revolution is it's,
• 19:04
  it's not as quite as visible as the accelerator revolution, but
• 19:08
  there's definitely a memory revolution underway.
• 19:11
  There's a big effort to put memory on its own fabric.
• 19:15
  So one of the first steps toward that was the CXL consortium,
• 19:18
  which was formed in 2019. And that was the realisation
• 19:23
  that we can't use these classic parallel bus architectures and
• 19:27
  steel memory much more. We're out of gas.
• 19:29
  On a matter of fact, if you look at the at a high level, the
• 19:32
  ratio of memory to compute cores has been deteriorating for years
• 19:36
  and years and now we've got to totally reverse that.
• 19:39
  So the only way they could do that was to actually start using
• 19:41
  more of a serial bus architecture.
• 19:43
  And the first one that come along is PC IE.
• 19:46
  So the CXL consortium now allows you to take memory modules and
• 19:52
  and plug them into a PC IE bus and extend the memory over the
• 19:56
  PC IE bus. And this is standard DDRDD RAM
• 20:00
  memory. So it's high, it's, it's
• 20:01
  reasonably high speed memory and allow you to expand within a a
• 20:06
  compute note and then across compute nodes that you may know
• 20:11
  already that a lot of people have built PCI switches.
• 20:14
  So now there's PCI, now there's what they call CXL enabled PCI
• 20:18
  switches coming out. CXL, by the way, stands for
• 20:21
  Express. Express computer Express link is
• 20:26
  a way to it's just a the acronym for this whole idea of creating
• 20:31
  additional pools of memory that are, that are external to the
• 20:34
  system buses, but allow you to pool and share memory.
• 20:40
  So sharing memory is really kind of the ultimate, because I think
• 20:45
  there there's use cases in AI where people want to save like
• 20:49
  the entire chat history and, and, and then be able to put
• 20:52
  that back into a prompt quickly or some pieces of it and, and
• 20:56
  have a, you know, more accurate answers or relevant answers and
• 20:59
  things like that. Well, you can imagine at some
• 21:02
  point in the not too distant future, we're going to have the
• 21:06
  ability to put an entire group, let's say they're all interest.
• 21:09
  This whole this group is all interested in travel to to
• 21:12
  Hawaii where we can put all their chat history in one pool
• 21:15
  and and share the responses and questions among an entire pool
• 21:20
  across most of our nodes. Yeah, So collaborate.
• 21:23
  Yeah, exactly. Collaboration is now going to be
• 21:26
  enabled. There's already people are
• 21:27
  trying to we can collaborate to a certain degree, but I think
• 21:30
  we'll be able to take it to a new level and to a new depth of
• 21:34
  of of of history. So so that's also exciting this
• 21:38
  ability to use a memory pool. Memory pool becomes basically a
• 21:42
  first class citizen over time in the data centre and and because
• 21:46
  repository of all this this chat history.
• 21:49
  If I'm understanding that correctly, is it it's like
• 21:51
  bringing Google Docs to your chat history and now I can
• 21:55
  comment on some of your chat history or I can just jump in
• 21:59
  and say, hey, now explain more this whatever you said from your
• 22:05
  answer, but now I get to take the wheel and drive.
• 22:09
  Yeah, it's kind of like, you know, it's like car
• 22:11
  applications, right, is a crowdsourcing kind of thing.
• 22:14
  You know, you look at the traffic, you have all these data
• 22:16
  points reporting on the traffic everywhere real time.
• 22:19
  So here you can Crowdsource the responses in one sense and get I
• 22:23
  think higher quality responses or more closer to real time
• 22:26
  responses based on changing conditions or whatever to what's
• 22:30
  going on. And so we we started prototyping
• 22:34
  this. So if, if I, I put a link in
• 22:38
  our, in our to a blog that we have on how we took Llama index
• 22:43
  and, and this very simple Rag LLM pipeline and started
• 22:46
  creating that shared repository. But we're still, we're still
• 22:51
  working out. That's more of a toy demo to be
• 22:53
  honest you at this point. But it does show that we can
• 22:56
  make some of those pipelines what I would call better,
• 22:59
  cheaper and faster. You know, better in the sense
• 23:02
  that we're going to get a higher quality response from
• 23:04
  aggregating all this chat history.
• 23:06
  You know, cheaper because we actually have because of the
• 23:08
  additional bandwidth of adding the CXL memory to, to these
• 23:12
  compute nodes, we get more bandwidth, higher queries per
• 23:16
  second, which you know, per, per per node.
• 23:19
  So that cut drives down the cost per per query and then faster
• 23:23
  because the we're recalling everything from memory.
• 23:27
  So when you look at some of these workloads with which
• 23:31
  require a rapid look up of of all these different memory
• 23:34
  objects, you know, it's hard to beat, you know, a random direct
• 23:38
  seek up off of a memory device as opposed to going into a
• 23:42
  storage device and loading a whole block at a time and trying
• 23:44
  to parse through that, you know? So really it comes back to what
• 23:48
  you were saying earlier on this elasticity and being able to
• 23:53
  have the elasticity of memory along with the elasticity of
• 23:58
  everything else, memory in both ways on the training side of
• 24:03
  things, but also on this inference side of things.
• 24:06
  The whole pipeline, even the pre training, the ingest area, you
• 24:10
  have the emergence now just got back from the race summit.
• 24:14
  The others emergence of these multimodal heterogeneous
• 24:18
  workflows, right multimodal meeting OK is coming in as
• 24:22
  voice. This guy is speaking Korean and
• 24:24
  now we got to turn it into the vectors and embed it.
• 24:27
  And then we got to run it through a translator and have it
• 24:29
  within, you know, within mill, you know, milliseconds come back
• 24:32
  and speak in English, you know, like a universal translator.
• 24:35
  So yeah, so that kind of modality switch, you know, makes
• 24:40
  the models a lot harder, especially when you get to
• 24:43
  video. The, the video images are much
• 24:45
  bigger than, you know, just, you know, prompt text prompts.
• 24:49
  And so I think for those you'll, you'll, we'll need to increase
• 24:53
  the token generation rate by at least 5X, you know, like the
• 24:57
  equivalent of 5X just to handle those kinds of things, which
• 25:00
  means more. Different though, when you're
• 25:02
  looking at the training side of this elasticity of memory versus
• 25:07
  the inference side and how you set it up and how you think
• 25:12
  about it. Yeah, though there's definitely
• 25:14
  differences. But you know, I think the other
• 25:17
  interesting thing about our industry is they, they were
• 25:21
  originally, they were very distinct, very distinct.
• 25:23
  As a matter of fact, you could just train on GP us and you
• 25:26
  could even use CP US to do the inferencing.
• 25:28
  And you only needed one or two of those to, to do it, you know,
• 25:31
  but now I think there's, there's convergence.
• 25:33
  I mean, there's, there's, there's inferencing going out of
• 25:36
  the training phase. That's how the some of these
• 25:38
  reasoning models got developed. And then there's, there's,
• 25:41
  there's training going out of the inferencing phase.
• 25:43
  The other way around, you know, this, this fine tuning and this
• 25:46
  need to have a Lora adapter per person, right?
• 25:49
  So you'll have your own Lora adapter.
• 25:51
  I'll have my own Lora adapter. And as soon as this general
• 25:53
  purpose model finds out, oh, it's you, it'll drop in your
• 25:56
  Lora adapter. No, no, as a customised
• 25:58
  responses for your for your, you know, your, your preferences or
• 26:02
  whatever, right. So I, I think, yeah, it's going
• 26:05
  to get, I think it's going to get moshed together more than
• 26:08
  more distinct. Yeah.
• 26:10
  And so we need a really elastic, you know, flexible and scalable
• 26:15
  memory architecture. And I think the this memory
• 26:17
  architecture needs to go up and out.
• 26:20
  So it needs to be a scale up and also a scale out, just scale up
• 26:24
  just because the going back to first principles in physics, I
• 26:28
  mean a nanometer, I mean, a nanosecond is a nanosecond,
• 26:31
  right? Yeah.
• 26:33
  So, so if you want a really compressed latency of of these
• 26:36
  more complex workflows, there's going to have to be some level
• 26:39
  of scaling up. On the other hand, if you want
• 26:42
  to handle the entire, you know, planets AI needs, you're going
• 26:46
  to need a scale out. So it's going to be interesting.
• 26:49
  And there there are not only new memory technologies that are
• 26:52
  disaggregated like CXL coming out, but they're also new
• 26:56
  fabrics that are coming out that also will support this scale up
• 26:59
  and scale out. So your audience should read up
• 27:04
  on. There's an emerging standard
• 27:05
  called the Ultra Ethernet Consortium, which will address
• 27:08
  some of the scale out issues. And then there's another
• 27:11
  consortium called the Unified Accelerator Link which will
• 27:17
  address memory scale up. So and yeah, so those are
• 27:21
  emerging standards along with the CXL standard.
• 27:24
  I just mentioned that I think they're all going to over the
• 27:26
  next couple years make significant differences in this
• 27:30
  memory bandwidth, capacity, latency problem.
• 27:33
  Well, I remembered in the I did a breakdown on a few of the
• 27:38
  Facebook engineering blogs or Meta engineering blogs and they
• 27:42
  were talking about their GPU's and how they train llama and all
• 27:46
  that fun stuff. And one thing that they
• 27:49
  mentioned was I think it was 25,000 H one hundreds or A1
• 27:56
  hundreds. I can't remember exactly which
• 27:58
  24 they split it up. So they had I'm pretty sure they
• 28:02
  had 48,000 in total, but they said we don't know what's going
• 28:06
  to be better. We want to test which one is
• 28:09
  better. And one of the tests that they
• 28:11
  did was with that Ultra Ethernet to see if it it was faster
• 28:16
  versus the no, I'll have to check again what the other
• 28:21
  option was that they had. Yeah, well, the people are using
• 28:23
  InfiniBand as one. Choice InfiniBand.
• 28:26
  That was it. The classic InfiniBand.
• 28:28
  Yeah, yeah. And then the other standards,
• 28:30
  well, part of it is there are no standards.
• 28:31
  So what I, I believe was Meta has done so far as sort of a
• 28:34
  proprietary way of doing what they call packet springs across
• 28:38
  Ethernet, which is one of the things that Alter Ethernet
• 28:41
  Consortium wants to standardise on.
• 28:43
  Yeah. So to so, so a lot of these
• 28:46
  networks I think we're and I'm kind of drifting off from memory
• 28:48
  into networks, but I'll do go a little further.
• 28:52
  A lot of those networks I think we're designed primarily for
• 28:54
  more like north-south traffic than east, West traffic that you
• 28:57
  see in these, in these clusters that that need to exchange at
• 29:01
  high speeds all the weights and values and all that kind of
• 29:04
  stuff. So that's really the ultimate
• 29:05
  problem to try to solve, you know, and it has to be reliable,
• 29:08
  you know, I can't, can't drop packets and retransmit this.
• 29:11
  It takes too long. So one thing that I, I did want
• 29:13
  to touch on was how this all effects operations, right?
• 29:17
  And the scheduling workflows and the pipelines around that, the
• 29:21
  compute instances, the composability of the GPUs and
• 29:25
  schedulings that now need to take into consideration the GPU
• 29:29
  status. You had mentioned how it's not
• 29:32
  just scheduling for the GPU status, but then the memory
• 29:37
  scheduling and all of that. And so you're really dissecting
• 29:41
  the scheduling work flows in ways that it feels like we're
• 29:45
  not necessarily thinking about it today, but it probably would
• 29:48
  help us to think about. It there's there's multiple
• 29:51
  dimensions to the scheduling problem, you know, 11 quite
• 29:55
  honestly is you know, it's more of a organisational thing,
• 29:58
  right? How how, how a company is
• 30:00
  organised or an enterprise Oregon, a cloud services
• 30:02
  organised and you know, so it's a, it's basically an allocation
• 30:06
  of a scarce resource. At the end of the day, it's a
• 30:09
  economic problem, right? So there's that dimension.
• 30:12
  OK, who, who has priority? How do we assign priorities to
• 30:16
  jobs? Some jobs are batch jobs.
• 30:18
  I mean, it kind of at a trivial level, you can say, well, this
• 30:20
  is a batch job, It can be bumped off.
• 30:23
  This one's an interactive web session.
• 30:25
  We can't, it has to be a certain kind of SLA.
• 30:27
  So you're going to have in in a, in a different infrastructure
• 30:30
  and they probably most likely, I would imagine get partition this
• 30:33
  area maybe just for more batchy kind of things in this area is
• 30:36
  more services kinds of things are ongoing streaming things.
• 30:41
  So that's one level of cut at this problem.
• 30:42
  And the other level of cut is that OK?
• 30:44
  How do I, you know, how do I balance among multiple different
• 30:52
  criteria, what the how, how resources would be allocated?
• 30:56
  So for example, do I care more about utilisation of memory and
• 30:59
  utilisation of GPU? Is that the primary dimension of
• 31:03
  scheduling or is the primary dimension of scheduling the SLA?
• 31:07
  You know, what is the and and throughput of a single session?
• 31:10
  What is the overall total throughput and what's the long
• 31:12
  tail latency of any individual? You know, maybe the maybe the
• 31:15
  most important criteria is actually making sure that the
• 31:18
  P99 of of a of a session is no more than X.
• 31:22
  Otherwise, I'm going to have a, you know, customer, you know,
• 31:24
  churn and drop out, you know, so all those things have to be
• 31:27
  figured out and then and then mapped to some sort of what I
• 31:32
  think will need to be a highly elastic infrastructure.
• 31:34
  You know, a lot of people are building their infrastructure on
• 31:37
  Kubernetes, which is great, but it doesn't go nearly far enough
• 31:41
  in in trying to address these these new problems.
• 31:43
  I mean, it was designed to do a horizontal scaling and vertical
• 31:48
  auto scaling. It has a lot of deep
• 31:51
  infrastructure, but now that the accelerator, you know, was
• 31:56
  designed for these CPU architecture, now the
• 31:59
  accelerator is becoming more important and, and the memory is
• 32:03
  becoming more important. So it's funny that the
• 32:05
  components of the no, not necessarily the actual note
• 32:07
  itself is becoming the, the main, the the the the, the these
• 32:11
  individual components is becoming more important to
• 32:13
  schedule more directly is what's what's happening now.
• 32:17
  And do you see a open source project that is built on top of
• 32:23
  Kubernetes coming through and and helping that?
• 32:26
  Or is it something that just Kubernetes itself the way it was
• 32:30
  thought through and the primitives there are not going
• 32:33
  to be able to do the trick? I think, no, I actually there,
• 32:38
  there's gosh, there must be at least a half a dozen scheduling
• 32:42
  projects going on, you know, that are Kubernetes related and
• 32:46
  you know, some of them are also, you know, trying to address
• 32:49
  various aspects. And it, it right now is still
• 32:53
  very much a fluid thing as it's not, it's not clear because,
• 32:58
  because the, the reason things are not clear is because things
• 33:02
  are changing so rapidly, even at the, the top level of the stack.
• 33:06
  Just the basically the way we put together and deliver models,
• 33:10
  you know, you know, I would say last year was the, you know, the
• 33:15
  year of, you know, model proving right now which model is the
• 33:18
  right model? Is it going to be an open source
• 33:20
  model? Is it going to be, you know, are
• 33:21
  we going to use AP is in the cloud?
• 33:23
  And, and those are the only good models.
• 33:25
  And I think that pretty much the people now have kind of decided,
• 33:29
  OK, most of these models, open source models are good enough,
• 33:32
  at least bit enough for large language models.
• 33:34
  It may not be good enough for other kinds of multimodal models
• 33:36
  yet, but definitely that, that areas is out of the way.
• 33:39
  And so this year, I think it was more about rag, right?
• 33:42
  How do we start delivering these, putting these things into
• 33:45
  products and services, these AI things?
• 33:47
  And, and toward the tail end of this year, it's been more about
• 33:50
  multi agentic rag and, and, and, and now now the trading area is
• 33:54
  going through another iteration. Now people want to start having
• 33:57
  these things reason more, you know, so, so there's there's so
• 34:02
  much flux is what I'm trying to say.
• 34:03
  There's so much flux at that level that whatever schedule you
• 34:08
  build is going to have to be very composable itself.
• 34:13
  And yeah, also probably able to teach itself over time.
• 34:17
  Yeah. Yeah.
• 34:18
  Have some sort of AI OPS element to it because it's a it's, it's,
• 34:23
  it's complicated, you know? So that's fascinating thing
• 34:26
  about the flexibility needs to almost be the number one design
• 34:31
  principle. Yeah.
• 34:32
  I think, I think architecturally it needs to be flexible, you
• 34:36
  know, almost to an extreme. And, and then like I said,
• 34:40
  there's these new trends right now, a lot of schedules don't
• 34:43
  even consider memory, right? It's just, yeah, you know, it's
• 34:46
  just like they didn't consider what AGP us when they originally
• 34:49
  built the original schedulers for, for for Kubernetes.
• 34:52
  Now, now we got to start spending more time at more, more
• 34:56
  of the action is there even though they're even though
• 34:58
  they're still tethered to ACPU. You talk to ACPU as a, as a
• 35:02
  Kubernetes node, but you're really, you're really driving
• 35:04
  the, the GP us that, that, that, that node contains and, and, and
• 35:08
  that, and those GPUs have their own network too, by the way,
• 35:11
  right? So they have their own envy mesh
• 35:13
  and things like that. So, yeah, it's, it's just, this
• 35:17
  is really a time, a lot of experimentation, I would say.
• 35:20
  And, and, but it is still evolving very quickly.
• 35:24
  And they're probably going to be multiple right answers is, is
• 35:29
  how I see it. I mean, that's how I see the,
• 35:32
  the like I said, I think there's at least a half a dozen
• 35:34
  scheduling projects. Some of there's Q there's
• 35:37
  Unicorn and I just heard another one from Alibaba, I think is
• 35:40
  working on one. You know there's so there's
• 35:41
  there's several and these are open source scheduler scheduling
• 35:46
  project. Yeah.
• 35:48
  I guess my biggest question is how much of that is a square peg
• 35:53
  trying to fit into a round hole? Yeah, I think he, you know, I
• 36:00
  think some of these projects have been driven by, you know,
• 36:03
  particular, you know, hyper scalers and there are or or
• 36:08
  large scale deployments and they're kind of unique set of
• 36:11
  circumstances. So that's why there's so many of
• 36:13
  these things. It's, it's going to be driven
• 36:16
  by, like I said, at least by organisational considerations
• 36:21
  and then economic considerations that also driven by, you know,
• 36:25
  the, the, their, their technology stack that they
• 36:27
  chose. And for that, for that
• 36:29
  generation of deployment. Yeah.
• 36:31
  So, yeah, it's not it's, it's, it's challenging.
• 36:36
  But I think, you know, my expectation is that some of this
• 36:40
  dust will start settling in the next couple years and we'll see
• 36:44
  more distinct patterns emerge. And, and I, I my hope it is that
• 36:50
  by that time you'll see memory get more respect in the industry
• 36:55
  or awareness and, and, and use as part of solving some of
• 36:59
  these, you know, AI problems, which again, people have this
• 37:04
  high level perception of their GPU bomb.
• 37:05
  But I think the really when I listen to a lot of these talks,
• 37:09
  at the end of the day, they're really memory bomb problems.
• 37:12
  Yeah, they they, they. Yeah.
• 37:14
  And that's why I. Feel like it is you coming back
• 37:17
  to first principles because you're saying, yeah, the GPU is
• 37:20
  there, you are bound by the GPU, but what do you want from that
• 37:23
  GPU? Where's the real bottleneck?
• 37:25
  If it's at the memory layer, then we might want to start
• 37:28
  there. No, exactly.
• 37:29
  I think the, I like to say that the in the, in the Western
• 37:33
  cultures, people say, you know, when you look at a problem,
• 37:37
  here's point A and here's point B and you draw a line between
• 37:40
  A&B and that's the direction you go to solve it, you know, or the
• 37:43
  direction, and it may be an Asian cultural is, is more of a
• 37:47
  spiral. They everything is a Ying.
• 37:49
  Ying is a circle. But I think reality in this
• 37:53
  industry is a, is a, is a, is a combination of both.
• 37:55
  It's, it's both a vector and, and also a spiral.
• 37:58
  So you have this spiralling around.
• 38:00
  So you have this rotation. So right now we're in a rotation
• 38:04
  where, you know, it's, you know, the perception at least has been
• 38:08
  where, where we have a huge GPU shortage or whatever.
• 38:11
  But I think and at some points it's been all the industry's
• 38:14
  been network bound and that and or storage bound.
• 38:18
  But I think memory bound is, is the next thing coming or or is
• 38:23
  it's already here as people just don't acknowledge it.
• 38:25
  And they're trying to, like I said, do as much Ledger domain
• 38:27
  as they can with, I call it Ledger domain with algorithms to
• 38:31
  to spread out the the memory problem of distributed as much
• 38:35
  as possible or or quantize it, right.
• 38:38
  You know, it's OK, we're not going to use such accurate, you
• 38:40
  know, FP16 down in four or whatever we're trying to do to
• 38:43
  shrink the size of things. But nonetheless, a lot of the
• 38:46
  fundamental equations say, hey, this you want, you want these
• 38:50
  bigger prompts and models are more sophisticated things.
• 38:52
  You know, your, your, your, your K VS are going to your weights
• 38:55
  are going to go up quadratically.
• 38:58
  So if it's quadratic with four bids, it's still quadratic
• 39:01
  versus quadratic with 60 bids. You know, it's just, it's just
• 39:04
  mushrooming. So I, I think there's a, like I
• 39:07
  said, I think it'll be memory's turn to be in the, in the docket
• 39:10
  and then in the limelight in the next, you know, 1218 months to
• 39:14
  solve some of these. Problems coming to the consumer
• 39:19
  hardware that we've got and this amplification of memory on my
• 39:23
  laptop or on my cell phone. Well, you know, it's right now.
• 39:28
  Right now it is kind of a cram down situation.
• 39:30
  So yeah, you have extra considerations in a, in a, in a
• 39:34
  like a handheld mobile device, you know, how the, the power
• 39:37
  budget and things like that. So actually I remember listening
• 39:40
  to a talk the other day from Qualcomm, It was amazing how
• 39:43
  much they're, they're able to start putting some of these, you
• 39:45
  know, visualisation AI kinds of tools.
• 39:48
  They're starting to put those even on, on a, on a phone, but
• 39:50
  they're going to need to carve out eight gigs, 10, you know, 16
• 39:53
  gigs, 32 gigs. And so there's some limit to
• 39:56
  that, how much you can quantize things.
• 39:58
  But so I think. To that point, I think yeah,
• 40:01
  they're they're they're squeezing out, you know, every
• 40:05
  last drop of blood we can out of algorithms and quantization.
• 40:09
  At some point you're going to just we're we're out of
• 40:12
  algorithmic tricks. We we've got to do something.
• 40:14
  And I think I think on the edge you can get away with that.
• 40:17
  You got a smaller screen and all that kind of stuff.
• 40:19
  You don't have to worry about the resolution and, and insert
• 40:22
  other factors like that. But yeah, I think for, for real
• 40:26
  for a lot of other use cases, we just got a, you know, refactor
• 40:32
  the refactor the memory stack as well as the GPU stack and
• 40:36
  everything else, you know. You're telling me there's not
• 40:39
  enough fancy math in the world to squeeze a little bit more out
• 40:44
  of that algorithm? Yeah, well, I mean that's.
• 40:46
  Got to go somewhere. Else, I mean, that's what
• 40:47
  happened in general. AI mean AI has been around for
• 40:50
  for years, decades. The problem was they didn't have
• 40:53
  enough compute power. And so finally the economics
• 40:55
  and, and, and the ability to connect these things in some
• 40:58
  reasonable network like InfiniBand became, made it
• 41:01
  viable to run these kinds of, of basically grinding problems.
• 41:06
  And, and so, yeah, so that was a limiter.
• 41:09
  And I think, like I said, to take AI to the next level and
• 41:14
  which I think is going to be very exciting, this, this
• 41:16
  multimodal real time stuff, the ability to, to implement
• 41:22
  robotics or whatever that's going to require, you know, more
• 41:28
  than just the algorithms to solve it.
• 41:29
  I mean, we, we're going to need algorithms.
• 41:31
  I don't want to belittle what those people are doing.
• 41:32
  They're doing some incredible work, But no.
• 41:37
  And then, then, but everything falls back.
• 41:39
  I think the rubber hits the road kind of in the, in the
• 41:41
  operations are we got to, it's nice to do all those R&D, but
• 41:44
  we've got to put it in production.
• 41:45
  We got to make it economical and we got to make it, you know,
• 41:49
  secure and, and, and, you know, meet SL as all that kind of
• 41:54
  stuff. So that's, so I'm, I'm kind of
• 41:56
  interested in that applying, applying that all this, this
• 41:59
  innovation and making this stuff really tangible for everybody to
• 42:03
  use. And Speaking of making it more
• 42:06
  economical, does the excess of memory, do you feel like there
• 42:12
  will be new strategies that will help make it more economical
• 42:16
  like we were talking about with the checkpoints and how you can
• 42:19
  kind of slowly roll the checkpoints from one stage to
• 42:22
  another via the memory path or because I when I think of
• 42:29
  gigantic pools of memory, I don't think cheap.
• 42:34
  Right. No either.
• 42:34
  Expensive. Necessarily.
• 42:36
  Bingo go in my head. Yeah, yeah.
• 42:38
  No, actually years ago when members got started, we, we, we
• 42:42
  the, the inspiration back then that was that was going to be
• 42:45
  the availability of cheap memory.
• 42:47
  So several years ago, Intel introduced something called
• 42:51
  Octane or persistent memory and it's cost per bit it is, is read
• 42:56
  access time was the same as DRAM.
• 42:58
  If I could plug it in the same DDR slots as a regular DRAM, but
• 43:02
  it's cost per bit was like 1/4, one third, 1/4 of of DRAM.
• 43:07
  So you could actually build a much larger memory pool and, and
• 43:13
  drive down the cost per bit. But so that's, that's one aspect
• 43:17
  the unfortunately they discontinued that was a
• 43:21
  proprietary architecture. They discontinued that.
• 43:24
  But it within the memory hardware industry, there are new
• 43:27
  kinds of memory technologies emerging also they're they're
• 43:32
  also kind of converged memory architectures emerging.
• 43:35
  So SSD is much cheaper than on a cost per bit than than DRAM.
• 43:42
  So people are trying to build these kind of hybridised memory
• 43:45
  slash SSD. Matter of fact, they're, they're
• 43:47
  referred to as memory semantic SSDs because the problem with
• 43:51
  the SSD is that, yeah, the, it, the cost per bit is slower, but
• 43:56
  the, the and the, and the, the access time, you know, look up
• 44:03
  time is a lot longer, you know, so maybe instead of 100
• 44:07
  nanoseconds, now you're at 10 microseconds, you know, it's
• 44:09
  like couple orders of magnitude higher.
• 44:11
  So, so part of the solutions is to try to use a combination of
• 44:16
  those in a in a very clever caching way.
• 44:19
  And, and some of these out some algorithms, not all of them are,
• 44:22
  are more predictable from from application standpoint.
• 44:26
  So you have a better chance of using that kind of caching
• 44:29
  algorithm there. Others are going to be totally
• 44:31
  random just because you know, if you're, if you're serving
• 44:35
  inferencing out there and who knows who's going to go up to
• 44:38
  the chat and ask them off the wall question next, You know,
• 44:41
  you never know, you have no idea of what what kind of access you
• 44:44
  might need. So, so I think and then there
• 44:47
  are fundamentally new memory technology.
• 44:50
  So I, I know some of the largest firms, VC firms are, are, are
• 44:55
  looking at funding some of these new generation of every
• 44:57
  technologies that will kind of bridge that substantial gap.
• 45:01
  There's like a couple orders of magnitude difference in
• 45:04
  performance, but also in cost per bit between DRAM and and
• 45:07
  SSD. So, so the yeah, I think that'll
• 45:11
  that'll also have to happen as well over time.
• 45:14
  No. The other piece that I wanted to
• 45:17
  dive into was around what you had said before with these
• 45:22
  basically pools of memory that you can use in an elastic way
• 45:29
  and so that you can avoid what I am sure everyone that is
• 45:33
  listening has had happened to them at one point or another in
• 45:36
  their life with those Ohm errors.
• 45:40
  Yeah, those are those are really annoying that that was the most
• 45:42
  annoying thing I learned about Kubernetes is like good, good
• 45:46
  thing is, is a very elastic infrastructure for micro
• 45:49
  services and all that bad news is if you have if you're a semi
• 45:56
  stateful, you have a lot of remember you're dragging around
• 45:59
  and you get killed. It's not it's not a pleasant,
• 46:01
  it's not a pleasant situation. Yeah.
• 46:03
  And so they're exactly, so I think, you know, it's
• 46:07
  interesting how Kubernetes has evolved because it was supposed
• 46:09
  to be a totally, you know, stateless micro services.
• 46:12
  In other words, all the all these database and storage
• 46:15
  problems, OK, not my backyard, put them outside the Kubernetes
• 46:19
  and connect to them. But I think that's not so easy
• 46:25
  to do, in my opinion, with some of these AIML workflows.
• 46:29
  And so, so over the years, obviously the Kubernetes
• 46:31
  committee has done a great job of accommodating stateful
• 46:33
  applications and, and replica sets and all this other kind of
• 46:36
  stuff, right? That's you see it in a more
• 46:38
  general purpose enterprise environments.
• 46:40
  And yeah, so I think I think this elastic pool, yeah, yeah,
• 46:46
  we're hoping to show this kind of demonstration probably in the
• 46:50
  not too distant future where, yeah, where as soon as you we
• 46:54
  detect memory pressure like this, things are starting to
• 46:58
  either because I think you can adjust.
• 47:00
  There is a project going on with a Kubernetes where you can spill
• 47:03
  the disc or do disc swapping, which is a heretical to original
• 47:08
  Kubernetes, you know, philosophy, but they're allowing
• 47:11
  it now. But so we can either detect
• 47:14
  those kinds of swaps or detect when just just before a a node
• 47:19
  gets shot, you know, preemptively shot for out of
• 47:22
  memory in the in the in the OR sorry, where a pod gets killed
• 47:25
  because it's about to run out of memory of the node.
• 47:28
  We can intervene in those and and inject memory or we can
• 47:32
  cycle the memory. So there's certain, there's
• 47:34
  certain things applications where the, the the the memory is
• 47:39
  actually sinusoidal or whatever it in its consumption.
• 47:43
  And we can adopt, you know, we can what we call wave ride the
• 47:47
  the amount of memory based on where the the application is, is
• 47:51
  running. Yeah.
• 47:52
  So those are those are things we're hoping to add to
• 47:55
  Kubernetes very shortly. How do you even go about that?
• 47:59
  That's what I'm fascinated by is a you saw the problem, you said,
• 48:05
  all right, nobody likes these own mares.
• 48:07
  And we think that we can fix it these different ways by
• 48:12
  injecting memory or trying to swap it out.
• 48:16
  What are you thinking about when you're attacking that problem?
• 48:20
  Well, the first area, I mean, it's a, from where I sit, we're
• 48:25
  kind of kind of a piece of middleware.
• 48:26
  So first thing is we've got to get our hardware partners ready.
• 48:30
  So we are working with several players that are building memory
• 48:34
  pools and and so we've got to get that piece of infrastructure
• 48:38
  ready. And then the second piece is
• 48:41
  then we need to integrate with the Kubernetes layer.
• 48:45
  I think there's some relatively easy ways to integrate with the
• 48:49
  Kubernetes letter or create a hook or an operator or whatever,
• 48:53
  but longer run, you know, we'll, we, we and I say we, the, the
• 48:58
  industry and A and a community are going to need to create some
• 49:01
  probably new standards, right? So, for example, in Kubernetes
• 49:04
  or something called AI, think it's ACSI is it's a standard for
• 49:11
  the storage layer, right? And and there has to be a
• 49:13
  equivalent standard for how do we bring memory as a separate
• 49:17
  first class citizen into a Kubernetes environment?
• 49:20
  Right now we can we kind of hack our way in and show something,
• 49:23
  but with with actually with the existing schedulers and things
• 49:27
  like that. But I think over time we will,
• 49:30
  we'll need to work with areas. There's a like, for example,
• 49:32
  there's another initiative called DRA dynamic resource
• 49:36
  allocation, where to make because right now everything is
• 49:38
  statically done. What's that note is set up by
• 49:41
  the the cube, whatever, it's static.
• 49:43
  But what if the thing changes, right?
• 49:44
  So what if the memory increases and decreases or what if the
• 49:48
  number of GPU available or increases or decreases?
• 49:52
  That kind of dynamicism, I think, is just a general thing
• 49:56
  that the Kubernetes industry is trying to face up to.
• 49:59
  How how do we make this even even more elastic than it
• 50:02
  currently is right now? Elastic at the node level, But
• 50:05
  yeah, but even below that, there's some of these other
• 50:07
  elements that have, you know, their own composability to them
• 50:12
  that need to be more dynamic, you know?
• 50:15
  And do you feel so I, I also like that you're looking at the
• 50:19
  node and you're saying inside of the node, there needs to be a
• 50:24
  way to be more flexible and more elastic and then the whole
• 50:30
  cluster and, and like you mentioned, up and down the
• 50:35
  stack, but also left and right in the stack.
• 50:38
  And we need to rethink and look at all these different areas.
• 50:43
  If we're coming into a world where AI plays such a big part
• 50:47
  of that, I wonder, is this only going to be the evolution for
• 50:57
  the use cases that we are trying to create AI products with?
• 51:02
  Or do you just see it that it's it's going to be better for the
• 51:05
  industry in general if we amplify on every vector that we
• 51:11
  can? Yeah, well, in with respect to
• 51:14
  things like memory, yeah, I mean, there's, there's, there's
• 51:17
  just tonnes of applications today that are getting UN killed
• 51:19
  and, and and evicted or whatever.
• 51:22
  And so all those are benefit from this.
• 51:24
  So yeah, there's definitely that kind of benefit.
• 51:28
  Yeah. And I think same on the HPC
• 51:30
  side, those, those guys that, that whole area, there's a lot
• 51:33
  of convergence between what goes on in HPC world and AI think
• 51:37
  that that area can also benefit from all the memory advances,
• 51:42
  general enterprise applications, databases.
• 51:45
  Yeah, I, I think, I think it'll be a broad based thing.
• 51:49
  But right now, you know, the, the way the AI movement is, you
• 51:53
  know, quite honestly, sucking all the oxygen out of every
• 51:55
  other initiative right now, it's all, and it's probably, it's
• 51:58
  probably the correct thing to do anyhow from a society
• 52:01
  standpoint, This is, this is the hottest area.
• 52:03
  So, but once it, once it settles down, I think some of these,
• 52:06
  yeah, some of these other innovations will obviously
• 52:09
  benefit the rest of the infrastructure and the rest of
• 52:11
  the applications that are still out there.
• 52:12
  They're not, they're not going to go away, you know, databases
• 52:16
  or anything like that, as an example.
• 52:18
  Matter of fact, they'll just, they'll be part of this overall
• 52:20
  infrastructure, right? They're going to be, they're,
• 52:23
  they're going to be our source of truth, our ability to avoid
• 52:25
  hallucinations, whatever, you know, So, yeah, so but and
• 52:29
  they'll, and they'll benefit from all these memory advances
• 52:32
  as well. Bernie, incredible talking to
• 52:33
  you and I want to just mention that I looked up the meta blog
• 52:38
  post and what I was looking at, they'd had the 24,000 or no, it
• 52:46
  was 16,000 GPU cluster and it had remote direct memory access
• 52:52
  over convergent Ethernet ROTE. That probably sounds familiar to
• 52:57
  you. I remember when I read that I
• 52:59
  said this could be in Greek, this could be in Chinese.
• 53:02
  I have no idea what that means. Yeah, RDB is, is called RDB and
• 53:06
  is Rocky Remote over Ethernet. Yeah, ROCE is the acronym.
• 53:10
  Yeah, but remote over converged Ethernet.
• 53:12
  So that's been actually around for quite a while.
• 53:15
  And that's the kind of the bread and butter of the current NVIDIA
• 53:18
  architecture. Well, like I said, I think
• 53:20
  people are ready to figure out how to take take this to the
• 53:24
  next level with and and standardise things with Ultra
• 53:27
  Ethernet. So that exactly the I thought
• 53:30
  they had used the Ultra Ethernet, but it looks like they
• 53:32
  didn't and they were using the InfiniBand for the other 16,000
• 53:39
  cluster. And yeah, from let me see, hold
• 53:43
  on a SEC. The ROCE was used for training
• 53:47
  the largest cluster. Despite underlying network
• 53:50
  differences between these clusters, they both provided
• 53:54
  equivalent performances. So what I remember from this is
• 53:59
  they said that the Rocky or ROCE as you mentioned cluster, they
• 54:05
  were able to optimise it for quick build time and the
• 54:09
  InfiniBand cluster they had full bisection bandwidth.
• 54:13
  Yeah, those, those two technologies are really, they're
• 54:16
  all based on the same physics, you know, here's, here's this
• 54:19
  network, you know, and whether you're running it used to be
• 54:22
  there was a quite a technology gap between InfiniBand and the
• 54:24
  Ethernet. But now that that gap is pretty
• 54:26
  much down to the physical gap that from the from a from a
• 54:32
  physical Phi layer is basically equivalent and is really the
• 54:35
  protocol layer. That's where the differences
• 54:38
  are. Yeah, yeah.
• 54:39
  So, so if InfiniBand is designed to be a lossless configuration
• 54:44
  architecture and, and Ethernet was always designed to be lossy
• 54:49
  and, and there was trade-offs for that, right.
• 54:53
  So, and, and now, you know, now they're all getting kind of
• 54:57
  mushed or even that's going to try to absorb that, that type of
• 55:01
  protocol and, and, and compete with InfiniBand, quite honestly.
• 55:05
  Yeah, so.