Latest / Agentic Conversations (formally mlops.community) / LLM on K8s Panel // LLMs in Conference in Production Conference Part II

Transcript

LLM on K8s Panel // LLMs in Conference in Production Conference Part II · Agentic Conversations (formally mlops.community)

• 0:00
  And now that is a perfect segue into because it feels like that
• 0:05
  was just like the warm up that was a little bit of the sound
• 0:09
  check before we bring on our next guest.
• 0:14
  What's up, Shree? Doing well, Demetrius.
• 0:17
  This looks like a great event. Yeah, man.
• 0:19
  So I've got a shirt that I want to show you all, but I'm going
• 0:22
  to get everyone on stage. We've got Raul with his new
• 0:25
  microphone. And how you sounding, man?
• 0:30
  Can you, can you get the ASMR field?
• 0:32
  Yeah, maybe not that we've also got Manoj, I think it is very
• 0:40
  late where you are. So but it looks light out.
• 0:43
  Is it already tomorrow? Fortunately, I'm travelling to,
• 0:46
  so I'm in Seattle right now. So it's it's the same time zone.
• 0:50
  All right, cool. So that is awesome.
• 0:53
  And then where is my man Patrick?
• 0:57
  Where there he is. Alright, I why am I in big?
• 1:00
  I'm not the star of this show. I'm going to as I make myself
• 1:05
  even bigger. No, Patrick, what's up, dude?
• 1:08
  I love the Martin guitar in the back.
• 1:10
  That is awesome. I love seeing you all.
• 1:13
  And three, I'm gonna leave it off to you.
• 1:15
  This is kind of a what we would call a mix between a panel and a
• 1:19
  fireside chat. We've got so many incredible
• 1:22
  Kubernetes people in the same virtual space.
• 1:25
  And while you kick it off, I'm going to get this shirt that is
• 1:30
  so good. It is not that I hallucinate
• 1:33
  more than chat GP T-shirt, it is the I got it just for everyone
• 1:38
  here. I don't know if you can see this
• 1:40
  that says Kubernetes is a gateway drug and that is what we
• 1:44
  are talking about running LM's on Kubernetes.
• 1:46
  I'm going to get off the stage now and I'll drop a link in case
• 1:51
  anyone wants to buy that shirt and support the community.
• 1:54
  Three. It's all yours, my man.
• 1:56
  Thank you so much. Thank you so much Dimitrius.
• 1:58
  Hey, hello and welcome everyone. Thank you so much for joining
• 2:01
  this, this panel, panel discussion, Fireside chat about
• 2:05
  large language models and Kubernetes.
• 2:07
  You know one of the more well known instances of communities
• 2:11
  and LMS coming together is Open AI and documenting a lot of how
• 2:14
  they've trained their models and actually other models on
• 2:16
  Kubernetes. So I asked that GPT to write
• 2:19
  like a small humorous description about this session,
• 2:22
  and this is what it came up with.
• 2:24
  In a world full of errors, may your code be bug free, may your
• 2:28
  containers be slim and efficient, and may they always
• 2:32
  save smoothly like a rubber ducky in a giant data leak.
• 2:36
  Is this even possible or am I hallucinating?
• 2:39
  Well, on that, maybe not so funny, no.
• 2:42
  It's safe to say that there is a lot of work to do, you know,
• 2:45
  especially when it comes to ellens and humour, and maybe
• 2:48
  Kubernetes has a role to play in that.
• 2:50
  And that's just the perfect segue for us to kind of get deep
• 2:53
  into these conversations about how.
• 2:55
  LMS and communities coexist. What do they, how do they work?
• 2:58
  And so on. So before we get into the
• 3:00
  details like, it might be a good time to just have quick
• 3:02
  introductions. Hey, I'm Manjot.
• 3:04
  I'm part of Lightspeed right now and investing in a lot of these
• 3:08
  new age MLM companies. Excited about that.
• 3:11
  But before that life actually started as as an infrastructure
• 3:15
  engineer where I was building internal infrastructure
• 3:17
  libraries at Google and from there I became a product manager
• 3:23
  in the Kubernetes team. This was, you know, far, far
• 3:25
  into. The Covenant is 1 not all ages
• 3:28
  when we were actually thinking about Mesosphere and other such
• 3:30
  things. So it was working primarily on
• 3:33
  networking and security and eventually you wanted to build
• 3:36
  something of my own. So left and started up in
• 3:39
  machine learning infrastructure and was lastly at Stripe.
• 3:43
  That's fantastic. That's fantastic.
• 3:45
  Welcome, Rahul. Yeah, hello everybody.
• 3:49
  I'm Rahul. I'm the founder of.
• 3:53
  For me, Kubernetes is not just a great gateway drug, it is a way
• 3:57
  to just stay high all the time, right?
• 3:58
  So I I actively think about how do you deploy the LM OPS stack,
• 4:06
  Apologies and to to use that phrase because it's not.
• 4:09
  Cool yet, but what exactly does training your own foundational
• 4:15
  models on Kubernetes look like? What does fine tuning large
• 4:18
  language model on Kubernetes look like?
• 4:20
  Or even deploying applications like the Q&A or search based
• 4:24
  applications, agent applications on Kubernetes and how that can
• 4:28
  help the increase the velocity of what your company delivers or
• 4:33
  deploys is is what I what I like to work on, excited to talk to.
• 4:38
  Everybody. Awesome, awesome.
• 4:39
  Thank you. Patrick, what about you?
• 4:42
  Hey, yeah, I'm I'm working as an architect for a startup.
• 4:46
  Right now. We're focused on LM planning,
• 4:49
  which has been super fun to do. Historically, I've worked a lot
• 4:53
  in Kubernetes. I worked for John Craig at a
• 4:55
  couple called Heptio. We got acquired by VMware.
• 4:58
  I did a lot of stuff that put some features in the Kubernetes
• 5:01
  and whatnot and over the last several years got to, you know,
• 5:04
  traditional kind of machine learning 1.0.
• 5:07
  And then last, you know year we've been focused on generative
• 5:09
  AI and now really kind of heavily focused on.
• 5:13
  Ohh planning. Awesome.
• 5:14
  Thank you so much. And just for for completeness,
• 5:17
  hi everyone, my name is Shree. I am currently an engineer at
• 5:20
  Outer Bounds and Outer Bounds. We are building a large scale
• 5:23
  machine learning platform for handling fully managed platform
• 5:26
  for handling data science and machine learning workloads.
• 5:29
  So yeah, welcome everyone. And maybe we can start off just
• 5:33
  a little bit about what are your impressions of first impressions
• 5:36
  of large language models on Kubernetes, especially maybe
• 5:40
  with respect to training. Where does communities come in?
• 5:42
  What does it offer? What is good?
• 5:44
  Sure, maybe I can kick it off. So when culinary is was, I mean,
• 5:49
  as you all might already be aware, Kubernetes is basically
• 5:52
  inspired by an internal Google product called Borg and it
• 5:57
  basically tried to achieve the same objective, which is how do
• 6:00
  you make sure workloads you know are scalable are reliable, you
• 6:04
  have containers and how do you have an orchestrator so that you
• 6:06
  don't have to manage that infrastructure.
• 6:08
  And that's what Kubernetes was promising to do.
• 6:12
  You probably noticed in that last sentence, I never seen
• 6:15
  anything related to machine learning.
• 6:17
  So so this historical perspective here, Kubernetes
• 6:21
  today grow up without keeping MLML, Core ML in mind and it was
• 6:26
  more about general workloads that traditional workloads that
• 6:28
  you might be running your APIs, your micro services et cetera.
• 6:31
  So it was very micro service heavy and focusing on that
• 6:35
  architecture. But for sure, there are lots of
• 6:38
  in the past couple of years. This is even before LMAO became
• 6:42
  super popular. In the last couple of years,
• 6:45
  we've been seeing a lot of these products, libraries, everything
• 6:49
  come up that help you basically run your machine learning
• 6:52
  workloads on top of Kubernetes because the principles remain
• 6:55
  the same. What are the principles?
• 6:57
  The principles are Kubernetes gives you an abstraction layer
• 7:01
  and orchestration layer on top of your containers that abstract
• 7:05
  away hardware, right? So effectively giving you more
• 7:07
  reliability, scalability and efficiency out-of-the-box.
• 7:12
  Sounds great. Yeah.
• 7:14
  What do you think, Rahul, you had?
• 7:15
  Something to say, Yeah, yeah. And maybe I'll take a different
• 7:18
  perspective on this one. So.
• 7:21
  A few like maybe a month ago we were talking in in this
• 7:26
  community called Tribeca that I'm part of super intelligent
• 7:28
  people and and one of the recurring topics that came up
• 7:32
  was enterprises are looking to deploy and train the models in
• 7:37
  house because they really don't want data to leave their VPC,
• 7:41
  right? And I think that is a business
• 7:44
  problem for which Kubernetes provides a good solution.
• 7:47
  Obviously you have vendors on the cloud service providers or
• 7:51
  even. Startups who have Kubernetes
• 7:54
  native solutions for you to deploy and train your machine
• 7:58
  learning models, but this is what you know.
• 8:02
  Hopefully what Kubernetes allows you to do is have a cloud
• 8:06
  agnostic platform for you to train and deploy your machine
• 8:10
  learning or large language models.
• 8:12
  Interesting. One of the one of the questions
• 8:14
  that maybe Patrick would love to hear your thoughts on this, one
• 8:17
  of the perspectives that gets talked about specifically with
• 8:19
  Kubernetes is that yes it has. Options to run a lot of
• 8:24
  workloads in the machine learning domain land up being
• 8:26
  batch workloads. And there is.
• 8:29
  Yes, Kubernetes offers big support for running batch
• 8:32
  workloads, but oftentimes it seems like it's a little bit of
• 8:36
  an overlooked area, or at least something that hasn't yet gotten
• 8:39
  the attention. It was always about
• 8:40
  microservices, orchestration of services that are long running
• 8:44
  services running servers, whether it is web servers,
• 8:46
  database servers, whatnot. But if you are talking about
• 8:48
  running bad jobs, yes there is a communities job, go do something
• 8:52
  about it. The community has come up with
• 8:53
  other. These things like Argo that have
• 8:56
  actually enabled some of this but still not sort of being the
• 9:00
  first class citizens for batch workloads.
• 9:02
  What's your take on kind of like given that a lot of machine
• 9:05
  learning workloads are batch and Kubernetes offers some support
• 9:09
  but not a lot. Is that is there an impedance
• 9:12
  mismatch there somewhere? Is there something better that
• 9:14
  can happen? I mean, I'd say a lot of machine
• 9:16
  learning 1.0 workloads for batch, and I would say more than
• 9:20
  2.0 like generative AI. There's thoughts actually
• 9:23
  streaming. I see far, I mean there still is
• 9:25
  some bad use cases, but I think we're moving away from Dags
• 9:29
  primarily into kind of like chains and whatnot.
• 9:32
  I think that's changed significantly.
• 9:34
  I would say what Kubernetes is built for which is micro
• 9:37
  services, you can almost think of LM's you know as they grow as
• 9:40
  micro services where you have you know these different parts
• 9:43
  of the models, these partitions that need to talk to each other.
• 9:46
  So you need to schedule containers and network them
• 9:49
  together, right. It's, it's really the same
• 9:51
  principles, it's just kind of being thought of a different
• 9:53
  way. So I think Kubernetes actually
• 9:54
  fits that. These kids?
• 9:55
  Really. Interesting, interesting.
• 9:57
  That's that's good. That's a good point.
• 10:00
  What about things like as we go deeper into kind of GLMS and one
• 10:05
  of the backbones almost, so to speak of LLM has been use of
• 10:09
  GPUs, pretty much every LLM that at least things that have
• 10:15
  generally been talked about always talk about how they've
• 10:17
  been trained on LLM or sorry, on GPUs and then all these other
• 10:21
  conversations about GPU shortages and whatnot Does do
• 10:24
  you think Kubernetes? Helps there.
• 10:26
  Are there things that are better because of Kubernetes for using
• 10:30
  GPUs and being able to kind of train models?
• 10:32
  There are there are lots of other companies nowadays which
• 10:35
  don't run on Kubernetes but still provide you with GPU
• 10:39
  access. What does that happen?
• 10:40
  What do you guys think? Yeah, I mean I think the biggest
• 10:44
  thing is cost, right. So I've worked a lot with
• 10:46
  startups, you know, who are trying to build generative AI
• 10:48
  applications and you know some of them don't want to use Open
• 10:52
  AI for, you know, the privacy reasons or whatnot.
• 10:54
  I yeah, the constitute views is is horrible.
• 10:57
  The start of times for long rides it's like you shut it off,
• 11:00
  you could have you know several minutes to try and booted it
• 11:03
  back up and load the model weights into the GPU like they
• 11:06
  could, they could all take a significant amount of time.
• 11:09
  So this helps with us where you know, obviously we can, you
• 11:12
  know, scale. Hands up and down, you know,
• 11:14
  trying to make that start up time as as fast as possible.
• 11:18
  But I think there's a lot more work that can be done here.
• 11:21
  Right now I'm kind of looking at, you know, can we like hot
• 11:24
  swap Lauras or Q Lauras in the models?
• 11:28
  Right. So could we instead of like
• 11:30
  having to wait to pull up a whole nerd, we just run these
• 11:32
  baseballs and then hot swap Laura Zendaya for special like
• 11:36
  fine tuning capabilities? Yeah, just adding on top of it,
• 11:40
  Patrick said. Right.
• 11:41
  Spot on about cost being one of the bigger.
• 11:44
  Constraints and problems that needs to be solved with GPU
• 11:47
  access, right. So I think in the first sort of
• 11:51
  wave, in the first set of set of changes and and and these
• 11:54
  changes are all, you know, weeks apart, but still just support
• 11:57
  for GPU was added. So you can run, you can enable
• 12:02
  GPU node pools instead of your CPU node pools which enables at
• 12:06
  least some GPUs to be used. But obviously that's not enough.
• 12:09
  Then you know you have these newer things like spot instances
• 12:12
  being available in in the various.
• 12:14
  Can really help with, you know, making your infrastructure and
• 12:17
  cheaper. But even that's not enough
• 12:19
  because just like with everything in infrastructure,
• 12:21
  the devil is in the details. So if you look at the components
• 12:24
  of Kubernetes and this is where going back to what I said
• 12:26
  previously, right, it was designed for traditional
• 12:28
  workloads, not necessarily machine learning workloads.
• 12:31
  So going back, details of how Kubernetes work something like
• 12:34
  scheduler, which I think Patrick briefly touched upon how
• 12:38
  Kubernetes actually decided to schedule and provision nodes,
• 12:41
  pause and everything else. There can be a lot of
• 12:43
  optimization over there. Because obviously there's cost
• 12:46
  and there's also things like, you know, networking, how how
• 12:49
  much data is being transferred between different nodes and
• 12:51
  different calls and and where they are actually located.
• 12:53
  So secondly, if you look at auto scaler again, in fact even
• 12:57
  funnily enough with traditional workloads as we used to say,
• 13:00
  auto Scaler is you know, easiest way where you can shoot yourself
• 13:03
  in the foot. I think it's even harder to make
• 13:06
  all the scanners work with machine learning and LLM
• 13:09
  workloads. So I think there's a lot of work
• 13:12
  where templates can be, provided you know defaults can be
• 13:14
  configured well. And the right sort of parameters
• 13:17
  can be optimised when auto scaling and LM based workload.
• 13:20
  Yeah, just carrying that thread of auto scaling, a couple of
• 13:26
  weeks ago we participated in a hackathon where we trained or
• 13:29
  fine-tuned a large language model on top of Kubernetes with
• 13:33
  RLF using the DSP library, right.
• 13:35
  And originally our our aim was like, hey, we're going to do
• 13:37
  this auto scaling thing and make sure that that works.
• 13:40
  The real challenge right now is GPUs are super rare to get hands
• 13:45
  on so. Even if you want to or you're
• 13:48
  not going to get it, a smart way to think about this is negotiate
• 13:52
  with your cloud provider to get to reserve those nodes that you
• 13:56
  need in advance so that your workloads at least are
• 14:00
  deploying. I think the other side where the
• 14:04
  costing issue that Patrick mentioned and we we talked
• 14:06
  about, I think and this is speculative on my part.
• 14:09
  So we'll know in a couple of months whether this is true or
• 14:12
  not. So as you know that age 1
• 14:14
  hundreds are going to be released and generally available
• 14:16
  or. Starting to get adopted and
• 14:18
  we're going to see the hoard of people start migrating to that
• 14:22
  and guess what that means A1 hundreds are going to be more
• 14:25
  available, cheaper and so on right.
• 14:27
  And so maybe the market will self correct with availability
• 14:31
  but remains to be seen. I think for now either you can
• 14:35
  wait and watch for two months or you know plan out a strategy
• 14:38
  where you reserve your instances and instead of thinking about
• 14:41
  like over optimising on auto scaling, try to get a
• 14:45
  repeatable. Platform in play which in place
• 14:49
  on which you can rapidly deploy and iterate on.
• 14:53
  I think the iteration part of your model is where should you
• 14:57
  should be focused and not on the iterative like the how correct
• 15:01
  or how awesome your Kubernetes platform looks like.
• 15:04
  Interesting. So thinking about large language
• 15:07
  models, so Rahul brought up this point a little earlier that if
• 15:11
  you think about large language models, you kind of sort of
• 15:13
  there are two or three kind of big domains of work, one is.
• 15:17
  Companies or teams starting to train or doing training a large
• 15:21
  language model completely from the ground up, like literally
• 15:23
  everything from the data to train the whole thing.
• 15:26
  It can sometimes take months. Open AI for example has
• 15:28
  documented how it took I think what 3 1/2 months or something
• 15:31
  for them to get it completely right.
• 15:34
  And there is fine tuning of the models and then there is also
• 15:36
  prompt engineering. So two or three different
• 15:38
  domains of work. So Rahul, how does Kubernetes
• 15:41
  play a role in this? Does it make it better?
• 15:43
  Does it? Is anyone type of work better
• 15:45
  suited for Kubernetes versus the other?
• 15:47
  Does that make a difference? And maybe even with GPUs.
• 15:50
  And and I think as as architects and people who are designing
• 15:55
  these platform level solutions, there's no one right answer.
• 15:59
  There's always tradeoffs. I'm Emmanuel Mason, Machine
• 16:04
  Learning engineer at Stripe and author of Building Machine
• 16:07
  Learning Powered Applications and IF.
• 16:09
  You don't want your. Machine learning models to
• 16:11
  explode Well, you should subscribe to this podcast.
• 16:27
  So the trade off here here, you know Kubernetes is is a great
• 16:30
  platform for you to be vendor agnostic, move fast and and and
• 16:34
  and you know be able to automate a lot of your workloads.
• 16:38
  But it's not. It comes with its own kind of
• 16:40
  challenges, right #1 being if you are a Foundational Model
• 16:44
  company, your data is in the petabytes if you are.
• 16:47
  If you are trying to fine tune your models, you know, obviously
• 16:51
  you can use PFD and Laura to kind of, you know, fine tune
• 16:57
  only a small part. But then you still have to have
• 16:59
  the whole model in memory. Guess what, these models are
• 17:02
  really large. The containers that these would
• 17:05
  run on go in the 10s of gigabytes and big data bigger
• 17:09
  models and everything on Kubernetes just becomes a, you
• 17:13
  know, a heavy lifting kind of effort and it it comes with its
• 17:17
  own problems just yesterday. I had to like SSH into the node
• 17:21
  to look at the journal cattle to see what's going on.
• 17:24
  Like hey, why is this part not pulling fast enough?
• 17:27
  Because some nodes it is 550, like 15 seconds, some nodes it's
• 17:31
  like taking a minute and a half. And so, you know these kind of,
• 17:34
  especially when you're talking about like 15 GB container
• 17:37
  sizes, things start getting really, really hard.
• 17:40
  And kind of sort of everything then you know follows that
• 17:43
  larger containers can mean bigger startup times.
• 17:47
  They also mean that it is potentially.
• 17:49
  Possible that you know parts may not come up because that much
• 17:51
  disc space is not available and then you know many many things
• 17:55
  and the cost can go up because you're pulling down, especially
• 17:57
  if you're every part that starts is pulling down 15 GB of data
• 18:00
  for every time. Can you maybe quickly talk a
• 18:02
  little bit about what do you think is a good dentist
• 18:05
  experience? If they have to use Kubernetes,
• 18:07
  like what would be a good? I'm I'm I'm guessing data
• 18:11
  scientists are probably not very very enthusiastic about wanting
• 18:15
  to deal with you know deployments and demon sets and
• 18:19
  DNS resolution and poor DNS and whatnot.
• 18:22
  I. Think I can maybe answer it in
• 18:24
  in one sentence on the best experience for data scientists
• 18:27
  to deal with Kubernetes is not dealing with Kubernetes.
• 18:30
  So and even organizationally, right, if you look at this and
• 18:33
  Rahul and Patrick Simon here, I think organizationally what I've
• 18:36
  seen typically is there's a different deployment team
• 18:39
  versus. Data science team that actually
• 18:41
  deploys these models and deals with all these complexities.
• 18:44
  In fact, funny enough some some organisations were telling me
• 18:46
  that in the past, at least with traditional machine learning
• 18:48
  workloads, they would write their models in Python, there
• 18:51
  would be a team converting that into like C code and then
• 18:55
  finally deploy like this. Just sounds you know 100 steps
• 18:58
  for me that are not optimised. Your thoughts?
• 19:02
  Yeah. I, you know, I think obviously
• 19:04
  traditionally it was used Python, right from Jupiter and
• 19:07
  trying to make that work as good as possible.
• 19:09
  I think what's interesting is generative AI is we're seeing it
• 19:11
  really stretch like you don't really need data scientists as
• 19:14
  much anymore, right? It's like these models are so
• 19:16
  smart that now just regular old engineers could pick them up and
• 19:19
  start building AI apps, right. So I think there's this like
• 19:22
  shift to where, you know, I think Python's like deal, but I
• 19:26
  think a lot of times it's going to be more actually like taking
• 19:28
  those models and like meeting engineers wherever they are.
• 19:33
  And one good point I think about all of this is that, you know
• 19:36
  this is almost kind of the other side of it, which is on the one
• 19:39
  side there is complexity, but the other side of that
• 19:41
  Kubernetes gives you is, is composability.
• 19:44
  Like there are so many different tools and services that we
• 19:47
  talked about literally in these what 8 minutes that we've had
• 19:50
  this conversation about different libraries, different
• 19:53
  tools and the amount of innovation happening in there,
• 19:56
  deep speed, Laura, theft, so many of them coming up.
• 20:02
  It's it's easier to get these up and running.
• 20:05
  It's easier to containerize this and kind of sort of deal with
• 20:07
  them in within containers because with the isolation
• 20:10
  boundaries compared to let's say saying that here is 1 GPU
• 20:13
  instance 4 of you go and SSH to that box and run whatever you
• 20:16
  want. I'm sure everyone will step on
• 20:18
  each other's toes saying I want Python 3.9, someone else wants
• 20:22
  deep seated version 4.7 and then it's just like it will be chaos.
• 20:25
  So there are benefits of Kubernetes.
• 20:28
  There is also the complexity of it, but the composability and
• 20:31
  the extensibility that you get with it to some extent, maybe
• 20:34
  even isolation that can be that is very that's very appealing,
• 20:38
  but it probably comes a little later.
• 20:40
  Do you think we are there yet? Do you think there are teams who
• 20:43
  actually can benefit from all this?
• 20:45
  You know, large scale platforms that can be built on Kubernetes?
• 20:50
  Yeah, platforms. Yeah, so, so one quick point
• 20:53
  that I think it's obviously a staging into nodes kind of makes
• 20:57
  makes sense in, in, in ways. But if you look at the non
• 21:02
  foundational model and we've talked about most of these
• 21:05
  foundational models and fine tuning kind of approaches here
• 21:07
  so far. The others, which I think
• 21:09
  Patrick was alluding to, was like, let's say you wanted to
• 21:12
  spin up a Q and a bot that you've created with some vector
• 21:16
  DB and some. Store you know data going in
• 21:19
  there answers maybe users assures opening I bot or
• 21:24
  anthropics new stock to compliant but like.
• 21:29
  Creating these services that can help support these kind of
• 21:32
  applications can also also work with Kubernetes, right.
• 21:36
  I think that's that's like the flexibility at the end of the
• 21:39
  day Kubernetes is not the silver bullet like like we we you know
• 21:43
  everybody here agrees it's it's not but you know there are these
• 21:46
  challenges that if you were doing this challenge on a
• 21:51
  regular node that you have set up, the community support is
• 21:54
  just not there. You try to Google search, you
• 21:56
  try to ask Chad GPD to help you out.
• 21:57
  It's not there I think Kubernetes.
• 21:59
  Just way more adoption and you know the support that can allow
• 22:04
  you to debug and I think that is part of the developer experience
• 22:07
  as well and not just having a dev box and messaging, but also
• 22:10
  if you get stuck how do you unblock yourself I think.
• 22:13
  It's still best in the business of, you know, having multiple
• 22:16
  services and like Royal Rahul just said, you know, you might
• 22:19
  be running a vector database, you might have a web app, you
• 22:21
  know, then you might have your own lab and you need those all
• 22:24
  to coordinate to build your applications.
• 22:26
  So there's still nothing as good as Kubernetes like run those all
• 22:29
  in one place. Orchestrate them together and
• 22:31
  have like 1 coherent API. Just adding, I think you
• 22:35
  mentioned these 3-4 areas as well right in the in the
• 22:38
  previous question on prompt engineering versus fine tuning
• 22:43
  versus training and then these service oriented architectures.
• 22:47
  I think the last bucket I'll add there is inferencing right out
• 22:51
  of all these sort of workloads for sure.
• 22:55
  Kubernetes right now is far more scale battle tested.
• 23:00
  I would say for the training and the fine tuning parts versus the
• 23:04
  inferencing part, but coming down to the inferencing part as
• 23:06
  well. When you have and and I think at
• 23:09
  least the applications I've seen ourselves sort of figuring out
• 23:12
  exactly what that architecture would look like on having an
• 23:14
  external vector app, not just for training but also for actual
• 23:18
  inferencing. That sort of also fits into the
• 23:21
  service or Indian architecture for which Kubernetes was
• 23:24
  basically developed. So having said all of this, I
• 23:28
  think like you know is there, I'm sure there are lots of
• 23:30
  people who are maybe starting out or who have always kind of
• 23:33
  sort of felt like, yes, we've been able to train models either
• 23:36
  locally or using some service, but we want to kind of
• 23:38
  standardise on something and Kubernetes seems like an
• 23:42
  industry standard almost. What's a good place to start?
• 23:45
  Like what would be your recommendation for where to
• 23:47
  begin and how would they how would they go about doing this?
• 23:54
  I'm I'm almost hesitant to say this, but you know a lot of
• 23:58
  companies have come up with simpler, you know, one line kind
• 24:01
  of commands that data scientists can run to get started if they
• 24:05
  don't have a team that can help you set up the platform right.
• 24:09
  So you just write your Python code, pytorch code or whatever
• 24:13
  you want for your training the large language model and then
• 24:15
  just have a one line command. They will take care of
• 24:18
  containerizing that code and deploying it on on on their
• 24:22
  Kubernetes. Thing, right.
• 24:23
  And so they are kind of taking care of the orchestration for
• 24:25
  you. I think that's a good place to
• 24:27
  start if you if you really want to get started and you don't
• 24:30
  have a team supporting you. But I would almost be hesitant
• 24:34
  to say that that's the best way in which you do a repeatable
• 24:38
  kind of interactive. So once you deploy like Majora's
• 24:40
  good point about inferences, right, once, once that large
• 24:42
  language model is an inference, then you have this human
• 24:45
  feedback loop that you need to think about, right.
• 24:47
  So people are supporting or downloading the replies that the
• 24:51
  agent is giving. OK, what do we need to?
• 24:53
  Iteratively improve on and now suddenly your data is again
• 24:55
  leaving your VPC to go into some other cloud.
• 24:59
  How do you connect all of that? Eventually you're going to have
• 25:02
  to have a robust platform on top of Kubernetes to support all
• 25:06
  that. And maybe it's not a bad idea to
• 25:08
  just have some help from from somebody who's helping deploy
• 25:13
  this on top of your Kubernetes platform or or or you know,
• 25:17
  learn it yourself. I think there's enough resources
• 25:19
  available in upcoming that that that would be super helpful.
• 25:22
  So it's a fair point. I think what you're saying is
• 25:24
  there is there is a layer that can be built on top that to some
• 25:28
  extent hides some of these complexities that people are
• 25:30
  talking about and it can provide that experience that people
• 25:33
  directly. You know, you write your code
• 25:35
  and run it with some special command or decorators or whatnot
• 25:39
  and it eventually goes to the back end and runs it on
• 25:41
  Kubernetes. As time goes by you might want
• 25:44
  to be interested in what is actually happening behind the
• 25:46
  scenes and that will be a good introduction to kind of start
• 25:49
  with something basic and then do this.
• 25:52
  Given that we like Demetrius is going to tell us that in a few
• 25:55
  minutes. I also want to touch up a little
• 25:57
  bit on a little bit of these non-technical aspects especially
• 26:00
  Manjot like your experience. I'm sure there are many people
• 26:03
  who have been thinking about you know building companies and
• 26:05
  products that kind of sort of aid and assist data science and
• 26:09
  ML applications. Like what do you see on the
• 26:11
  non-technical side like Kubernetes or LMS. Does that
• 26:15
  seem like things that have that where products can be built that
• 26:19
  can be good viable businesses going forward?
• 26:21
  Yeah. So there's no real answer right
• 26:23
  now. I think this space is moving so
• 26:25
  fast and everything is so recent that people are still figuring
• 26:29
  out, OK, what are the best tech stacks for hosting an
• 26:32
  application in production? What is the best way to use
• 26:35
  Kubernetes or not? Like, what are the alternatives?
• 26:37
  Right. So, so every day you see like
• 26:40
  new things coming up, like there's COVID, the hosted model,
• 26:43
  there's Replicate, which is like completely like you don't have
• 26:45
  to care about anything, which Rahul touched upon, right.
• 26:47
  There's so much happening. And to answer your question on
• 26:51
  are there you know. Business is possible, I think
• 26:55
  for sure there is, if there were ever a white space to create
• 26:58
  like you know, a cloud offering from scratch, this is a, this is
• 27:02
  a very interesting white space. There's such a dose of GPUs
• 27:06
  right now. There is also lack of knowledge
• 27:09
  on what is the best way to host, manage and serve your
• 27:14
  applications, right. So there is a real white space
• 27:17
  right now in terms of a being the thought leader that hey, you
• 27:20
  know what like you care about. What are the big problems like?
• 27:23
  Serving in an application like hallucinations, right?
• 27:26
  That's one of the biggest core challenges for enterprises.
• 27:28
  Second big problem is cost which Patrick also mentioned.
• 27:31
  And last can be just reliability and latency.
• 27:35
  So when someone thinks about creating the next, you know,
• 27:39
  infrastructure platform to serve LLM applications, these are the
• 27:43
  broad 3-4 problems that they have to tackle.
• 27:46
  And there is a good chance that the answer under the hood will
• 27:49
  be Kubernetes. But the I think the idea is to
• 27:52
  abstract these things away. Horses display all the moving
• 27:55
  parts. Interesting.
• 27:58
  What do you think about the case?
• 28:00
  Go ahead. Go ahead, Patrick, go ahead.
• 28:02
  Get LM will probably be the abstraction on top right, which
• 28:05
  already kind of starting to see we're probably yeah.
• 28:07
  I'd also say it really depends on the size of the organisation
• 28:10
  right. Startups are going to really
• 28:12
  wrestle with the conservatives and the costumes is already
• 28:14
  really high And if you have Kubernetes on top of that you
• 28:17
  could even higher you know if you compare that to running
• 28:19
  something like Cloud Flare worker using Open AI API that
• 28:23
  gets a lot cheaper, right. So I think you have to kind of
• 28:26
  weigh like the size of your organisation and you know the
• 28:28
  capital they have available. Uh, I'll take one question from
• 28:32
  the audience here. There is a question about can
• 28:34
  you describe the pros and cons of Kubernetes for element
• 28:38
  training versus the pros and cons for using Kubernetes for
• 28:43
  LLM inferencing? So, Kubernetes for training
• 28:46
  versus Kubernetes for LLM inferencing, What are the
• 28:48
  tradeoffs? I think, Rahul, you've done both
• 28:51
  of these to some extent, yeah. And this is this is a long
• 28:54
  question but long answer which I'll try to keep short.
• 28:57
  I think when I'm I'm advising my clients I'm I'm telling them
• 29:01
  about like I'm I'm optimising for velocity and you know having
• 29:07
  a platform that is easily like making that's helping you deploy
• 29:11
  models faster and I think that is one thing that I think
• 29:14
  Kubernetes is great on for training.
• 29:17
  For the inference side, I I defer to majority.
• 29:20
  If you had raised that point. Yeah, no, I think the infant
• 29:23
  side, right, I we've already discussed some of the challenges
• 29:26
  present in managing a model as well as challenges in Kubernetes
• 29:30
  specific components on how they are designed today versus how
• 29:33
  they're supposed to work. I think one more challenge I'll
• 29:36
  mention is a lot of these. You know, a lot of the LLM
• 29:40
  workloads might be using these. You know, specific libraries and
• 29:44
  drivers that work with accelerators, some specific
• 29:47
  hardware component, right? And the whole point of
• 29:49
  Kubernetes. And containers issue abstract
• 29:52
  all of that away. So I think the world is still
• 29:54
  sort of also struggling right now finding that sweet spot of
• 29:58
  OK, I need to abstract away hardware, but like this world is
• 30:01
  not quite abstracted away 100% so influencing.
• 30:05
  I think there's a lot of work to be done.
• 30:07
  If someone were to start building an application today,
• 30:11
  you are better off picking up some of these open source
• 30:14
  solutions present that at least abstract some of that complexity
• 30:17
  away. But I think we still see a lot
• 30:19
  of development. Eventually new products and
• 30:22
  services being built here and there is also a little bit of
• 30:25
  this inferencing at times can be real time online inferences
• 30:32
  versus training being kind of bad jobs.
• 30:34
  So it goes back to one of the original earlier questions we
• 30:37
  talked about that at least for bad jobs is of course it's
• 30:41
  there, it's useful versus Kubernetes for inferencing can
• 30:44
  be like you know server based processes and whatnot.
• 30:46
  So there will be some of that that comes out.
• 30:49
  You mentioned about the abstractions like one of the
• 30:51
  abstractions that that's available today is multiple open
• 30:55
  source projects out there. One of them that at least I'm
• 30:57
  involved in called meta flow. So that you can check that out
• 30:59
  in case people are interested in learning about abstractions that
• 31:02
  can deal with Kubernetes or hide the complexity of Kubernetes.
• 31:06
  But but yeah, I think this was, this was great and you know we
• 31:11
  write on queue. We have someone, we have a
• 31:13
  person with a Red Hat here. I have a pirate Demetrius.
• 31:19
  Yeah, I've. Rated the ship and I have come
• 31:22
  to steal the show back. That was absolutely incredible.
• 31:27
  Thank you all and I will remind everyone.
• 31:30
  Well Rahul, I may need to make new shirts now like Kubernetes
• 31:34
  is a gateway drug or I guess Kubernetes is the way to stay
• 31:37
  high, that is you can't really see it because of the lighting
• 31:41
  there. Maybe you might be able to see
• 31:43
  it all through. You know, I I, I wanted to make
• 31:46
  a joke about sort of wearing your heart on your sleeve your
• 31:50
  you know Kubernetes is a gateway drug Trump's me there.
• 31:53
  Yeah, it's. Hard to follow that one.
• 31:54
  Man, that is so true. I know I did a little bit of a I
• 31:58
  did a a wardrobe change and I busted out the I hallucinate
• 32:03
  more than chat GP T-shirt that you can get here in case anyone
• 32:07
  wants to grab a copy of that. We've got it here folks on the
• 32:13
  panel I must say awesome. And Raul, I'm going to give you
• 32:17
  a shout out right now because you are running the after party
• 32:20
  in San Francisco. You got like a bus or what have
• 32:22
  you got? That's right.
• 32:24
  So if you're in San Francisco, you'd probably know Mission Bay
• 32:27
  has this awesome outdoor space called Spark Social.
• 32:31
  There's Sun outside. It's a nice day for us to, you
• 32:33
  know, mingle. We have a bus, double Decker bus
• 32:37
  that we'll be meeting up. There's 120 people already
• 32:40
  signed up. So I I, I think we can get a
• 32:43
  couple of dozen more. So just join us.
• 32:46
  Spark Social is a big place. This has been an awesome panel.
• 32:49
  Thank you everybody for your valuable thoughts.
• 32:52
  Yes. Yes.
• 32:53
  Thank you so much everyone. This was a lot of fun and a lot
• 32:57
  of interesting conversations. Over to you, Demetrius.
• 32:59
  Yeah. And any more questions for these
• 33:01
  Kubernete experts? The cats as they like to call.
• 33:04
  I think that's where the cool kids call it.
• 33:06
  So I'm gonna pick that up, go ahead and throw it in the chat,
• 33:10
  throw it into Slack, wherever it may be.
• 33:13
  I'll see you all later. Hi, this is Michael Eric.
• 33:18
  I'm the founder of Pametan Data Innovation and Confetti AI and
• 33:22
  this is a quick public service announcement.
• 33:24
  If you're tired of having nightmares about the machine
• 33:26
  learning models you have in production, then I encourage you
• 33:29
  to subscribe to the ML OPS Community Podcast.
• 33:32
  It's where all the cool ML OPS kids hang out and learn how to
• 33:35
  tame their ML models.