Latest / Agentic Conversations (formally mlops.community) / Serving LLMs in Production: Performance, Cost & Scale // CAST AI Roundtable

Transcript

Serving LLMs in Production: Performance, Cost & Scale // CAST AI Roundtable · Agentic Conversations (formally mlops.community)

• 0:00
  Is, I think, good for both of our projects, because as you get
• 0:05
  more and more and more models and as people are building more
• 0:08
  and more AI tools and plugging in AI into their applications,
• 0:14
  you're just going to see. I mean, I imagine that I'd be
• 0:17
  very curious to get your guys the thoughts on this because
• 0:19
  right now I suspect that when people start out, they not
• 0:23
  really think about serving, right?
• 0:26
  I mean, you have to kind of make it to some level of maturity.
• 0:30
  Either you made it past, you know, you're like initial like,
• 0:35
  you know, first like a prototype, but like a proof of
• 0:37
  concept. And then you manage to get some
• 0:40
  people need to use the thing before you really care about.
• 0:44
  Is that right? Like how do you think about, by
• 0:48
  the way, we're just diving right in.
• 0:50
  If you guys are joining us, we had a fascinating conversation
• 0:53
  even in the green room and we're just opening the door to let
• 0:57
  everybody in. If this is the if if you haven't
• 1:01
  yet joined us, this is the ML OPS community.
• 1:04
  This is going to be our first panel for the year 2026.
• 1:08
  Find us ML OPS dot community. If you're not yet a member of
• 1:10
  the Slack workspace, we have probably one of the best Slack
• 1:13
  workspaces in all of tech. Honestly, just join the
• 1:16
  everybody that's in building anything meaningful in in, in
• 1:21
  machine learning and OPS and AI is all in that Slack workspace.
• 1:25
  So join that, subscribe to the newsletter, podcast, all of that
• 1:28
  stuff. Wherever you find your podcast,
• 1:29
  you'll find us there. OK, I was just starting to ask
• 1:32
  you guys the question. Then today we're going to be
• 1:35
  focusing on serving. And that isn't it feels like
• 1:39
  it's just a Grand Canyon of of considerations there.
• 1:42
  There's so much going on there and for most people, and perhaps
• 1:46
  maybe for a lot of people that are tuning in, they might never
• 1:49
  have even conceptualized the difficulty and the complexity of
• 1:53
  this terrain of serving because they might still be in that pre
• 1:57
  serving camp. They're just in a they're not
• 2:00
  yet wearing those glasses to show them.
• 2:02
  Oh my God, actually, once I actually make the production,
• 2:05
  when people, once people actually start to use this
• 2:07
  thing, it's a whole can of worms for how to actually serve these
• 2:11
  things. So what have you seen?
• 2:14
  Well, first of all, is that true based on your experience, is
• 2:16
  that? 100% definitely.
• 2:19
  Yeah, that's totally true. Like first you need to learn to
• 2:23
  crawl, then then you can actually walk, right.
• 2:26
  So I think Fiona can actually share some great stories that
• 2:29
  we've seen with our customers exactly struggling with this.
• 2:32
  So, so if that's the case, then maybe the two of you can start,
• 2:36
  can introduce yourself real quick, who are you?
• 2:39
  And then and what you do, you're both from Cast AI.
• 2:43
  What is it? That is the lens through which
• 2:46
  you guys are looking at this increasingly complex landscape,
• 2:50
  right. Just so that we know when people
• 2:51
  are going to ask questions in the chat, folks write down
• 2:55
  questions in the chat, If, if, if you have any, we will know
• 2:58
  how to best direct those. Oh, nice, ceremonious.
• 3:02
  So yeah, it does all these funky stuff.
• 3:06
  We will know how to Orient and to direct the right question to
• 3:09
  the right person. Joanna, Igor, first of all,
• 3:12
  thank you guys for joining today.
• 3:14
  Do you want to start? Give us just a a brief
• 3:16
  introduction to each of you, Joanna, if you want to start.
• 3:20
  For sure, it's nice to be here. So I'm Joanna, I'm a product
• 3:24
  manager. I have a background in fintech
• 3:27
  and entertainment and today marks the second year to date
• 3:32
  since I've been with Cast AI. Congratulations.
• 3:35
  The balloons should have come for you.
• 3:36
  Maybe that's what it was. That's you know.
• 3:39
  Exactly celebrations. Yeah, On my side, I'm just a
• 3:45
  little bit shy over the year with the guest.
• 3:47
  I currently stuff machine learning engineer there working
• 3:51
  on a platform product for that for the AI before that worked
• 3:55
  with the AI when it was not that popular.
• 3:57
  So before the GPT and the recommendation systems and your
• 4:01
  usual textual classification stuff, and it'll be before, it
• 4:05
  was super huge. Nice.
• 4:08
  Before it was cool, before it was a household name.
• 4:10
  At some point it was crazy. I mean, everybody just started
• 4:13
  saying things like DeepSeek. I'm like, how do you even know?
• 4:16
  All of a sudden just the most technicals.
• 4:18
  OK, You wanna do you wanna share with us?
• 4:20
  Maybe you have a presentation to share.
• 4:23
  And afterwards Igor does too. Maybe just as a way to kind of
• 4:27
  bridge us over to that presentation, you can tell us a
• 4:30
  little bit about, are there, are you seeing these challenges that
• 4:34
  customers are having? Is that the primary thing that
• 4:36
  you're engaging with is figuring out how how to go from the
• 4:40
  crawling to the running? 100% So last year, what we
• 4:47
  noticed is the majority of the customers that we talked to,
• 4:51
  they were just experimenting, right?
• 4:53
  You have an idea, you think AI is going to help you solve it,
• 4:58
  right? Or you find out, you identify a
• 4:59
  problem and you try to solve it with AI, but you don't start by
• 5:03
  thinking about the infrastructure and you don't
• 5:05
  start about thinking about thinking about scaling, right?
• 5:10
  When you're building a car, you're not thinking first about
• 5:13
  how to build a factory to build a car, right?
• 5:15
  You're thinking about the dynamics of that car and the
• 5:17
  mechanics, and everything is the same here.
• 5:19
  So that's definitely something that we noticed.
• 5:22
  And I think the year 2026, we'll see more experiments or more
• 5:27
  projects moving from experimentation and POC, finding
• 5:32
  product market fit, scaling and then starting to hit a lot of
• 5:37
  challenges and blockages in infrastructure.
• 5:40
  Awesome. OK.
• 5:41
  So a lot of questions that I already have about exactly what
• 5:45
  that pathway looks like. I might hold off until you do
• 5:48
  your presentation and we'll get off the stage.
• 5:52
  We'll give you the space. I could see your screen right
• 5:56
  now and we'll be back in 15 minutes.
• 6:00
  Does that work? Awesome.
• 6:03
  See you soon. See you soon.
• 6:06
  OK, So what I'm going to talk a bit about the lessons that we
• 6:11
  learn building our product and talking to our customers over
• 6:16
  over the years. And then Igor is going to take
• 6:19
  you through the nitty gritty technical details of everything,
• 6:23
  which I know that he's excited to, to get started with.
• 6:27
  But so some time ago we read the papers that proved that you
• 6:34
  don't need the biggest, the baddest model of the bunch for
• 6:37
  every task, right? We saw that you can save 8090%
• 6:43
  in cost by picking the right model for the right job.
• 6:47
  Small models, you use them for low complexity tasks, while
• 6:51
  larger models you use them for planning and for reasoning and
• 6:54
  so on. So we did the normal thing.
• 6:58
  We set out to build an intelligent LM router, something
• 7:02
  that identifies the prompt complexity and routes it to the
• 7:06
  best available model. Sounds good, right?
• 7:09
  So we built it. We tested with real users and
• 7:13
  that helped us identify 2 problems.
• 7:17
  The first one is small node. The small models, they need fine
• 7:21
  tuning more often than not the off the shelf version models
• 7:28
  weren't accurate enough for production use cases.
• 7:31
  So while we did see cost savings up to more than 90%, there was a
• 7:39
  tax to pay in accuracy. So in production, we, we, we
• 7:44
  immediately hit this kind of this wall, right.
• 7:48
  So if you want to use small models, more often than not you
• 7:51
  need to fine tune them. And secondly, these open source
• 7:56
  models, they need to be hosted somewhere.
• 7:59
  Like I said in in the beginning, the majority of the teams that
• 8:03
  we we were talking to last year, they were still experimenting,
• 8:08
  right? They had an idea and they were
• 8:10
  trying to get to product market fit and they weren't thinking
• 8:14
  about building complex pipeline or infrastructure, which makes
• 8:19
  sense. So they went for the convenient,
• 8:21
  they went for open AIAPIS and they started building from
• 8:27
  there. But telling these customers,
• 8:32
  getting them to move to open source models from the
• 8:36
  convenience of the model API, it was challenging because all of a
• 8:42
  sudden we were telling them that now you need to build this, you
• 8:46
  need a Kubernetes cluster and you need to orchestrate your GPU
• 8:51
  and figure out auto scaling and VLM parameter tuning and all
• 8:55
  this stuff, right? So it was kind of a steep
• 8:58
  learning curve. So we step back.
• 9:03
  We now we understand we have these problems.
• 9:05
  We took a step back and we asked, you know, we already have
• 9:09
  a successful product in Cast AI that automates and optimizes
• 9:14
  Kubernetes clusters. Why not build something that
• 9:17
  lets customers deploy open source models directly in their
• 9:22
  Kubernetes clusters? We can handle the GPU
• 9:25
  orchestration, the auto scaling, the VLLM parameter tuning, our
• 9:30
  users can focus on their AI applications.
• 9:33
  So that became our product that became AI neighbor.
• 9:36
  But then we learned the next lesson.
• 9:39
  We hit the next challenge, which is that AI engineers often rely
• 9:44
  on or have to rely on their DevOps teams and colleagues to
• 9:48
  create the clusters to connect them, to manage them.
• 9:52
  And what this means in reality is AI teams get stuck waiting
• 9:57
  for infrastructure access. No cluster, no model access, no
• 10:01
  progress. So we decided to put our money
• 10:05
  where our mouth is, and we started hosting the models
• 10:09
  ourselves, dog footing our own platform.
• 10:13
  We focus on optimizing the token true booth, the GPU
• 10:17
  orchestration, utilization and so on.
• 10:20
  And our users can just call our endpoints and pay for what they
• 10:24
  use, right? When they're ready, they can
• 10:28
  naturally progress towards managed GPUs and self hosting
• 10:32
  and a more complicated more control over the model, the
• 10:36
  infrastructure and self hosting. So that journey taught us
• 10:42
  something important, that there's no single right answer.
• 10:48
  Different situations need different approaches.
• 10:51
  Which brings me to the first real lesson.
• 10:55
  The biggest mistake we see teams make is assuming there's one
• 10:59
  right way to deploy a lamps. We're either an API shop and we
• 11:05
  only use private and model APIs, or we self host everything.
• 11:13
  But that kind of binary thinking can cost money and can slow you
• 11:18
  down. How we see it in reality is that
• 11:23
  this is a spectrum and where you land depends on your specific
• 11:28
  situation and your specific workloads.
• 11:31
  Let's have a quick look at some of the deployment options that
• 11:34
  you typically have. First, there was the model API.
• 11:41
  The model API. They make sense for when you get
• 11:44
  started, when you're experimenting you.
• 11:47
  Like I said, you have an idea, you want to validate, you have
• 11:50
  some hypothesis, you want to validate them.
• 11:52
  You don't want to lose the time building the underlying
• 11:55
  infrastructure just to get there and see that your hypothesis was
• 12:00
  invalidated. So easy way to get started is
• 12:04
  model AP is you pay per token. No infrastructure to think
• 12:08
  about. You can use the models
• 12:11
  out-of-the-box and it's also it works well for small to medium
• 12:17
  scale, especially if you have workloads there that are input
• 12:22
  heavy, right? Your next option is to use
• 12:28
  Manage GPU. So this is kind of the middle
• 12:31
  tier. Your AI code runs in a container
• 12:35
  on some managed GPUs. Here you're not paying per
• 12:38
  token, but you're paying per compute usage.
• 12:41
  Still don't pay. You're still not paying for the
• 12:43
  full GPU. You're only paying for help for
• 12:46
  the compute usage of your container.
• 12:49
  This means that you can run your own models, you can have control
• 12:53
  over your inference, over your server configuration.
• 12:56
  You control the application layer.
• 12:58
  We handle the GPU orchestration underneath all the provisioning,
• 13:02
  the scaling, the capacity planning and so on.
• 13:05
  And this type of option works well for use it for teams that
• 13:11
  have custom AI models that have fine-tuned their models and have
• 13:17
  maybe not enough usage to justify a full GPU.
• 13:23
  And they have fine-tuned their models and they cannot use the
• 13:26
  model API. They have spiky or unpredictable
• 13:31
  traffic or when you do heavy pre and post processing.
• 13:43
  OK, so this, that's the managed GPU.
• 13:49
  And thirdly, we have self hosting.
• 13:52
  You see how we kind of, we started with the self hosting.
• 13:56
  So we started from the from the back and we worked our way to
• 13:59
  the front. So when the self hosting makes
• 14:02
  sense. First of all, this is when your
• 14:07
  model runs completely in your own VPC, right?
• 14:11
  You start to pay for the actual GPU, you reserve the GPU, you
• 14:16
  have to pay for it regardless of how much you said you have.
• 14:21
  So you can use tooling to make your models more were run more
• 14:25
  efficiently. Of course orchestration, auto
• 14:28
  scaling, optimization, hibernation, but the
• 14:32
  infrastructure in the end is yours.
• 14:36
  This works out very well for teams that have strict data
• 14:41
  privacy rules that are handling sensitive or processing
• 14:46
  sensitive information. So they need to have maximum
• 14:50
  control over the model, the model weights the data and so
• 14:54
  on. Or for output heavy use cases or
• 15:03
  and especially for teams that are already running using
• 15:06
  Kubernetes because that just makes the transition so much
• 15:10
  more smoother. So, but what's the right answer
• 15:14
  here? The right answer that we or that
• 15:18
  the answer, what we see in reality after talking to lots of
• 15:22
  customers is that the reality is hybrid.
• 15:26
  You, you choose your deployment option based on your specific
• 15:31
  use case needs, right. So I'm not talking about
• 15:34
  organization needs, it's use case based.
• 15:39
  So you, we see that most mature teams end up running a mix.
• 15:44
  They use AP is for some workloads, especially those that
• 15:49
  can utilize out-of-the-box models with no customization,
• 15:54
  manage endpoints for others, and self hosted for the stuff that
• 15:58
  really demands it. And the key here is to design
• 16:02
  for optionality and to design for movement.
• 16:05
  Your needs will change. So can you move between these
• 16:09
  tiers without rewriting your application?
• 16:12
  Because if migration requires a view, right, you'll stay where
• 16:15
  you are. You're either overpaying or
• 16:18
  underserving. Lesson 2 that we learned through
• 16:23
  our journey is that self hosting is harder than it looks.
• 16:28
  It looks hard, but it's actually even harder than that.
• 16:31
  So tell me if this sounds familiar to any of you.
• 16:35
  Let's say that you've decided self hosting makes sense for
• 16:39
  your use case. You need 70 B.
• 16:43
  You need to run a 70 bit in parameter model.
• 16:47
  You need real time latency and that means serious hardware.
• 16:50
  So you need at least one node running 8H1 hundreds to service
• 16:57
  in production. So you go to your Cloud Console
• 17:01
  to provision, but you can find it because there's no quota for
• 17:04
  it. Or at least there's no quota for
• 17:06
  your region or at all. OK, you request the quota
• 17:10
  increase. You fill out the form, you fill
• 17:12
  out the justification. It gets rejected.
• 17:15
  You explain why you need it. You provide an adequate
• 17:19
  explanation. A week later, it finally gets
• 17:22
  approved. Yay, we got the, we got the GPU
• 17:25
  quota. Now you go back to your to your
• 17:29
  Cloud Console and you try to provision your note, but
• 17:32
  surprise, there's insufficient capacity in that region.
• 17:36
  So now you have permission to use the GPU, but it doesn't
• 17:41
  exist. And this is the GPU market
• 17:44
  reality right now. OK, but let's say you do get the
• 17:49
  capacity. Let's keep past that.
• 17:51
  You got the capacity. You maybe you have a tool in
• 17:55
  place like SDI that can expand your clusters across regions and
• 17:59
  clouds to provision provision the GPU.
• 18:02
  Cool. Now what?
• 18:04
  Now you have an expensive node that you have to, you have to
• 18:09
  pay around 20 K dollars a month. But here's the question that
• 18:15
  nobody asked so far or maybe it wasn't really considered.
• 18:20
  Do you have enough usage to justify this?
• 18:23
  Because if you're paying 20,000 a month for a node that runs at
• 18:28
  50% utilization, you're actually paying.
• 18:31
  Thank you for the inference and thank you for the IO GPU.
• 18:36
  There's ways to tackle this of course, like time sharing, which
• 18:40
  can solve the utilization problem depending on your usage
• 18:44
  and your memory needs. So instead of dedicating a GPU
• 18:48
  node to 1 workload, you can share the capacity across
• 18:52
  multiple nodes, maybe teams use cases and so on.
• 18:57
  So when one workload is idle, another one can use the
• 19:00
  capacity, the GPU stays busy and the effective cost per token
• 19:04
  goes down. And that's not theoretical.
• 19:07
  One of our education tag customers actually migrated from
• 19:11
  AWS stage maker to a timeshare setup in Cast AI.
• 19:15
  And the result was that they got they it was 40% cheaper with the
• 19:23
  same or better latency. This allowed them to expand
• 19:27
  faster. So it's not just about cost or
• 19:30
  the cost savings, but what those costs, cost savings unlock you
• 19:34
  to do. But there's the important thing
• 19:39
  here is that there's not one-size-fits-all solution.
• 19:44
  Different workloads have different trade-offs and you
• 19:47
  always have to balance these trade-offs.
• 19:50
  You have a latency sensitive customer facing up.
• 19:54
  Well, maybe you want dedicated capacity to guarantee response
• 19:58
  times. So you might be running at least
• 20:02
  one on demand node and maybe you can burst into spot.
• 20:06
  Capacity to handle the traffic spikes.
• 20:11
  Or maybe you have an internal batch processing pipeline, in
• 20:14
  which case I'm sharing makes perfect sense for you.
• 20:18
  Or maybe just spiky unpredictable traffic where
• 20:22
  managed endpoints with instant scale up where you don't have to
• 20:25
  wait for the GPU node to come online.
• 20:30
  It's a better fit, but the mistake here is assuming that
• 20:34
  you can have one architecture that works for everything,
• 20:37
  because it doesn't. What we learned is that you need
• 20:42
  to evaluate each workload individually.
• 20:45
  What's the latency requirement? That's very important, and it's
• 20:48
  going to drive a lot of important decisions.
• 20:50
  What's the traffic pattern? What's the cost sensitivity, and
• 20:54
  what's the data? What are the data restrictions?
• 20:59
  And my colleague Igor is going to explain in a bit why this
• 21:04
  matter when it comes to infrastructure decisions.
• 21:09
  But underneath all of this, you increase your quota, you have
• 21:14
  the GPU capacity and everything. You're not done right.
• 21:19
  There's still real complexity to be tackled here.
• 21:23
  If you're hosting, let's say you're using LMS, you're hosting
• 21:27
  LMS. VLM is powerful, but it's not quite plug and play.
• 21:33
  You still need to figure out your batch sizes, your KD, cache
• 21:36
  configuration then. So parallelism settings, the
• 21:41
  difficult work for experimentation, but for
• 21:45
  production settings, you need iteration and you need
• 21:49
  expertise. We believe that Kubernetes is
• 21:54
  the right operating system for AI.
• 21:57
  But auto scaling GPU workloads isn't like auto scaling web
• 22:00
  service long. Here you have long called start
• 22:05
  times, you have expensive idle resources.
• 22:09
  You're getting the right scaling processes requires deep domain
• 22:13
  knowledge. The difference between a default
• 22:16
  deployment and the well tune setup can be two, three times in
• 22:21
  effective capacity. Cool.
• 22:26
  So here are my key takeaways. First, every path leads to
• 22:31
  infrastructure. It's not a matter of if I will
• 22:34
  have to handle the infrastructure, but when I will
• 22:37
  have to make these decisions and handle the infrastructure layer.
• 22:43
  Secondly, you need to match your deployment options to your
• 22:48
  workload needs. There's like I said, there's no
• 22:51
  single right answer. It really depends on your your
• 22:56
  latest, your SL OS, your performance, your usage, your
• 23:00
  cost sensitivity and so on. And 3rd when you need to be
• 23:04
  agile, solve your core problem first by or outsource the rest.
• 23:11
  Because if GP orchestration isn't your competitive
• 23:14
  advantage, then it becomes a distraction if your job is to
• 23:19
  build great AI applications and it's not to become
• 23:23
  infrastructure experts, right? And just one more thing before I
• 23:29
  wrap this up and I give the mic to my colleague Igor.
• 23:34
  We talked about the friction of getting started, DevOps
• 23:37
  bottlenecks, GPU, Quora Hill, weeks of waiting before you can
• 23:43
  experiment with open source models.
• 23:45
  So if that resonates with you, we're opening up our inference
• 23:49
  endpoints. There's a feat here.
• 23:51
  No credit card, no commitment. Test the open source model, see
• 23:55
  what performance looks like. Decide if it fits for you.
• 24:00
  We're going to start with a coding use case, allowing you to
• 24:02
  connect cast AI in open code, for example, or any other tool
• 24:07
  that you're using, and execute coding tasks for free.
• 24:11
  We're going to roll out gradually, so there's a wait
• 24:14
  list. Sign up if you want early
• 24:16
  access. I'm going to share the link in a
• 24:18
  bit in our chat and I'm happy to bump anyone here to the front
• 24:25
  all. Right, Joanna, thank you very
• 24:29
  much. Brilliant.
• 24:30
  I'm going to take the screen share off and thank you very
• 24:35
  much for that. That was excellent.
• 24:36
  I think a key take away in my mind is the spectrum component
• 24:43
  that is you're, it's not you shouldn't think about it.
• 24:46
  Is my company going to be self hosting?
• 24:48
  Is it's just about a specific use case.
• 24:51
  Does what exactly are the needs of this specific use case and
• 24:54
  how do we build a flexible and resilient and expansive enough
• 24:59
  system that can incorporate all of the different needs?
• 25:03
  Is that does that take away resonate right, OK.
• 25:06
  Yes, and making sure that you can be flexible enough so that
• 25:10
  you can move between them and you don't lock yourself into an
• 25:13
  expensive option for example. Wonderful.
• 25:16
  OK, Joanna, thank you very much. Igor, Are you ready to jam?
• 25:21
  Always then. OK, so your screen is up.
• 25:25
  I'll be back in 15 minutes too. And we'll take questions both
• 25:29
  for you and for Joanna that come from the audience.
• 25:32
  So folks, put them in the chat and I'll see you soon.
• 25:36
  See you firstly, Adam and Joanna.
• 25:39
  Hi everybody. So OK, the title of the whole
• 25:42
  talk is a bit is a bit weird. We have a 15 minutes for
• 25:46
  somebody, it will be maybe boring for somebody exciting.
• 25:49
  The goal of this 15 minutes is to actually introduce you to how
• 25:54
  you should think about these workloads, how you should think
• 25:57
  about the approaching them and talking with your stakeholders
• 26:02
  or maybe you are the stakeholder, how you should be
• 26:05
  doing actually decisions. All right.
• 26:08
  So after this, I hope you will have questions that will kind of
• 26:14
  go into your mind and you will be thinking about, I never
• 26:17
  actually thought about that. Now I need to think about that
• 26:20
  and that unlocked something for me.
• 26:22
  So I would say that most production means will actually
• 26:27
  come from your profiling and the measuring, the benchmarking that
• 26:32
  you actually do, and it won't really come from these exotic
• 26:35
  techniques. We'll mention the exotic
• 26:37
  techniques that they are needed actually for some things, but
• 26:41
  not for everything. So your regular continuous
• 26:43
  batching or your quantization and just understanding your
• 26:47
  workload and knowing your bottleneck, that will probably
• 26:50
  get you 80% of the way, right? And now it will depend on your
• 26:53
  use case. Do you actually need that
• 26:55
  additional 20%, which is exponentially harder actually.
• 27:00
  So first thing that you would need to think is which type of
• 27:04
  workload do you actually have, right?
• 27:07
  Ask yourself for that question. So here you can see on the Y
• 27:10
  axis, you can see the output tokens.
• 27:13
  On the X axis you can see the input tokens.
• 27:15
  Notice that I don't have the actual numbers written anywhere.
• 27:18
  That's because the numbers or the even the magnitude of the of
• 27:22
  the order, it really depends on the on the company of the
• 27:26
  specific, on the specific client, on the specific use
• 27:29
  case. But generally you need to think
• 27:31
  in the following way. If you have a generative
• 27:33
  workload, what that usually means is you have a bunch of
• 27:38
  output tokens that you need to generate.
• 27:40
  That's very expensive. And you know that simply by
• 27:44
  looking at all these API's that are provided to you because they
• 27:50
  are usually charging more for the output token.
• 27:53
  And we will talk in a bit. Why is that right?
• 27:56
  You have your summarization workload, which is like I give
• 27:59
  you a bunch of stuff to actually process and generate the only
• 28:03
  few tokens. This is super cheap.
• 28:07
  This is like heaven for the GPU's, right?
• 28:10
  For example, your coding use case, when you put a huge
• 28:14
  context into the, when you put actually the huge context into
• 28:17
  the into the into the LM, that's great for the GPU utilization,
• 28:21
  right? The rest of it will actually
• 28:24
  depend on how many tokens you want to generate.
• 28:26
  Do you want to generate the whole file or something a bit
• 28:29
  small? Then you have the chat like
• 28:32
  workload. That's like how it actually how
• 28:35
  actually everything started back in 2023, let's say 2022.
• 28:39
  You know you will have like your 200 tokens or 100 tokens on the
• 28:43
  input. You would be fine with the
• 28:45
  answer that was like 50 to 100 tokens like like today with the
• 28:51
  technology, with the algorithm improvements and the kernels
• 28:54
  that we have, this is like if you have this use case, I'm
• 28:57
  pretty sure you will be able to handle it or we are able to
• 29:01
  currently handle it really, really seamlessly.
• 29:04
  And then you have the average use case like that's, that's
• 29:07
  kind of how the things are looking right now.
• 29:10
  That's like you are not too heavy on the generative side,
• 29:13
  you are not too heavy on the summarization workload, but you
• 29:16
  know, it kind of balances out in between in between South and
• 29:20
  this graph will actually influence all the decisions
• 29:22
  later like this probably the single most important thing to
• 29:25
  understand about your workload, right?
• 29:28
  And one more thing I would like to mention here is that one
• 29:32
  specific deployment it usually does not feel does not fit all
• 29:37
  of the use cases, right. So, you know, if you think about
• 29:40
  deploying one specific, one model to cover all of these,
• 29:44
  depending on your workload type, if you mix everything together,
• 29:50
  it usually doesn't work out. So after you understand your
• 29:54
  workload, you need to think about your metrics.
• 29:57
  I can just split them in three tiers.
• 30:00
  This is not like any kind of specific or regular or
• 30:03
  scientific tie ring, tie ring methodology.
• 30:08
  It's just what I think, how I think about the metrics itself.
• 30:13
  So usually you know, time to 1st open inter token latency or
• 30:17
  maybe end to end, end to end request latency depending on
• 30:21
  your workload ID. Those are the ones that your
• 30:23
  customers or the users of our system actually feel.
• 30:26
  So when you, you know, if you want to summarize 1010 pages
• 30:33
  long Google Docs, right, you actually care about time to 1st
• 30:37
  token. You don't really want to wait 15
• 30:40
  seconds, 20 seconds for the first token.
• 30:43
  And then once it you know, starts streaming, like you'll be
• 30:46
  fine. You can improve on inter token
• 30:48
  latency, but that's, you know, time to 1st token actually keeps
• 30:51
  your customer. If you have some ejectic use
• 30:54
  case, for example, maybe in that case you know agents are talking
• 30:57
  one to another and you don't really care which phase is
• 31:00
  taking how much, but you care in general about the end to end
• 31:05
  request latency, right? You just want your agents to be
• 31:07
  as fast as they can possibly be. Then after you kind of figured
• 31:12
  out what's the, what's OK, what do you expect?
• 31:15
  You will use that in the TH tool.
• 31:17
  So you will balance that with the throughput.
• 31:20
  You will actually check your GPU utilization to see what's
• 31:23
  happening there because you want the GPU utilization high because
• 31:27
  you you will connect that with the tier 3.
• 31:30
  From the Tier 1 you will decide on goodput.
• 31:34
  So goodput is basically it's just kind of umbrella umbrella
• 31:39
  term for multiple metrics in from the tier one, it says like,
• 31:43
  OK, now when I have my time to 1st talk and I have inter token
• 31:46
  latency, I need to define what is acceptable for my customers.
• 31:52
  So you know, for each request, let's say time to 1st token
• 31:55
  needs to be under 500 milliseconds and inter token
• 31:59
  latency needs to be under 50 milliseconds.
• 32:01
  And you define your output like that and you track all the
• 32:04
  requests that are actually under under that under that limit,
• 32:10
  right? And you also need to understand
• 32:12
  your concurrency limit. Concurrency is very important.
• 32:14
  Why? Because you know with this
• 32:15
  generative workloads, once somebody submits a request,
• 32:19
  usually it's not like fire and forget, right?
• 32:22
  It's like you actually receive request, then GPU is actually
• 32:26
  doing some crunching depending on how many, how many work it
• 32:29
  needs to do. So you need to understand how
• 32:32
  many of these requests in parallel you can actually handle
• 32:34
  at any point in time because that will be that will be your
• 32:38
  limit. But all these metrics are kind
• 32:41
  of still also let's say defined by the good put right, so or
• 32:46
  confined by the good food You don't want, you know, you don't
• 32:49
  want throughput of 1000 requests per second if your goodput is
• 32:53
  very bad. And then in the end, of course,
• 32:56
  you need to look at the cost and the utilisation.
• 32:59
  You want to bring your utilisation of the GPU to the
• 33:03
  maximum and you want that cost per token to be as less as
• 33:06
  possible. And we will talk how to how to,
• 33:09
  how to reach that. So when you're talking with your
• 33:12
  state stakeholders about, about this, actually you need to tell
• 33:17
  them that, you know, they can have it all.
• 33:21
  You can have low cost, you can have high throughput and low
• 33:24
  latency at the same time. You can imagine it like a here
• 33:28
  in this graph, I'm kind of missing maybe let's say accuracy
• 33:30
  standpoint, It's not that big of a deal.
• 33:33
  But when you include quantization and methods such as
• 33:37
  quantization like accuracy, it's kind of important.
• 33:41
  But imagine it's like a bowl that you are kind of stretching
• 33:44
  between these three topics, right?
• 33:46
  So here I have just simple examples.
• 33:49
  Please ask questions so we can actually answer on some
• 33:52
  specifics later. But for example, if you want
• 33:55
  bigger batches, right? In this case, your latency will
• 33:58
  actually become worse, but your throughput will become better
• 34:02
  because now you will patch more requests together, you will have
• 34:07
  a higher GPU utilization and your cost because of that will
• 34:10
  go down, right? If you just add more GPUs,
• 34:14
  obviously you have more hardware, so your throughput
• 34:18
  will be better even if you do nothing.
• 34:20
  Probably your latency will also become better just for the sake
• 34:24
  of it, because you have a bigger system.
• 34:25
  But now we are paying for the more GPUs, right?
• 34:28
  Quantization is a tricky one. Like here I have everything like
• 34:31
  everything is better. It's not really true.
• 34:34
  It really depends on the quantization type and on the
• 34:36
  hardware that you use and on the kernels that you run.
• 34:41
  But we can talk about it later. So I'll give you one example.
• 34:44
  Usually when you do quantization now, for example, you can do
• 34:47
  bits and bytes quantization, right?
• 34:50
  You will have, let's say 4 bit, you will have 4 bit weights and
• 34:56
  16 bit activations, right, or something like that.
• 34:59
  And then what happens is your model is actually smaller on the
• 35:02
  memory side. It means that you know you can
• 35:05
  actually move, you need to move less memory on your GPU when you
• 35:09
  are doing differencing, which is good and now you can fit it in
• 35:13
  the smaller GPU. But when you are actually doing
• 35:15
  computation right, you have this process of de quantization.
• 35:20
  So from the four bits you need to go back to the 16 bits, you
• 35:23
  do your computation in the 16 bits and then you go back to,
• 35:28
  then you go back to the four bits again.
• 35:31
  So that part is still equally compute having.
• 35:35
  So there is no like compute optimization there, right?
• 35:40
  So it's a, it really depends on the combination of the hardware
• 35:44
  and the technique. And of course you can just go
• 35:47
  like for example here with the cheaper GPU from A-100 to L4,
• 35:51
  you will have less high bandwidth memory.
• 35:55
  So obviously the cost, the cost is going down because the GPU
• 35:59
  itself is cheaper, throughput is worse.
• 36:03
  Well, obviously, and now I notice that my actually latency
• 36:07
  is in green, but it says worse, so it should be for us, right?
• 36:12
  So here you have two Reds, 1 green, let's say.
• 36:17
  So you need to think about the trade-offs.
• 36:19
  You need to connect them with the metrics and the type of the
• 36:22
  workload that you have. This is what you need to discuss
• 36:26
  with the stakeholders. And then after that you actually
• 36:31
  can move forward, right? So when you are thinking about
• 36:35
  starting to serve an LM, some options K will come naturally to
• 36:40
  you when you are actually trying to, you know, research the stuff
• 36:43
  either using perplexity or open AI or should GPT write whatever
• 36:48
  you're actually using these days to research code.
• 36:52
  VLM is the like all all three of these are inference engines.
• 36:57
  Inference engines are just like a specific piece of software
• 37:01
  that's written to efficiently execute forward pass in the in
• 37:06
  the neural network, right? In this case, Transformers
• 37:09
  specifically. So VLM is the most popular out
• 37:13
  of them all. It probably has the biggest
• 37:16
  community around it. It's probably the easiest to to
• 37:20
  start with in terms of documentation, but it has it's
• 37:23
  new answers, right? And we will, we will, we will
• 37:26
  see actually those later. This reality is a bit harder.
• 37:29
  One that's from the NVIDIA. It's actually has a specific
• 37:34
  format that you need to kind of transfer your model into, and
• 37:37
  then you use the specific format to actually do the inference.
• 37:41
  It's super optimized for the NVIDIA GPUs, so you can
• 37:46
  sometimes get better performance than using, for example, VL.
• 37:50
  But there is also an option where you can like end up in
• 37:53
  test RT you can plug into VLM as a back end.
• 37:57
  So one more option there as you link, it's, it was neck to neck
• 38:04
  at the beginning with the VLM, it still has its own, let's say
• 38:09
  advantages. What I would say is it depends
• 38:12
  on the use case. As you link, for example, excels
• 38:16
  in the use case where you have like a long session, you
• 38:19
  actually have a context that you are tracking something like, you
• 38:23
  know, something like a long chat session that's that's
• 38:28
  favourable, favourable for the SDLAC.
• 38:32
  But these these are not enough. And today you have multiple
• 38:37
  projects which are actually doing some high level
• 38:42
  orchestration on top of the inference.
• 38:44
  So for example, AI bricks from the I think it's from the Pi 10s
• 38:49
  or the I think it's their project actually now it's the
• 38:52
  under the VLM umbrella. It does the IT does the geneal
• 38:56
  orchestration Dynamo from the NVIDIA LMD is again tied to the
• 39:02
  VLM community, But I think it's a redhead project.
• 39:07
  Don't don't take me for my word, Caser.
• 39:10
  That one was that one existed even before the LMS and the LM
• 39:15
  cash, which is LM cash is a bit different than all of the above.
• 39:19
  All of the above are actually doing some kind of the
• 39:22
  orchestration for the inference engines and LM cache is actually
• 39:28
  expanding, expanding your cache to a cheaper storage, right?
• 39:33
  But this is also a crucial part. So let's let's talk why these
• 39:37
  why these projects actually exist and why do we need them?
• 39:41
  So there is a concept in the inference which is called
• 39:46
  prefill, which is called prefill and decode, right?
• 39:49
  So this is how we actually split one request.
• 39:52
  We split it in two phases. One phase is a prefilled phase,
• 39:56
  the other one is a deco phase. So the prefilled phase is
• 39:59
  compute happy. That means that once you
• 40:02
  actually send your data to do a forward pass, so all the stuff
• 40:06
  that you actually that you consider your context strength
• 40:09
  before you generate the first token, that's the text.
• 40:15
  That kind of fills in that prefer phase, right?
• 40:18
  And the GPU to actually work on that, it needs to do quite,
• 40:24
  quite number chunking. But once you generate the first
• 40:30
  token and you need to generate every other token after that,
• 40:33
  right? Because it's auto regressive,
• 40:34
  you need to have a full context before that you have this decode
• 40:37
  phase, which is super slow. Decode phase is super slow and
• 40:43
  doesn't fully utilize your GPU in terms of the compute because
• 40:47
  what is happening usually is it's memory bound, right?
• 40:51
  So on this graph here, this is from the paper Diskserv, which
• 40:57
  kind of explain the disaggregated serving.
• 40:59
  And disaggregated serving I would say falls in that
• 41:03
  additional 20%, right? So when I mentioned at the
• 41:06
  beginning like your regular stuff will take you 80% of the
• 41:10
  way, this would be that 20% depending on the size of your
• 41:15
  workloads and also of course the hardware that you have
• 41:20
  available. So here you can see on the left
• 41:23
  you can see the graph. On this graph we have the input
• 41:27
  length of 128 tokens and you have a bit size.
• 41:31
  So what is compared here is multiple requests batched
• 41:36
  together. The blue line represents the
• 41:40
  requests that are based together, but they have one
• 41:44
  prefill request. So you have like you have all
• 41:48
  the decoding ones and then you have one prefill there and you
• 41:52
  have only the decode here, right?
• 41:54
  So you can see, you can see what's what's happening on the
• 41:58
  graph. You can see how is it slowing
• 41:59
  down And the difference even gets bigger when you actually
• 42:03
  go, when you actually go, when the to increase actually the
• 42:07
  input thing. So what is the solution for
• 42:10
  that? Basically the solution is
• 42:12
  desegregated serving. Desegregated serving basically
• 42:16
  means, OK, we know we have prefill, we know we have decode.
• 42:20
  What we need to do, we need to split them apart.
• 42:22
  So what we will do is we will optimize.
• 42:27
  We will basically deploy, let's say if we take VLM, we'll deploy
• 42:30
  1 instance of the VLM on a specific node.
• 42:34
  For that node, we will select the hardware that fits the user
• 42:38
  case perfectly. We will benchmark that part and
• 42:39
  we will just send a prefill request there.
• 42:42
  Then once the prefill is done on that node, we will copy
• 42:45
  basically the key to cache. We will move it to the decode
• 42:49
  node. Decode node can have a different
• 42:51
  type again of the GPU that's specialized, for example,
• 42:54
  exactly for decoding. And there you specialize just to
• 42:58
  basically decrease inter token latency, right?
• 43:00
  So your pre fail instance is specializing or optimizing with
• 43:04
  the time to 1st token. Your decoding instance is
• 43:06
  optimizing on the inter token latency.
• 43:09
  And the beautiful thing is that you can actually you can
• 43:14
  actually optimize these two things in isolation, basically,
• 43:17
  right? So you can choose a different
• 43:19
  GPU's, the different node, etcetera.
• 43:22
  Be careful with this because if you just try to deploy something
• 43:25
  like that and you don't really have enough of the traffic for
• 43:28
  this or you don't really have hardware for it, it can just you
• 43:33
  know, you can shoot yourself in the full space.
• 43:36
  Yeah, because you can see like performance degradation of even
• 43:39
  30% depending on on what's your current use case.
• 43:44
  So don't forget about the hardware, like if I want to like
• 43:49
  we are talking about different, different engines, they're
• 43:51
  running with something, right? And that something is harder
• 43:54
  today. We all know like NVIDIA
• 43:56
  currently has supremacy. So look at the specs of your
• 44:00
  hardware. Understand what does it mean
• 44:02
  when you have like integer 8 tensor cores, integer 4 tensor
• 44:05
  cores, what your model actually requires, right?
• 44:09
  So you in the in the huggy face config file, open that config
• 44:14
  file, look at the model. Look what's the quantization
• 44:17
  used on the model? Is there any what cores do you
• 44:21
  actually need? What's the size of the model?
• 44:23
  Is it mixture of express like all that, all that counts.
• 44:27
  Actually look at that stuff, understand your GPU, understand
• 44:31
  how fast it can move the memory back and forth.
• 44:33
  What's your benefit? How many flops can you actually
• 44:37
  do? And then, you know, that's just
• 44:41
  like the pre selection so that you know that you chose the
• 44:44
  right GPU for your workload. You still need to benchmark it
• 44:48
  after because usually no, no amount of simulation or kind of
• 44:54
  theoretical framework will get you exactly to the numbers that
• 44:58
  you will have in production. You know, because we cannot
• 45:01
  account for, I don't know, networking latency in some cases
• 45:07
  or how the physical boxes are actually set up etcetera.
• 45:11
  Attention, right? Understand the attention of your
• 45:14
  model, understand what's happening there.
• 45:16
  Here I have like few attention types.
• 45:20
  So multi head, group query and multi query attention.
• 45:23
  I won't go too deeply. The point is here, the queries
• 45:28
  what is incoming to your model, these are the blue boxes, right?
• 45:32
  The keys and values, these yellow and red boxes are what
• 45:36
  are you saving in your Kiwi cache, right?
• 45:38
  So we can see that based on the different attention, you are
• 45:42
  saving different amounts of these blocks.
• 45:45
  Which means that depending on whatever attention is in your
• 45:49
  model, it will use a specific amount of QV cache for let's say
• 45:55
  1 full context length request. You need to understand that you
• 45:59
  need to calculate that number. You need to see the trade off
• 46:02
  like how many, how much QV cache you actually need to have a
• 46:07
  successful inference. And we can we can talk later.
• 46:10
  What's that number and how how you how can you decide on it?
• 46:16
  What we learned also so far from serving these these models, I
• 46:19
  want to share this with you. Quantization really do work.
• 46:23
  So here this is this is something to do with the
• 46:27
  quantization per SE. This is just like from different
• 46:31
  memory cards from NVIDIA, but you can see throughout the
• 46:34
  throughout the years, NVIDIA is shipping more and more
• 46:38
  specialised hardware for a specific for a specific
• 46:42
  operations. For example, a 100.
• 46:44
  They have like an integer 18 score right or FB 16.
• 46:48
  But if you go to the newest one, it will have 14.4.
• 46:52
  So they optimize for the quantity.
• 46:55
  They kind of enable you to run 4 bit quantization even faster
• 46:58
  right? The Hopper architecture, the
• 47:01
  H-100, like they only had it only had FB8 and the at the end.
• 47:09
  I would also want to mention, I think I'm a bit over my time,
• 47:12
  but this is the last type. So again, I would I would also
• 47:16
  want to mention that kernels are probably the most important
• 47:19
  piece. So when you understand all of
• 47:22
  the pieces above, let's say you take your quantization, right?
• 47:25
  And you take the, I don't know, you take your, you take your
• 47:31
  quantization, and then you quantize the model.
• 47:34
  Now you run it and there's not much improvement.
• 47:36
  Why? Well, simply because it's not
• 47:40
  executed efficiently on the hardware.
• 47:42
  And these things at some inferences happen immediately,
• 47:46
  right? They have a way to figure it out
• 47:48
  through the config of the model. They understand for the specific
• 47:51
  model how to run it. But usually, in most cases, you
• 47:54
  can actually select it by hand manually.
• 47:57
  What do you want to run? So for example, I know Marlin is
• 48:01
  a really optimized kernel that fuses some of the operations
• 48:06
  together for the four bit rates and 16 bit activations.
• 48:12
  It gives you like a huge increase in throughput when you
• 48:16
  compare it with with the baseline, right.
• 48:18
  So don't you know, don't just ignore these things.
• 48:21
  You kind of want to go deep on on it to, to figure out your,
• 48:26
  your deployment. And thanks.
• 48:29
  That's that's everything from my side for now.
• 48:32
  I hope I unlock those questions for you, Adam.
• 48:35
  I'm I'm ready for the chat. Nice, Igor, thank you very much.
• 48:41
  That was excellent. And as I suspect that I feel
• 48:45
  like I could keep you here for many, many hours and you want to
• 48:49
  suggest it as much. So OK, Well as I let questions
• 48:55
  come into the chat, maybe I can take a stab at a couple of them.
• 49:00
  So one take away that I had from the last couple of slides that
• 49:06
  you shared is people shouldn't be scared to just prop open the
• 49:13
  model, understand what's happening under the hood.
• 49:17
  Look at how attention is built and architected.
• 49:20
  Look at the sizes, look at the D type, look at all of the stuff.
• 49:24
  And then make informed decisions about what infrastructure, what
• 49:29
  hardware to use, what compression techniques, what
• 49:32
  different, different things you can do to further optimize
• 49:36
  serving. First of all, is that true?
• 49:40
  Is that the right take away? And second of all, a lot of
• 49:43
  people, if they come into this through playing with the model
• 49:47
  AP, is they might not feel the kind of confidence that they
• 49:52
  could do this properly, right? Like maybe people are, are
• 49:55
  application engineers. They're yeah, sure they can.
• 49:57
  They, they play around with the, with the open AIAPIS.
• 50:01
  But now you want them to understand KV cash?
• 50:04
  How high of a jump is that? And what does it take for them
• 50:10
  to overcome what might feel like a lack of expertise?
• 50:17
  Well, you know, for that of them, I don't have like a super
• 50:20
  great answer. The time is probably your best
• 50:23
  friend. Like if you have time and you
• 50:24
  can just dig deep, do that, right?
• 50:27
  But have in mind, depending on your project, you cannot just go
• 50:32
  running around and you know, trying to learn every new thing
• 50:36
  and then you know, hope it works because you will miss something
• 50:39
  along the way. So do it in stages, right?
• 50:43
  Even if you go to go to decide with company like ours to
• 50:46
  actually do some stuff to serving, right?
• 50:49
  In the meantime, like we are actually talking to you, we want
• 50:53
  you to understand your workloads.
• 50:55
  You will learn things at some point, you will be free to self
• 50:58
  host, right? So it's a, it's a process.
• 51:00
  It doesn't happen overnight. Right.
• 51:02
  Yeah, exactly. That's why we started with the
• 51:05
  self hosting option. We talked to a lot of users and
• 51:10
  then we realized, oh, OK, wait, there's a big barrier here that
• 51:13
  we need to break 1st and then work our way up to to this level
• 51:19
  of complexity. Yeah, I'll give.
• 51:23
  I'm sorry, Adam, I'll give you one more regarding this.
• 51:25
  Like I think the problem with the companies in in the space is
• 51:30
  like you need to fight not to become consultancy because
• 51:33
  people have so many questions and it's like it's not the great
• 51:37
  business model. Well, you could see that I'm
• 51:40
  trying to drag you into consulting mode right now
• 51:43
  because, well, when I started honestly in, in data
• 51:48
  engineering, I, that was my strategy because I knew very
• 51:51
  little when I started. But there was a lot of startups
• 51:53
  that were building a lot of very cool tools.
• 51:55
  And so I would just ask them a million questions, you know,
• 51:59
  like, how do you think I should order, you know, because
• 52:01
  otherwise there's very little access to learning so much of
• 52:05
  this stuff. And people that are building
• 52:06
  tools, they often have expertise and they can share that
• 52:09
  expertise because they want to get their tool across.
• 52:10
  And so it was like a win win. Although it's at some point you
• 52:13
  end up getting a lot of books and stuff and that's fine.
• 52:15
  Do you feel like that what you guys are building at Cast AI is
• 52:21
  almost going to be a substitute for that type of expertise or it
• 52:26
  will only it will simply enable people that already have that
• 52:29
  expertise to do more and get more bang for their buck?
• 52:34
  You know, I think it's both, right?
• 52:38
  It's usually you can sit on 2 stools, but in this case, it
• 52:43
  seems like it's both because, you know, for power users, you
• 52:48
  just help them get rid of their problems.
• 52:52
  I, I don't want to, you know, manage these things.
• 52:54
  I know how they work, but I want to spend my time actually
• 52:58
  designing my use case, designing my application, making things
• 53:02
  that, you know, agentic flow actually works and delivers
• 53:05
  results. And then at some point I can
• 53:07
  think of, you know, moving away freely, right?
• 53:12
  In that, in that whole process, as I said, you are usually not
• 53:16
  alone, especially if you go with the route that we serve it.
• 53:20
  For example, in your VPC, you have that you almost have it
• 53:24
  like it's almost like you have a team next to you in your company
• 53:29
  that actually does that for you. And you can always go there and
• 53:31
  ask, right. So it's it really depends on on
• 53:35
  you and how much you actually want to learn.
• 53:38
  Yeah. Honestly, between me and Igor, I
• 53:40
  think both of us have been involved in almost every
• 53:44
  customer conversation so far. So if you reach out to us, these
• 53:49
  are the first phases that you're going to see and you're going to
• 53:51
  talk to. Because I think it's important
• 53:53
  to understand where our customers are coming from,
• 53:56
  what's their level of expertise so that ultimately we can build
• 54:00
  a product for them to self-serve.
• 54:02
  We're not there yet in totality because we still see different
• 54:07
  use cases. Like I said, it's a spectrum.
• 54:10
  We're trying to identify the patterns.
• 54:11
  And then, you know, our job as a product team is to build a
• 54:14
  product to, to solve that problem, to productionize this.
• 54:19
  Nice. I'm OK.
• 54:20
  We have another couple of questions, maybe two or three
• 54:23
  questions in the chat and I will attempt to those just real quick
• 54:29
  if one of you might have the gradual tab open, make sure that
• 54:34
  it's because I hear, I do hear some echo, so it might be coming
• 54:39
  from from that. OK, so first we have Samid Ahmed
• 54:44
  Khan is asking. I understand that there are
• 54:46
  different quantization techniques that work
• 54:47
  differently. But if we talk about the W4A16
• 54:51
  model, that model has to go back to 16 bit weights.
• 54:56
  Won't be computation takes similar compute power as W16A16.
• 55:00
  What benefit would it give except storage?
• 55:02
  I mean, the, the storage was the first reason why you would
• 55:06
  actually do the quantization like that, right?
• 55:09
  Like now it's easy to think like the Ruben I think is coming out
• 55:12
  or it actually came out. So it has like a TB of high band
• 55:16
  memory. But two years ago, three years
• 55:18
  ago when these methods were actually created, your A-100 was
• 55:22
  the was the best choice, right? And for these bigger 400 billion
• 55:27
  models at the time or more, you're actually the the memory
• 55:31
  is a big problem, right? So you want to reduce the memory
• 55:35
  because there's one more thing. Remember that the deco phase.
• 55:38
  In the deco phase, what you are doing, you need to move that
• 55:41
  model from the high bended, high bend memory to actually to let's
• 55:46
  say much more speedier the L1 cache or the memory, that memory
• 55:51
  that's really close to your computation, right?
• 55:54
  And depends on the, depending on the NVIDIA hardware like it's
• 55:57
  called differently, but you are moving those bits every time in
• 56:03
  the deco phase. And for each deco phase, for
• 56:05
  every new token, you do it all over again, right?
• 56:08
  So there is, there is, that's your, that's the crux, that's
• 56:14
  the goal. But the computation, the
• 56:16
  computation itself, if you have activations in the 16 bits, you
• 56:21
  cannot multiply 16 bits by 4 bits, right?
• 56:24
  You the precision and the output that you would get after that,
• 56:29
  it wouldn't be stable, you would get gibberish.
• 56:32
  So it destroys your like yeah, I I hope I explained it well.
• 56:39
  Samir, if you have a follow up, put that in the chat.
• 56:42
  OK, so Siddharth Thapar Igor shared some inference
• 56:47
  optimization ideas. How do those apply to
• 56:49
  specialized LMS like large embedding or ranking models?
• 56:53
  Any additional ideas that apply to such models?
• 56:55
  Which of these techniques are useful?
• 56:57
  Also, which ones to apply to training optimization as well?
• 57:03
  Yeah. So let's say, let's take an
• 57:06
  embedding, right, Embedding model or RE Ranker, right?
• 57:09
  It all depends what's your architecture of that model.
• 57:13
  So if you have if you have some, some encoder in there, right, if
• 57:18
  you have a tension mechanism in there, like these ideas still
• 57:21
  apply to that. It's the model is called
• 57:25
  differently because usually the last few layers are different or
• 57:28
  like, you know, bits of the architecture actually serves a
• 57:34
  different purpose. But the crux of it in the
• 57:36
  transformer, like it's always the same.
• 57:38
  So you can you can reuse this concept across across different,
• 57:43
  of course different models. We got, we got 2 more minutes
• 57:46
  unless folks in the audience have another question, I want to
• 57:51
  ask them two, one for them and one for me.
• 57:53
  OK, so I'll start with a question for them.
• 57:57
  Do you, have you seen some blind spots that people have when
• 58:03
  right now they're still using model APIs, but they're doing
• 58:07
  something about their system, something about their
• 58:09
  architecture, something about the way that they're envisioning
• 58:13
  interacting with these APIs that ends up making it even more
• 58:18
  difficult for them when they make that transition to self
• 58:22
  hosting. Anything they could do today,
• 58:27
  insofar as they're not yet self hosting, What can they do today
• 58:30
  so that that transition can be made even more easy?
• 58:34
  Is that for me or for the audience?
• 58:37
  I'm asking on behalf of them. Listen, in that case, I would
• 58:42
  say pay more attention to your users, right?
• 58:46
  So how much of the tokens you're actually doing on input, how
• 58:50
  much on the output? What's your usual, let's say
• 58:53
  session size or the context? Like all these things are
• 58:56
  important. Like keep track of that because
• 58:58
  you're taking for granted what some platform is actually doing
• 59:02
  it for you. And when you will try to go self
• 59:05
  hosting, you will see like it's not that easy right?
• 59:09
  Server set use case, 200 tokens in, 200 tokens out.
• 59:12
  Great, superb works on one request.
• 59:15
  Try to scale it out. Wonderful.
• 59:18
  From from my side, I would say it's set your expectations
• 59:22
  right. When you're transitioning from a
• 59:24
  large 400 plus billion model to a 20 billion model, set your
• 59:29
  expectations right. Don't expect the same level of
• 59:33
  accuracy right off the box. You're going to have to put a
• 59:35
  bit of work to, you know, compare.
• 59:40
  The results nice. Thank you.
• 59:42
  OK, I want to steal. Can I steal a couple of more
• 59:44
  minutes just for final? OK, so one from Muhammad
• 59:51
  Abdallah, how do you know when serving many requests using in
• 59:55
  my app that I'm moving from Tier 1 to Tier 2 and so on,
• 59:59
  considering the latency of responses, so.
• 1:00:04
  I wonder if the Tier 1, Tier 2 is referring and Muhammad,
• 1:00:07
  please clarify. I I suspect you, you came up
• 1:00:11
  with these different metric categories, right?
• 1:00:15
  But and then these are you said almost like you could start by
• 1:00:18
  paying attention to the first one, second one, because the
• 1:00:20
  first one was closer to the user.
• 1:00:24
  Is there some phase transition by which you should now start to
• 1:00:28
  pay attention to to the OR should it always be like a
• 1:00:31
  continuous process? So ideally continuous process
• 1:00:34
  solids, right? The tears are there to kind of
• 1:00:37
  what's the first thing you look at, right?
• 1:00:39
  So the first thing is that you look at is I, this is my
• 1:00:43
  workload. I want to improve my time to 1st
• 1:00:46
  talk, right? But at some point, because you
• 1:00:48
  will be constrained by your hardware or what you have, you
• 1:00:52
  need to see big. I'll give you an example, right?
• 1:00:55
  You take an A-100 on a 100, you put some smaller model, let's
• 1:00:59
  say like 7 billion, right? And you can get 1000 requests
• 1:01:03
  per second. Like, OK, thousand requests per
• 1:01:06
  second is a silly number, but let's say 50 requests per second
• 1:01:10
  and you can maintain that. Great, right?
• 1:01:13
  But maybe with the 50 requests per second, your time to 1st
• 1:01:18
  token is like 600 milliseconds. And you want to kind of to go
• 1:01:22
  down a bit, you really cannot serve 50 requests per second At
• 1:01:26
  that point. You need to find the lower
• 1:01:28
  number that saturated GPU that kind of makes happy both your
• 1:01:33
  hardware and your customer, right?
• 1:01:36
  This is the, this is the trade of the need to that you need to
• 1:01:39
  look at. So I would say when starting,
• 1:01:42
  simply try to use the product yourself, see what's usable for
• 1:01:47
  you. And based on that or based on
• 1:01:50
  the feedback of customers, define what are the numbers that
• 1:01:54
  you need to see on the output in terms of like latency, right?
• 1:01:58
  And then for that latency, try to find the other numbers, try
• 1:02:01
  to find the concurrency, try to find the throughput that you can
• 1:02:04
  sustain on some hardware or budget that you have.
• 1:02:11
  Yeah, you wanna. Do you want to add to that?
• 1:02:13
  Is that how you've seen people kind of like interface with the
• 1:02:17
  problem? Yeah, I would say the first
• 1:02:19
  thing that we ask and we try to define whenever we have a
• 1:02:22
  customer discovery session talking about their customer
• 1:02:26
  problems, what they want to do and setting up their their
• 1:02:30
  models is define the SL OS like what do you actually need,
• 1:02:34
  right? And then all the decisions they
• 1:02:37
  stem from that, right. So make sure that you understand
• 1:02:41
  your SL OS, you connect it with the metrics like you were said,
• 1:02:44
  and then you adjust the infra and kind of automate based on
• 1:02:48
  that. But everything starts with the
• 1:02:49
  SL OS. And if you have, if you can come
• 1:02:51
  to us with a good put definition, like what's what
• 1:02:55
  does good put mean for you? We're going to become best
• 1:02:58
  friends. Yeah, Yeah.
• 1:03:01
  I suspect that this is again an invitation to understand what
• 1:03:05
  our users and customers need. And there's really no, no amount
• 1:03:10
  of AI that can do some hocus pocus unless you understand what
• 1:03:15
  your users and customers need. And you'll just continue to be
• 1:03:19
  LED astray by fancy technology and science and engineering
• 1:03:24
  unless you understand that. So I think that's a good take
• 1:03:27
  away. And maybe last one here, Nathan
• 1:03:32
  is asking in so far as anybody is interested in going deeper
• 1:03:38
  into serving and inference, is there some either, you know,
• 1:03:43
  let's make it more interesting. What do you imagine the future
• 1:03:46
  of serving an inference looking like?
• 1:03:49
  Are there specific areas that you think are likely to to grow
• 1:03:55
  and become more mature and more interesting?
• 1:03:57
  Like where if anyone were to want to do research, let's say
• 1:04:00
  into any one of these domains in serving, where would you Orient
• 1:04:03
  them to? Well, for now, I think the one
• 1:04:06
  of the most exciting parts are mixture of experts and like the
• 1:04:11
  models are getting sparser and sparser.
• 1:04:14
  So you have less and less actually experts activating on
• 1:04:18
  each request, right? To improve your to improve your
• 1:04:21
  inference, there's one part. And the second part is you have
• 1:04:24
  so many companies trying to tackle at this point in time
• 1:04:28
  kernel optimizations, right? So you have all these different
• 1:04:32
  projects that are kind of, hey, we will compile or we will
• 1:04:36
  actually generate the best code for you to run with this
• 1:04:40
  specific high performance chip, right?
• 1:04:43
  That's the open field. So many competitors there.
• 1:04:46
  And I think that field is maturing.
• 1:04:48
  I hope it will mature soon because it will make our lives
• 1:04:52
  as like machine learning engineers, much easier.
• 1:04:56
  Awesome, nice. If anybody wants to follow up
• 1:04:59
  with you guys and get to know you more and connect, are you on
• 1:05:03
  LinkedIn, Twitter? Where are you?
• 1:05:06
  Where can. People find you in your.
• 1:05:07
  LinkedIn. I'm hiding I'm on your LinkedIn.
• 1:05:12
  Nice, if you want to drop your LinkedIn below, that would be
• 1:05:15
  cool too and you can put it in the chat.
• 1:05:18
  Joanna and Igor, thank you very much for coming.
• 1:05:20
  This was a true. Education.
• 1:05:25
  Thank you very much. Thank you.
• 1:05:28
  And everybody else, thank you for for tuning in.
• 1:05:31
  We made people asked whether the slides will be available offline
• 1:05:34
  after the fact. If you're saying yes then that,
• 1:05:37
  then that's the answer. Yeah, why not?
• 1:05:39
  OK. So that's the answer for you,
• 1:05:42
  Joanna. Igor, thank you very much and
• 1:05:44
  best of luck with all of your work and everybody else.
• 1:05:46
  Thank you for tuning in and we'll see you next time.
• 1:05:49
  Thank you. Thank you.
• 1:05:52
  Bye bye.