Latest / Agentic Conversations (formally mlops.community) / MLOps vs. LLMOps Panel // LLMs in Conference in Production Conference Part II

Transcript

MLOps vs. LLMOps Panel // LLMs in Conference in Production Conference Part II · Agentic Conversations (formally mlops.community)

• 0:04
  But without further ado, I think it's time for a little panel
• 0:08
  reach out what's happening. I'm hoping my phone doesn't ring
• 0:13
  and say seek shelter because we're in the middle of a Thunder
• 0:16
  and tornado storm in the greater Philadelphia area.
• 0:18
  So fingers crossed. If it does then you get you stay
• 0:25
  safe. I'll jump on and and play your
• 0:28
  part for you. So then we'll also bring on poo
• 0:33
  poo poo poo. Where?
• 0:34
  Who else do we have here? Mr William?
• 0:37
  What's up dude? And Chris.
• 0:40
  Ohh this is a great panel. Where's Aparna?
• 0:44
  Yes. And last but not least we've got
• 0:47
  Alex. But Alex calling Alex to the
• 0:50
  stage. Where is he?
• 0:51
  There he is. Alright, so there's a tonne of
• 0:55
  us. I'm gonna hop off, reach.
• 0:57
  I'm gonna hand over the talking stick to you and we'll get it
• 1:01
  started. Alright, thanks Ringo.
• 1:05
  Welcome panel. This is take a quick second to
• 1:08
  go around the room, introduce ourselves and then we'll dive
• 1:11
  right into the questions. I'll kick it off.
• 1:13
  My name is Richard Sachdeva, I work for JP Morgan Chase and I'm
• 1:17
  in the Chase Rewards department and lead a team of data
• 1:20
  engineering and operations and we work on the business side to
• 1:24
  help them make data-driven decisions.
• 1:26
  I'll pass it over to you, Aparna.
• 1:29
  Awesome. Hey everyone, good to get to see
• 1:32
  you all. Excited to be on this panel.
• 1:34
  My name is Aparna, founder of Arise I Rises, ML observability
• 1:39
  and I guess LM observability would be the topic of of today.
• 1:43
  So excited to you know have have a great combo with all of you.
• 1:49
  Wonderful. Thanks, Aparna.
• 1:50
  Over to you, Chris. Hey everyone, I'm Chris Van
• 1:54
  Pelt, co-founder of Weights and Biases.
• 1:58
  I'm repping the brand today. Our mission is to build
• 2:02
  developer tools for Machine Learning or LM Prompt
• 2:07
  Engineering Engineers. Welcome, Chris.
• 2:12
  Alex. Hey everyone I'm Alex, I'm one
• 2:16
  of the Co founders and CEO at Snorkel.
• 2:17
  I'm also affiliates and assistant professor at
• 2:20
  University of Washington and before that I was working in
• 2:23
  historical project at Stanford. All things what we call data
• 2:25
  centric guy. So labelling but also curating,
• 2:28
  sampling, filtering, cleaning all the data that goes into
• 2:31
  either training models from scratch or fine tuning, pre
• 2:34
  training, instruction tuning large language models or
• 2:38
  foundation models as we like to call them.
• 2:40
  So excited for this this chat and reaches.
• 2:42
  Thanks for hosting. The course and this panel would
• 2:45
  not be complete without William. William.
• 2:48
  Next. Hey folks, yeah my name is
• 2:51
  William's been in the dead end AI space for about, uh you know,
• 2:54
  6-8 years now. Already leading ML platform
• 2:58
  teams, more info teams, and building in the open source
• 3:00
  space. Recently most focused on feature
• 3:03
  stores but really sinking my teeth into the LLM space in the
• 3:07
  last couple of months. Well, thank you all for being
• 3:10
  here. This is definitely a great panel
• 3:12
  and I'm excited for all of the learnings and well and nice to
• 3:15
  see you again. So I'll kick it off with you.
• 3:17
  Could you walk us through what do you think in your experience
• 3:21
  with the the high level differences between LM OPS and
• 3:25
  ML OPS? And it's quite a tongue, tongue
• 3:27
  twister, LM OPS I I always make sure I have the right number of
• 3:31
  elves. Yeah, I feel like this question
• 3:35
  we can probably expand and talk about for 30 minutes to an hour.
• 3:40
  So I'll try and keep it as brief as possible, but.
• 3:44
  I think a big part of it for me is abstraction and
• 3:47
  generalisation. Um, to me LM OPS is.
• 3:53
  Um, language oriented, right. And so there's a big question
• 3:57
  around how you frame your use case as a builder to what now
• 4:01
  mom can do, because it's so generalised.
• 4:04
  But I think we've seen a democratising effect of these
• 4:09
  alms bringing in new folks that couldn't previously.
• 4:16
  Traditionally most stacks and solutions, but I think the key
• 4:21
  difference for me is the offline flow, right?
• 4:24
  That's an area that has become somewhat optional right now.
• 4:28
  And I think for application builders that has enabled new
• 4:32
  use cases to be unlocked and for folks that weren't in the space
• 4:36
  to suddenly build right, you can just start with the model in
• 4:40
  production frame your use case. Around an Elm and and get that
• 4:46
  in front of users and get traffic onto that model and then
• 4:50
  figure out the next steps and the offline flow where there's
• 4:54
  debt collection, whether it's training.
• 4:57
  That that whole step has become a secondary step as opposed to
• 5:00
  the first step. And I think that's, that's a key
• 5:03
  thing and I'll kind of leave it there and with the rest of the
• 5:05
  panel kind of expand on that. Yeah.
• 5:09
  Thanks, William. Anybody wants to?
• 5:13
  Add on to what William just said.
• 5:16
  Yeah, I mean I think one's point was was a really good one.
• 5:19
  You know when we think about data scientists and ML engineers
• 5:23
  and what the workflow has been like for called statistical AI,
• 5:28
  traditional AI. There's there's a lot of names
• 5:30
  and I'm hearing floating for you know, non LM type of models.
• 5:36
  But but what happens is a big chunk of that work is in the
• 5:40
  training of of the model. And I think that, you know, when
• 5:45
  we think about how do you go improve the outcomes, you think
• 5:48
  about, well, I'm going to go and train it to to get X percent
• 5:52
  increase in performance. That's become a fundamental
• 5:56
  shift in this new llama's world is that there's so much you can
• 6:02
  do with prompt engineering. And actually, I think this is
• 6:07
  kind of what William's point was, which is in the, you know,
• 6:10
  the offline world. Might not look like training, It
• 6:14
  might not look like fine tuning. It might just be iterating on
• 6:18
  what is the prompt, what is the context that I might want to
• 6:21
  incorporate into the prompt, and it's a very fundamentally
• 6:25
  different. Different workflow and different
• 6:28
  way of thinking about improvement than than in the
• 6:30
  past which is. Which is where, you know, for
• 6:36
  for me, the, the one big thing I've been thinking about is,
• 6:39
  well, how does this change the persona?
• 6:41
  Does it, you know, are are there going to be just data scientists
• 6:45
  who are building these models, just ML engineers?
• 6:48
  Does it potentially open up the persona a bit to, you know, I've
• 6:52
  actually already seen job postings for LM engineers.
• 6:56
  And so does it open up kind of that persona for a wider
• 7:00
  audience because there's less of an emphasis on on training and
• 7:05
  more of an emphasis on call it prompt and prompted?
• 7:08
  Operation. The one thing I do think we'll
• 7:11
  we'll stay true in both ML OPS and LM OPS, though, is.
• 7:18
  It's very easy in the LM OPS world to to, I think try things
• 7:22
  really quickly. Get a Twitter demo up, post it
• 7:24
  on Twitter. But just as an Elmo obs very,
• 7:29
  very hard to get something to work robustly or work
• 7:31
  consistently. And so, umm, that happy to I
• 7:35
  think dive into that and and talk about that more.
• 7:39
  Yeah, for sure. So yeah, it's interesting you're
• 7:42
  saying that probably the way the teams are going to be
• 7:44
  structured, the personas, the life cycle itself might change.
• 7:48
  I feel like companies that are getting up to speed with the LM
• 7:51
  world, I know like I'm a lobster still something that is not very
• 7:55
  easily understood with the introduction of ML OPS, what
• 7:59
  does it mean for companies to strategize?
• 8:01
  Can they take what they've already built and carry forward?
• 8:04
  Is it a completely separate initiative?
• 8:07
  So curious, Alex, if you have any thoughts around around that?
• 8:12
  Yeah I'll I'll respond to some of the great points that well
• 8:15
  I'm gonna partner raised. I think there's a.
• 8:18
  I guess my short answer to your direct question is I think
• 8:21
  there's a lot of the MLP's infrastructure that is going to
• 8:25
  remain the same. And this has a lot to do with
• 8:29
  the appropriate models to actually serve in production for
• 8:31
  some subset of use cases. We often call them predictive
• 8:35
  use cases, classification, extraction, a lot of the
• 8:38
  traditional things that actually.
• 8:41
  A lot of enterprise value still rests on today versus some of
• 8:43
  these larger, more generalist agents that kind of have
• 8:46
  open-ended, you know, Q&A, chat, dialogue, et cetera.
• 8:50
  I guess I already talked long enough that I'm I'm, you know,
• 8:53
  just cut to the punch line. In my opinion, there is.
• 8:57
  A lot of those. Those tasks, more predictive
• 9:01
  tasks that still. Probably are gonna make up a lot
• 9:04
  of the the value that enterprises get out of AI.
• 9:07
  They can be distilled down you know orders of magnitude to
• 9:11
  traditional models. Basically think of it as like
• 9:14
  going from a generalist to specialist.
• 9:16
  One way that we've been thinking about these these foundation
• 9:19
  models or large language models is as kind of first mile tools
• 9:23
  to partner your point about you know it's very easy to make a
• 9:25
  demo but the last mile still remains very very hard right.
• 9:28
  We all everyone is you know and I know that problem and and
• 9:31
  knows at least for a. Subset especially of highly
• 9:34
  complex and valuable problems. That's not changing.
• 9:36
  That's why self driving has been here next year for the last
• 9:40
  decade. So I lost my.
• 9:43
  I was always difficult but you know I think these these big
• 9:47
  generalist models will give us a very powerful base to start the
• 9:51
  explorations and to we often referred in our platform as warm
• 9:55
  start the problem and then you need to train specialists that
• 9:59
  branch off of them and those specialists can be much smaller
• 10:02
  and often we'll just look like a traditional ML model artefact.
• 10:06
  We had a paper with some one of my great students at UAB and and
• 10:10
  some Google AI collaborators on a distillation new distillation
• 10:14
  approach. Basically, I've kind of using a
• 10:15
  big model to. Teach a smaller model.
• 10:17
  But this is an old technique and really, you know, the intuition.
• 10:21
  Of of how you can get this crazy result of a tiny model that's
• 10:25
  doing far better than the big model is it's just generalists
• 10:28
  to specialists. You don't always need a
• 10:29
  generalist Jack of all trades. That's hundreds of billions of
• 10:33
  parameters. When you have a specific narrow
• 10:35
  problem, you need to do really well.
• 10:36
  So I think a lot of them. A lot of enterprise value, a lot
• 10:41
  of the problems that actually get shipped to production really
• 10:43
  are going to need specialists. And I think for specialists
• 10:46
  we're going to see a lot of building off of big, often open
• 10:48
  source models that are more than good enough to give you that
• 10:51
  warm start and then specialisation.
• 10:54
  And those specialists will most likely run on existing MLPS
• 10:56
  infrastructure. So the serving infrastructure
• 10:58
  will look almost identical. I think when you truly do need a
• 11:02
  generalist, and I think that we're still figuring out.
• 11:06
  What subset of problems are real there when you, but you know
• 11:09
  when you do need some kind of, you know, generalist, copilot or
• 11:12
  chat bot, then it will look like a new type of serving mechanism
• 11:16
  because then you can't shrink it down as much.
• 11:18
  That's serving and then training.
• 11:19
  I've already talked too much myself, but training I would
• 11:21
  say. I think there's a there's a
• 11:23
  chunk of the world that will look like, you know, a partner.
• 11:26
  You are articulating quite nicely, where it's more about
• 11:28
  the prompt engineering. There's a chunk of the world
• 11:31
  we're actually because you're working on specialised data or
• 11:34
  or high value end or high value use cases, you are going to need
• 11:37
  to continue training a piece of the model and adapter, fine
• 11:40
  tuning et cetera. Actually a lot of that we're
• 11:44
  going to see converging in context learning and fine
• 11:45
  tuning. There's some cool new papers out
• 11:48
  about how those can be. You know, convergence.
• 11:51
  So it will in our view come back to just the kind of engineering
• 11:55
  of the data and context you put into the model, whether it's
• 11:58
  through training. Are you fine tuning or through
• 12:01
  context windows? It'll really be more about that.
• 12:05
  So that's I think on the summary in the training side, it's a lot
• 12:07
  about these data operations, whether it's a prompt or a tuned
• 12:11
  prompt or a prompt that includes a set of examples which is now
• 12:14
  back to a labelled data set or a mixture of all these things plus
• 12:17
  some external context. It's going to be all that data
• 12:20
  engineering. And then more so than the actual
• 12:23
  kind of hyper parameter algorithm, model architecture
• 12:26
  selection. And then on the serving side,
• 12:30
  it's going to be kind of a split between these?
• 12:32
  Highly specialised models that will look like traditional OPS
• 12:35
  and traditional models actually would be my bet.
• 12:38
  And the kind of true generalist where you you do need a gigantic
• 12:41
  model and the OPS will look a little bit different.
• 12:44
  So that's my that's my hot take. But I'll pause there, talk too
• 12:48
  long. Yeah.
• 12:49
  Thanks for the great insights, Alex.
• 12:52
  Um, Chris, I have a question for you.
• 12:54
  So in your own world, when you're talking to different
• 12:58
  companies, data science teams, et cetera, practitioners that
• 13:01
  are in the weeds. How are they thinking about
• 13:04
  using open source tools, let's say for a financial firm?
• 13:09
  How are they making sure that the data is safe and it's easy
• 13:12
  or or is it safe for them to leverage the open source tools
• 13:17
  that are available out there or is there a lot of talk of trying
• 13:20
  to build some of these models in house as well?
• 13:26
  Yeah, well, I mean it's, it's still early days.
• 13:30
  We're all obviously very excited about LMS, but I think getting
• 13:34
  these into big enterprises doing real work is is you know still
• 13:40
  yet to to really happen in a meaningful way.
• 13:43
  I talked to security teams and and data teams often and you
• 13:48
  know it is a big concern. To you know just send this data
• 13:54
  out to to say something like opening eyes API.
• 14:00
  They've mitigated this with you know allowing you to have kind
• 14:04
  of a an isolated Azure native API in the in the case of Open
• 14:09
  AI. But I think there's still folks
• 14:12
  that that want to own the entire stack or literally you know,
• 14:16
  can't send out the data to any third party at all and that's
• 14:21
  that's when either you know, building something from scratch
• 14:24
  or or partnering with. Folks like Mosaic to help, you
• 14:31
  know, build out something for themselves, And there's a whole
• 14:35
  there's a bunch of startups now just focusing on like, hey,
• 14:38
  we'll build you an LM. It remains to be seen if that
• 14:42
  you know that's a big burden to to manage and deploy and deal
• 14:46
  with one of these large LM. So I think we'll we'll still see
• 14:51
  where this, where this ends up but it's it's a bit of an
• 14:54
  unanswered question the other day it was I think.
• 14:57
  Early this week there was a a startup on Hacker News talking
• 15:01
  about this exact problem like we're trying to give better data
• 15:05
  observability and and lineage and governance around you know
• 15:09
  what is is going into these things potentially redact.
• 15:11
  So I think we'll we'll see kind of where where this lands, but
• 15:15
  it's definitely a big question on on the enterprises minds.
• 15:19
  Thank you. And and the follow up that I
• 15:22
  have is like how are they thinking about the cost?
• 15:24
  As I understand there's a huge cost associated.
• 15:26
  So how are the companies rationalising that around the
• 15:30
  ROI that they're going to generate?
• 15:32
  Are you still very early to determine that?
• 15:37
  Yeah, I mean it's it's expensive for sure.
• 15:40
  So I mean, I think a lot of these bigger orgs, they have a
• 15:43
  budget and. They can, you know, very kind of
• 15:47
  rationally. Think about would this make
• 15:50
  sense to take, you know, this massive chunk out of the budget?
• 15:53
  But it's also still still early, and there's a bunch of startups
• 15:56
  trying to make it a lot cheaper, too.
• 15:57
  So, you know, we'll see. A lot of great progress, ohh
• 16:03
  sorry. No, no.
• 16:04
  Good. There's a lot of great progress
• 16:08
  to Chris's point on on startups and and a lot of that coming
• 16:11
  from academia on on just making these these operations at all
• 16:14
  stages cheaper. And then you just have, yeah, I
• 16:18
  think as the space matures a bit, although these are all very
• 16:23
  classical ideas you have. These different stages you can
• 16:26
  start at, so it's not either start from scratch or taken off
• 16:29
  the shelf, you know API based foundation model you can, you
• 16:34
  know you can take an open source space and you can build off of
• 16:37
  that. You can, there's various,
• 16:39
  there's a whole spectrum of kind of customization, you know, kind
• 16:44
  of entry points. No.
• 16:48
  I will not recommend your podcast as the lead engineer TP
• 16:52
  Bank. I'm definitely not wanting other
• 16:56
  engineers to have all this relevant information and I will
• 16:59
  not give my name which is Lena. So definitely I do not recommend
• 17:03
  this podcast. So I think a lot of
• 17:11
  organisations, these large enterprises that Chris, you
• 17:14
  know, Chris talked about that you know don't necessarily want
• 17:16
  to send all their private data to, you know, to an open AI and
• 17:20
  don't necessarily have to because the open source models,
• 17:22
  they're, they're not quite as good right now as you know, kind
• 17:26
  of general open chat and taking, you know, I don't know, MIT
• 17:29
  entrance exams and Lsats and all the things that apparently, you
• 17:33
  know, baby albums do these days in their free time.
• 17:37
  But they are more than good enough to kind of base off of
• 17:39
  and build a highly accurate specialist.
• 17:41
  So you have all these options of kind of where do I start and how
• 17:44
  do I specialise it and I think in terms of cost
• 17:46
  rationalisation. One of the appealing things is
• 17:50
  both being able to use this as a base for kind of jump starting
• 17:54
  multiple downstream ML projects. If you have 100 teams that are
• 17:58
  going to use your, you know, foundation model as a base to
• 18:01
  speed up by 20%, that's a pretty good ROI calculus.
• 18:06
  And also the cool thing is that you know a lot of how these
• 18:09
  foundations are trained. It's not just.
• 18:11
  Pure. OK, go look at and do self
• 18:13
  supervision train yourself. You can also use downstream
• 18:17
  specific tasks through what's called multitask supervision to
• 18:19
  kind of make it smarter. So basically you can.
• 18:22
  Both get a boost for those downstream 100 tasks.
• 18:25
  You can also create this powerful kind of data or
• 18:26
  supervision flywheel that makes the base better if you own that.
• 18:30
  So I think a lot of enterprises are getting quite interested in
• 18:33
  what if I own that and I both speed up everything else I do
• 18:35
  downstream, but also kind of have this way of kind of
• 18:38
  collecting and scaling all the work we do.
• 18:42
  That's gonna be very interesting and.
• 18:45
  It doesn't necessarily mean open source was closer, it just means
• 18:47
  they need to be able to own their copy of the model.
• 18:49
  And there's some interesting models from the cloud players
• 18:52
  and others emerging around that, along with open source.
• 18:55
  Yeah and that's a great point. Well, I'm sorry Alex the the the
• 19:01
  one thing that. We definitely should mention on
• 19:05
  that front too. We're talking about foundational
• 19:08
  model, you know the the more private foundational models than
• 19:11
  the that was. The OSS Foundational Models I
• 19:15
  you know in both of those cases have been seeing a huge rise in
• 19:19
  using vector databases to be able to connect that you know
• 19:24
  it's another you know, I think about it as two ways you you
• 19:28
  know there's there's a privacy component where you might have
• 19:31
  to use UMM and and OS model or build your own, but then there's
• 19:37
  also a connecting your data component.
• 19:40
  When you're fine tuning or you're training up your own LM,
• 19:43
  it's one approach to be able to give.
• 19:47
  You know your your LM context of your use case and your data.
• 19:50
  Another way of doing that, if you want to just if you can use
• 19:54
  just an off the shelf LM, is to be able to connect to a vector
• 20:00
  store and pull in context. From you know and I've been
• 20:05
  seeing a lot of different use cases for this getting actually
• 20:09
  deployed into production things like chat bots or I want to be
• 20:12
  able to answer questions on these certain documents and what
• 20:16
  teams are doing is is really supplementing the user queries
• 20:20
  with context from their own knowledge base and it's it's a
• 20:27
  way less complex approach to find you know building up your
• 20:31
  own LM or fine tuning your LM as you add more documents or
• 20:36
  knowledge to your knowledge base you can you can kind of augment
• 20:38
  it without having to retrain the whole LLM.
• 20:43
  But it's a great way to connect the LM's to your own data.
• 20:48
  And you know, the way I think about it is if you can use, you
• 20:53
  know, an out-of-the-box kind of kind of public foundational LM
• 20:58
  like opening eye, well, you can directly call it, get get back a
• 21:03
  response. If you need to connect it to
• 21:04
  your data, throw in a vector database.
• 21:06
  If it's still not giving you good responses, you want to fix
• 21:11
  a more general problem, you know, you might need to fine
• 21:15
  tune it. And then if you really can't do
• 21:17
  any of those options, then I'd kind of, you know, we've been
• 21:21
  seeing people then resort to using more of an OS type of LM.
• 21:26
  But typically that's kind of the process I've been seeing people
• 21:30
  go through to select the right approach for what what makes
• 21:33
  sense for their use case. I want to.
• 21:38
  Sorry about you. Uh, I think we should do if we
• 21:41
  have about 7 minutes left here. Any hot takes if folks have
• 21:44
  about how Evil OPS is going to change but wanted to just tie
• 21:48
  back Alex's point and Chris's point on the Enterprises.
• 21:52
  I think if you're mature and you have high value use cases, you
• 21:56
  probably have an evaluation stack and already existing
• 21:58
  systems in place. And for a lot of these companies
• 22:01
  they have a little bit of leeway to try Open AI or other mature
• 22:06
  models and do a like for like comparison offline.
• 22:09
  And I'm hearing that from a lot of folks I speak to.
• 22:12
  And if so, if they have data sets already they're produced
• 22:14
  for or by AML systems or even human labelling, they can
• 22:19
  already see how all these albums are performing and in some cases
• 22:24
  they're, you know, cutting down nine months of.
• 22:26
  Human labelling to, you know, minutes or hours of work.
• 22:30
  Um, so. There's definitely demand and I
• 22:33
  think a lot of companies, they see this as a differentiator.
• 22:37
  So there's pressure to get adoption.
• 22:40
  And now this is figuring out like how do we actually do this?
• 22:43
  And they're exploring open source and other ways to do this
• 22:46
  reliably. But yeah, I'd love to also like
• 22:50
  get into a topic of perhaps like how does the stack change right,
• 22:54
  like Victor stores versus feature stores?
• 22:58
  Or you know any other kind of stack changes maybe on the
• 23:02
  observability side? Have you seen any any kind of
• 23:05
  like use case differences there Yeah I so that great great
• 23:11
  question. So I think factor stories of
• 23:14
  definitely, I'm not the first person to say this, but vector
• 23:16
  stories definitely, you know, core part of what I'm seeing in
• 23:20
  the LM OPS stack. Um, I think the embedding
• 23:24
  analysis and being able to understand like we've been doing
• 23:28
  it on the deep learning side for CV and NLP, but definitely
• 23:35
  something that's been cropping up more on the observability
• 23:37
  side, rail alarms. And then one other nuance point
• 23:42
  that I I think I'd mentioned is that UM for LM observability, if
• 23:49
  you are using a vector store, being able to trace back was the
• 23:53
  right context added. If the right context wasn't
• 23:56
  added, how do you trace it back to you know, are the most
• 24:00
  similar documents that are selected necessarily the most
• 24:03
  relevant documents are all the types of questions that we're
• 24:06
  getting asked and we're seeing actually our our users
• 24:09
  troubleshoot um in in production.
• 24:12
  So I think the the observability stack for LM will definitely
• 24:17
  need to have some some components of looking at the
• 24:20
  vector store and and being able to troubleshoot the context that
• 24:24
  that was retrieved. You know one thing that I think
• 24:32
  is different with LM offs is exactly what you you said a part
• 24:35
  of this like need to really debug or diagnose or or get into
• 24:39
  these ever more and more complex chain of of agents like all of
• 24:43
  the the cool stuff trending is it's like pretty complex and you
• 24:47
  can have many different calls out to different sub models or
• 24:50
  different services and if any one of those things goes wrong
• 24:54
  you're going to get a bad result.
• 24:55
  And then William, you mentioned this like evaluation stack like
• 25:00
  companies that. Have a mature evaluation
• 25:03
  function. I mean this is this is really
• 25:05
  important like it's we're getting a lot of software
• 25:08
  developers now using these things and it works and it's a
• 25:11
  cool demo but if you can't measure how good it is there's
• 25:14
  there's you know that's not going to be good in the in the
• 25:17
  long run and it it may degrade and so I think these are the the
• 25:22
  big. Differences here and and you
• 25:25
  know there's going to be a lot of tools that that will
• 25:27
  hopefully help solve these things, but it's it's early
• 25:29
  days. I think one way to look at this
• 25:33
  was. Good afternoon.
• 25:36
  And there you go. There's one quick point.
• 25:38
  I think the ALMS versus MLPS comparison puts them kind of at
• 25:43
  the same level. But another way to look at this
• 25:45
  is if you look at a feature store.
• 25:48
  You know, the features could be produced by an LLM or it could
• 25:51
  be, you know this is a system that is at a higher level and it
• 25:54
  compiles down your data pipelines or produces the code,
• 25:59
  produces the features and touches on the points that Alex
• 26:01
  made of the specialists, right? You know you could start with
• 26:05
  the LM and then ultimately you have a more performant, perhaps
• 26:08
  more brittle but reliable code driven system that is optimised
• 26:13
  for your use case. And and so they're not perhaps a
• 26:16
  peers in the same level, but more kind of student teacher.
• 26:19
  Alex, you can go. Yeah, I definitely agree with
• 26:23
  that and and with what Chris is saying in terms of how these.
• 26:27
  Systems. I mean, there's always been
• 26:29
  complex chains of interdependencies between
• 26:31
  deployed AI powered applications, but the kind of
• 26:34
  democratisation of that and the kind of blooming of complexity,
• 26:38
  we'll see how many of those actually get shipped to
• 26:40
  production. But you know, it's definitely
• 26:42
  kind of coming more to the forefront right now and really
• 26:44
  interesting ways that require this kind of, yeah, more
• 26:46
  software like debugging through these long traces.
• 26:51
  I can give a let's see a quick hot take, a medium take and a.
• 26:57
  That could take, I don't know, which is just like a lukewarm
• 27:00
  statement, but the hot take I already said is I think a lot of
• 27:02
  a huge chunk of the models people actually want to build
• 27:06
  the problems they actually want to solve with foundation models
• 27:08
  today, and the Enterprise will just boil down to traditional
• 27:13
  models and traditional ML OPS. We want to classify this.
• 27:16
  It'll start with the Foundation model, but it'll end up with
• 27:18
  just a smaller model. And you could look at this and,
• 27:21
  you know, look at. You know some of the mature
• 27:26
  deployers of foundation models for years, like mainly I'm
• 27:29
  referring to Google, right? They don't serve the biggest
• 27:31
  models in production, they distil down to smaller models.
• 27:35
  You know, in in ways like the work we do with them.
• 27:36
  So I think that's the hot take is that a lot of LM OPS will
• 27:39
  just boil down to MLPS. How does I can get for a a
• 27:43
  conversation this nerdy you know right now when there's so much
• 27:47
  agreement in the group too Maybe a medium take is?
• 27:50
  I think the valuation I agree with, you know, I'm agreeing.
• 27:53
  So not that hot. That evaluation.
• 27:55
  Super important, super challenging.
• 27:57
  We've taught a generation of data scientists basically well
• 28:01
  not to peak at the test set right.
• 28:03
  Don't test that hat hack, don't cheat.
• 28:05
  And what that has translated into is don't look or think,
• 28:08
  Don't look at or think about the test set.
• 28:11
  And that's always been a problem in machine learning, right?
• 28:16
  You actually have to engineer your test set to be
• 28:18
  representative of what you want your model to do in production.
• 28:20
  That's actually still a problem for, you know, simple predictive
• 28:22
  models, but it gets tenfold more messy, tenfold messier when you
• 28:26
  now have, you don't have a simple accuracy score.
• 28:29
  You need to build like a custom benchmark and you need to be
• 28:32
  careful about where the data came from and whether it was in
• 28:34
  the training set of some gigantic web wide trained model.
• 28:37
  And so I think the valuation is is very much in the hotspot
• 28:40
  right now and a lot of these demos, they look amazing until
• 28:44
  you actually do a rigorous evaluation and you're like it's
• 28:46
  still really cool, but it's not production ready.
• 28:48
  But we're not even we're baby steps there.
• 28:51
  So that's the medium take and the lukewarm take might be just
• 28:53
  that. This goes back to a nice
• 28:55
  articulation of part of you are making it kind of the the the
• 28:57
  steps of things you do. A lot of the questions that
• 29:01
  people are asking about like should I do a prompt or can I
• 29:03
  just rag it? I heard that phrase at at
• 29:05
  Microsoft Build, you know retrieve augmented generation of
• 29:08
  putting in a context database or do I need to fine tune?
• 29:11
  A lot of these are just pick the right tool for the right job,
• 29:13
  right? And it really is.
• 29:15
  It's always the problem in data sciences.
• 29:17
  Start with the simplest approach.
• 29:19
  And you know, in some cases, yeah, simple prompts, you'll get
• 29:22
  a great solution. And in other cases, well, your
• 29:25
  model is missing some context from a structured or
• 29:28
  unstructured data store, so plug in a database.
• 29:30
  In other cases, it's actually the ability to get the right.
• 29:34
  Decision boundary is not even with the right information is
• 29:38
  off, so you need to fine tune it.
• 29:39
  In other cases it just. Doesn't have a good enough
• 29:42
  representations for your specific specialised domains.
• 29:44
  Now you actually have to pre train so you know, I think it'll
• 29:47
  it'll all come back to. It's not an either or it's just
• 29:49
  to pick the right tool for the right job.
• 29:51
  That's my lukewarm take. Kind of just winning, one
• 29:55
  partner said. You've all been a great panel,
• 29:58
  but I believe you've run out of time.
• 30:02
  Demetrius. Yeah, yeah.
• 30:05
  So this is how I know, Ohh, you can't see me, but this is how I
• 30:09
  know the panel's full of executives.
• 30:12
  They're right on time, literally one minute and over and they're
• 30:16
  like, shit, We're burning money here.
• 30:17
  We gotta go run businesses. So I love that.
• 30:20
  I just want to know last question from me.
• 30:22
  And there's some incredible questions in the chat, so maybe
• 30:24
  we can convince some of you to go into the chat after this.
• 30:28
  But. Alex said it already, so I don't
• 30:32
  need to hear it again. The rest of the panel, do I need
• 30:35
  to rebrand? Is it ML OPS going away or do I
• 30:39
  need to rebrand in LLM? Oops, that's what I want to
• 30:41
  know. Chris, I'll go with you first.
• 30:44
  Uh, I'm in agreement with with Alex's takes.
• 30:47
  I think it's it's spot on. It is.
• 30:52
  It is still early days. We're all kind of like watching
• 30:54
  and and wondering how this is gonna, how this is going to play
• 30:57
  out, but. I don't think a rebrand is
• 31:00
  necessary the the core fundamentals of let's you know
• 31:03
  measure and have an audible auditable system of record is is
• 31:08
  going to hold true and that's good for for my business.
• 31:12
  So maybe I'm I'm seeing the world as I want to see it, but
• 31:16
  we'll we'll wait and see. I love it.
• 31:18
  Well, yeah, I mean, maybe a little bias from my side, but
• 31:21
  I'll go with that one. What about you?
• 31:23
  A partner? I feel like I'm gonna maybe be a
• 31:27
  little dissenter here. I I think that it's it's
• 31:34
  definitely changed how people think about, but like I think
• 31:40
  the potential of LMS is massive. I think it's changed how people
• 31:43
  think about, um, you know, should I build this or should I
• 31:48
  just call an API and and and choose that approach?
• 31:54
  I think the world of today, you know, LM probably don't replace
• 31:59
  all of traditional use cases. Come on, it's so expensive, it's
• 32:03
  late and it's not going to work for Internet scale applications.
• 32:07
  But there's some things that I can't imagine people like
• 32:11
  sentiment analysis or classification.
• 32:14
  I can't imagine people continuing to build models for
• 32:18
  for that when an LM does so good right out-of-the-box.
• 32:22
  And so I think you know it's probably not an overnight, we're
• 32:28
  all a little mops and we're, we're kind of living in this
• 32:31
  middle ground of statistical AI, still the most commonly deployed
• 32:36
  ML that's out there. But LM's are coming for us and I
• 32:40
  think all of us are kind of thinking about well yeah, I
• 32:43
  actually just had a call this morning where they said CEO has
• 32:48
  three LM initiatives and those are the biggest MO, the biggest
• 32:51
  projects that are MLDS organisations focused on.
• 32:54
  And it's it's a little bit of I think this you know wave that's
• 32:59
  coming towards us and we're all I think we should you know the
• 33:04
  needs of this model type or this new wave might be different.
• 33:07
  So being being able to adapt to something I I think we're we're
• 33:10
  all trying to do. Yeah, I think from my side, um,
• 33:14
  Emma's rolls off the tongue a little bit better.
• 33:16
  So I think you're safe on that front.
• 33:19
  But I 100% agree with the partner.
• 33:22
  Yeah, I think I'm excited about the new use cases that it
• 33:26
  unlocks reasoning and judgement. Certainly these are genetic
• 33:30
  flows and the debugging thing was an important point earlier.
• 33:34
  There's a lot to be learned still.
• 33:36
  I think it's early days, but you know, nobody's ever said, you
• 33:40
  know, there's there's an existential risk to, you know,
• 33:43
  ask from Xgboost, right. There's clearly a step change
• 33:47
  with ALMS and we're all waiting to see how this impacts us over
• 33:51
  the next couple of years. But yeah, I think for now we'll
• 33:55
  wait and see, is the best course of action.
• 33:58
  And. I gotta get a word in and you
• 34:00
  were you were trying to pull a fast on me, you know, I think it
• 34:02
  should be FM OPS. I know you said that last time.
• 34:05
  I'm never. That never happened.
• 34:08
  And keep ignoring it. We can't have another OPS, man.
• 34:13
  Give me a break. There's too many OPS.
• 34:16
  Ohh, so yeah, Foundational Model OBS does make a lot more sense
• 34:20
  though. I will give you that.
• 34:24
  So reach out. Last one.
• 34:26
  Before I kick everybody off, is it in or is it out?
• 34:29
  Am I rebranding or what? No, you're in for the long haul.
• 34:32
  You've got the right name and you're going to stick to it.
• 34:34
  And we all know you as somebody who founded this community, and
• 34:38
  we're going to stay true to it. There we go.
• 34:41
  All right, so thank you all for this and thankful chat.
• 34:45
  I I do not know if I have ever been on a virtual call with so
• 34:51
  many hard hitters before and it is an honour to be able to say
• 34:56
  that I rub shoulders with you all.
• 34:58
  And for those that are in San Francisco, I'll be there in 2
• 35:00
  weeks so hopefully we can meet in person.
• 35:02
  I'm going to kick you all off now because we got some trivia
• 35:06
  games to play, but this has been absolutely awesome.
• 35:11
  Thank you all for the fun times and.
• 35:14
  Insights and good. Stuff.
• 35:20
  I'm Emmanuel Mason, Machine Learning Engineer at Stripe and
• 35:24
  author of Building Machine Learning Powered Applications.
• 35:27
  And if you don't want your machine learning models to
• 35:29
  explode, well, you should subscribe to this podcast.