Latest / Agentic Conversations (formally mlops.community) / Data Privacy and Security // LLMs in Production Conference Panel Discussion

Transcript

Data Privacy and Security // LLMs in Production Conference Panel Discussion · Agentic Conversations (formally mlops.community)

• 0:01
  All right, everyone. I just want to jump in real fast
• 0:03
  because I have a huge announcement.
• 0:05
  We will be having a part two of our large language models in
• 0:09
  production. The first conference was such a
• 0:12
  success that we had to follow it up because there were so many
• 0:15
  incredible speakers that reached out after the first one.
• 0:18
  I couldn't help myself. So it will be on June 15, and it
• 0:22
  is going to be online. So anyone from anywhere can
• 0:25
  join, but if you thought the first one was good, then boy.
• 0:29
  Are you in for a treat? Eat, we have some confirm
• 0:32
  speakers already, I'll just go through a list of them, backed,
• 0:35
  a from data bricks, you know, the founder of the heat of
• 0:38
  Rick's and Spark and all that good stuff.
• 0:40
  We recently just had him on the Pod.
• 0:42
  He's coming back to Keynote. You've got chip hewn, who is an
• 0:46
  amazing person and incredible follow on social sharing.
• 0:51
  So many insights and has been dropping knowledge bombs when it
• 0:54
  comes to large language models recently you've also got Scott
• 0:58
  McKay from mmmm, .a I You've got the Deep speed team is going to
• 1:04
  be there from Microsoft that is pretty epic and Sahar more from
• 1:08
  Strife. Another stripe alumni is
• 1:11
  Emmanuel Emison, he's formally of stripe.
• 1:14
  Now he's working at one of these large language model companies
• 1:16
  that you've definitely heard of, but I don't think I can spill
• 1:19
  the beans on yet, and we've got Lance Martin in as from number
• 1:24
  station. George Matthews from inside
• 1:27
  Partners. This is just a glimpse at the
• 1:29
  whole list. Go go sign up.
• 1:31
  You've got all the details in the link in the description.
• 1:34
  Huge shout out to the sponsors redis for showing us some love
• 1:38
  and making sure that this event happens.
• 1:41
  Our guila any scale continual chroma.
• 1:44
  Thank you all, and you can expect all kinds of bad musical
• 1:49
  improvisation. The brakes like I normally do
• 1:51
  it. And who knows?
• 1:52
  I may make that I hallucinate more than chat GP t-shirt
• 1:57
  available for sale again. Check it all out.
• 2:00
  Links in the Scription, we'll see soon.
• 2:06
  How let's get through this panel on data in security.
• 2:11
  Diego is going to be leading us through.
• 2:13
  Nice. Seeing you again Senor, Diego,
• 2:16
  I'm gonna bring on my man Vin dude Vin.
• 2:21
  I gotta tell everybody about two years ago, he came up with the
• 2:25
  best quote ever, which is kubernetes, is a gateway drug.
• 2:30
  And now I think we need to update that quote and it It is
• 2:34
  G. PT is a gateway drug.
• 2:36
  I'm just going to leave it there for you.
• 2:39
  I will let Diego take over because he is the moderator of
• 2:44
  this panel and everyone so really excited about leading
• 2:48
  this panel. So like the core of it is as in
• 2:51
  the title data, privacy and security.
• 2:53
  So we've talked about what does it mean to data privacy and
• 2:56
  Security in the context of a large language models?
• 2:59
  What is mean to trust these new AI system.
• 3:02
  So there's a Questions around the host Nations and we talked a
• 3:05
  little bit in my, in my note around the, you know, low
• 3:08
  affordability like higher for together use cases.
• 3:10
  So without further Ado, I have some amazing panelists here who
• 3:14
  are at the core of this, is it fair to call the previous
• 3:17
  generation of a mellow feels like makes me feel so old.
• 3:20
  I'm going to really quickly start and let them introduce
• 3:23
  themselves. And ask your name, what do you
• 3:25
  do? And kind of, like, give me one
• 3:28
  little thing about like into space.
• 3:30
  What you've been thinking about. So he'll we'll start with you
• 3:32
  since On my top right here, if that's thanks for having us.
• 3:36
  So my name is sahil. I'm an engineer at you.com,
• 3:41
  which is a conversational AI. So attention and one of the
• 3:45
  topics I've been thinking about a lot is, how do we best
• 3:48
  combined a lot? He's advances of a i, with
• 3:51
  advances in information retrieval.
• 3:53
  Then we Schuster founder and CEO of V squared.
• 3:57
  But in technology, for over 25 years data science, machine
• 4:01
  learning for over eleven, I do data strategy and AI strategy
• 4:06
  because well, they wouldn't let me do any cool projects until I
• 4:09
  figured out how to get them paid and make people money.
• 4:12
  So, that's now my majority specialty figuring out how to
• 4:16
  make money with these models. What I've been thinking about is
• 4:20
  what comes after this, because these this Is just the beginning
• 4:25
  and we've got probably a few other revolutions coming
• 4:29
  afterwards. Things like, Robotics are going
• 4:32
  to come to the front to. So that's what I've been
• 4:34
  thinking about. What's next?
• 4:36
  Hello, I'm give orc and crap again, I'm co-founder and CTO at
• 4:42
  zero systems. I'm leaving the both product
• 4:46
  strategy and protocol strategy. And what between 0.
• 4:52
  We develop compilers for knowledge workers which are
• 4:57
  augmenting them in very like, you know, high-value
• 5:01
  sophisticated tasks and actually what we do.
• 5:05
  But what we think the future will be in the intersection of
• 5:09
  automation of sophisticated workflows Foundation models and
• 5:14
  we are super excited to see how the businesses are being
• 5:17
  transformative in this culture. Harry One very excited to be
• 5:21
  here. My background is also an machine
• 5:23
  learning and a lot of like previous generation and machine
• 5:25
  learning as Diego said. So I've been working in ml
• 5:29
  since. Yeah, I think eight or nine
• 5:30
  years, everything from ml research in, you know,
• 5:33
  classically I decision making under uncertainty deep learning
• 5:37
  also work in autonomous systems and self-driving, doing machine
• 5:40
  learning and deep learning for a few years.
• 5:42
  And then most recently, the founding engineer at an mL of
• 5:45
  startup also doing machine learning infrastructure, apply
• 5:47
  to Mel. I'm a top of mind.
• 5:50
  Me and I guess also the reason that I'm here talking on this
• 5:53
  panel is that I'm the Creator off and open source, Library
• 5:56
  called guard rails which as you would expect as Garfield's to
• 5:59
  the output of large language models and makes them a little
• 6:01
  bit more reliable and safe, so be here.
• 6:04
  So who's this start? Let's high level.
• 6:06
  Like how should we think about the difference?
• 6:09
  In some of these data privacy concerns that exist today
• 6:12
  between how we were building models?
• 6:14
  Maybe a couple of years ago. And now using some of these kind
• 6:17
  of foundational model apis, like, How do we frame this?
• 6:20
  I think it's the biggest piece to that's missing from the
• 6:25
  conversation. Is that we're not thinking about
• 6:28
  the patterns, they can uncover used to be, we could find fairly
• 6:31
  simplistic patterns in the more complex, machine learning and
• 6:34
  then eventually deep learning models.
• 6:35
  God, the more complex, the patterns that could be
• 6:38
  discovered this. When we think about data,
• 6:40
  privacy data, security, with respect to these types of
• 6:43
  models, it's no longer the data itself.
• 6:45
  It is the patterns within the data that can be.
• 6:50
  You uncovered and create vulnerabilities for anyone that
• 6:55
  has a significant amount of data out in public.
• 6:59
  There's a study that was done by coal short at Pepperdine where
• 7:04
  they realized, they could through some creative prompting,
• 7:08
  get these models, to give a basically craft a VC pitch in
• 7:13
  the style of different VCS and it was, it was convincing.
• 7:18
  So there are Patterns that are built into the data sets that we
• 7:24
  throw out there all the time on social media and they allow for
• 7:28
  more than just our data to be learned.
• 7:31
  There are deeper patterns now and I think we need to start
• 7:33
  thinking about the implications. So so how do we think about
• 7:39
  going to continue? And now I'll ask you like, you
• 7:41
  know, you're doing, you're working on information retrieval
• 7:45
  and a large scale. Like how do you how are you
• 7:47
  thinking about like the framing around?
• 7:48
  These kind of like this pattern. Things like when do you use?
• 7:51
  When do you think you can use these open, generic people over
• 7:55
  generic apis versus need to bring in models and house.
• 7:58
  Like, how do you frame that That's a interesting point
• 8:03
  around some of these patterns and some older abilities there.
• 8:06
  I think there's probably my head try to ways in which we approach
• 8:11
  taking about some of these issues.
• 8:12
  So on one hand, a lot of these models, we State still make up
• 8:16
  content, I think we're all aware of that.
• 8:18
  So in some ways, you know, there's obviously a lot of
• 8:21
  technical work to be done on reducing whose Nation are
• 8:24
  grounding essentially, a lot of these models, that's one area,
• 8:27
  but then on the other side, just because something is
• 8:30
  hallucinating. It doesn't mean that it's not a
• 8:31
  useful tool for people. So I think it also is somewhat
• 8:34
  of a product question. So how do we make sure that
• 8:36
  people have the right expectations when they're using
• 8:38
  a product and most most language model, they can get the most out
• 8:41
  of it. So I think when we're thinking a
• 8:43
  lot about this type of stuff, those are the two angles in
• 8:46
  which I think a lot about it is this not only a technical
• 8:49
  question, but also product question around, what are the
• 8:52
  right expectations of? We presented people using these
• 8:54
  tools and how do we make sure that they're not going to be
• 8:57
  misled? And they know how to
• 8:58
  responsibilities that. We have a great, great point.
• 9:01
  Interns are like how to use it and so I'm gonna I'm gonna move
• 9:03
  to you Shreya maybe for the audience, kind of frame, some of
• 9:08
  the challenges around hallucinations and kind of like
• 9:11
  how you need to think about it. And then obviously, you know,
• 9:13
  tell us a little bit about your work in terms of what's your
• 9:16
  hypothesis here. Like, you know, you're obviously
• 9:18
  built this like fairly popular open source project that
• 9:21
  scatting a lot of adoption. So very curious to hear from
• 9:23
  you. How you frame it in this
• 9:25
  context? Yeah, I think Hallucination is a
• 9:27
  very interesting problem. Like when people think about
• 9:30
  hallucinations, It's actually you know like a combination of
• 9:32
  problems that all kind of get grouped under this umbrella term
• 9:35
  of hallucination. I think my some of those
• 9:37
  problems are basically falsehoods, Etc.
• 9:39
  Even like you have multiple conflicting sources and you
• 9:42
  aren't able to trust like which one is the golden source that
• 9:45
  you should kind of Base your answer on.
• 9:46
  So it's a bunch of like kind of complex problems going on here.
• 9:50
  I think, like rounding honestly, is the way to go, is the way to
• 9:54
  kind of solve these solve. You know, very domain-specific
• 9:57
  hallucination problem. So I do think that creating
• 10:00
  better All those reading bigger models.
• 10:02
  You know, that are kind of like trying to, you know, be less
• 10:05
  susceptible to. This is like one way to go about
• 10:06
  this. But at the end of the day, as
• 10:09
  all of us have worked with the mouth, we can know that it's
• 10:11
  really hard to get that level of certainty with any machine
• 10:14
  learning model. And so, being able to take
• 10:16
  something that is so powerful. And you know, then add
• 10:20
  constraints on top of that, that make it work really well for
• 10:23
  your specific use case, I think is more just more more tractable
• 10:27
  essentially as a problem to solve rather than just make
• 10:29
  hallucination go away as a Blanket thing for LMS. And so
• 10:33
  one of those things, I, I believe people have touched on
• 10:35
  this before, as well, but like, is the way to go is, you can
• 10:38
  essentially connect what you believe are like good data
• 10:41
  sources or good kind of like fact-checking, you know, either
• 10:44
  agents with tools or are even just like embeddings off good
• 10:48
  data sources with your LM outputs and then any kind of
• 10:52
  like LM outputs that are generated would get you know,
• 10:54
  validated against those like ground who.
• 10:56
  So that's kind of like the guardrails way to do it which is
• 10:59
  that you use your large. His model to generate something
• 11:02
  that, you know, functions and then on top of that for your
• 11:05
  domain, you think about, like, what are my constraints and then
• 11:08
  impose them, you know, using like external sources or are
• 11:11
  external data connections, give her.
• 11:13
  I'm going to move over to give her a hand.
• 11:14
  This one suit, you work in a very applicable space that
• 11:17
  you're building for information workers inside Enterprises kind
• 11:20
  of like these assisted like accuracy matters.
• 11:23
  Trust matters, security and privacy matters.
• 11:25
  What are some of the principles that you're thinking about
• 11:27
  today? As you develop the product,
• 11:29
  obviously don't reveal anything properly.
• 11:30
  For Terry are, you know, a trade secret.
• 11:32
  But I'm very curious, like how you're thinking about because it
• 11:35
  has to be relevant. I mean, you're trying to get the
• 11:37
  trust of these Enterprises to not only connect their data but
• 11:39
  also like use your system inside their workflows.
• 11:42
  Walk us through that. Absolutely Carlos Lee the
• 11:45
  Enterprise is over the sea and belief in the power of the
• 11:50
  generative AI to power the large language models.
• 11:53
  So it is like very close to them like they can live with that the
• 11:57
  charge you 50 for all 14 like you know, personal use.
• 12:00
  This but when it comes to the Enterprise use cases and we are
• 12:05
  working with a large Enterprises such as Fortune 500 companies,
• 12:10
  the largest law firms in the world.
• 12:11
  So their data is very confidential and is actually
• 12:15
  their clients data and they have contractual obligations about
• 12:19
  how they're going to govern their data and of course not,
• 12:23
  there's like, you know, there's a huge like, you know, cousin
• 12:26
  between the opportunities and the reality.
• 12:30
  And what we are doing. We strongly suggest not to use
• 12:35
  the apis like judge a beauty for the confidential data and for
• 12:39
  those cases just bring a i inside organizations because
• 12:43
  nowadays are already a lot of great models for several billion
• 12:47
  parameters, you can bring inside the security perimeter of the
• 12:51
  Enterprise fine-tune on on their domain specific data.
• 12:55
  Just give a product iterate on the human feedback and the reach
• 12:59
  the level of the quality. And the users were trust your
• 13:02
  system saw Diego. You're absolutely right.
• 13:05
  A trust for Enterprise systems, is absolutely must in case.
• 13:09
  They will lose the confidence in your product.
• 13:12
  You'll just ignore that totally absolutely into moving forward
• 13:15
  and kind of like, the trust. I think we have to frame.
• 13:18
  One of the things I'm always kind of curious about have been
• 13:19
  thinking about it. Like, there seems to be this
• 13:21
  people applying the, the use cases to a one-size-fits-all,
• 13:24
  like framework and it's problematic, right?
• 13:26
  Because there's times that like you shouldn't care, right?
• 13:29
  Go use an API, whatever. Is faster, cheaper.
• 13:31
  Whatever gets you there and there's other times that you
• 13:33
  should spend the time to bring it all into your point.
• 13:35
  Bring all the AI into the Enterprise, but that has a cost,
• 13:39
  right? It's if you can be expensive,
• 13:41
  you need to have knowledge and stuff like that, and to be able
• 13:44
  to do the back and forth, based on like how you're thinking
• 13:46
  about security and privacy and not applying everything with
• 13:49
  that, you know, one size fits. All I think is really important.
• 13:51
  I want to kind of push a little bit on the, you know, what have
• 13:55
  you seen an industry in terms of people?
• 13:57
  Like, let's talk about that framing, right?
• 13:58
  Like how to think about the use cases.
• 14:00
  A frame. How to, you know, if I'm
• 14:02
  listening to this talk right now and I'm trying to bring some of
• 14:05
  these use cases into my organization like how should I
• 14:08
  frame it right? In terms of thinking about how
• 14:10
  to try to break down the problem?
• 14:11
  Should I go in route? A should I go and Route?
• 14:13
  Be like who can help me guide the kind of like questions that
• 14:16
  I should be asking myself in terms of like were to run this
• 14:19
  and how to run it? Any of you can jump in on
• 14:26
  things. Like, you know, when you are a
• 14:27
  Carl thinking, about the use cases, at first, you need to
• 14:31
  understand that currently you need to put every single truth
• 14:34
  that you have in the future like under the question because you
• 14:37
  had some on the understand like what is solvable, what is not
• 14:40
  solvable route. Now everything's changed.
• 14:42
  You need to come to your business and understand what
• 14:46
  which are like, you know, the major kpis that will contribute
• 14:50
  to the like in a growth of the your business.
• 14:53
  But where are you know Larry or obstacle to Noah where the
• 14:56
  protests are sophisticated? You are spending a lot of like
• 14:59
  morning with you know less input and understand if you can
• 15:03
  automate that and bringing the people who understand that and
• 15:07
  they say oh no with new technologies is it is possible
• 15:10
  when you work on data strategy a lot.
• 15:12
  Now if I came to you today and I need, I said, I need some advice
• 15:15
  on how to think about this. Can you frame it for me?
• 15:17
  Yeah, definitely. What I tell companies is I tell
• 15:21
  him it's just kind of an arc. There's a whole bunch of use
• 15:24
  cases that you could have, but they haven't been proven yet.
• 15:27
  And if you're not meta, don't be first because you don't have the
• 15:33
  capabilities, you don't have the background, you don't really
• 15:35
  have the domain expertise in this area to be the Pioneer for
• 15:40
  a particular use case. Wait until someone else proves
• 15:44
  it out and then enter in. But that doesn't mean you have
• 15:47
  to wait, to your competitors, get into the market.
• 15:50
  You can look at each use case as a Corey, you look at the ability
• 15:55
  to service customers and there's so much that you can do there,
• 16:00
  but you also at the same time, have to be protecting your
• 16:02
  customers, which is not something most companies are
• 16:06
  thinking about, what happens when their customers submit a
• 16:09
  query across your website. Where does that go?
• 16:12
  How is it stored? What are the Privacy
• 16:16
  implications that you're not thinking about?
• 16:20
  And that's why I say a lot of times you don't want to be You
• 16:23
  first company to do this because you don't have that in-house
• 16:27
  capability to really at an expert level.
• 16:30
  Think about these things but at the same time what you should be
• 16:33
  doing is some really targeted opportunity.
• 16:36
  Discovery thinking about different categories of
• 16:40
  capabilities and problems that you've seen solved in other
• 16:44
  Industries. And then asking yourself how
• 16:46
  could I apply that problem solution and fit it into
• 16:51
  something that I have in my current business.
• 16:55
  Internal use cases are always great for proof of Concepts
• 16:59
  because you're banging that it inside.
• 17:01
  And the risks are significantly lower, then turning around, and
• 17:06
  you see companies like Google did this, Microsoft did this,
• 17:09
  they consumed it internally and then they turned around and let
• 17:13
  companies externally play with it.
• 17:15
  And so a lot of these paradigms are what I'm talking to clients
• 17:19
  about now. But it's really important to be
• 17:21
  thinking through you. Use cases and connecting to
• 17:25
  Value propositions. Not well, it sounds cool.
• 17:29
  So let's do it. No, no.
• 17:32
  Start with the ROI. If there's not a significant
• 17:35
  Roi, you know the if there's not a big dollar value on the other
• 17:38
  side of this, why distract yourself from your core strategy
• 17:43
  and your current competitive advantages, you have to adopt
• 17:48
  this at some point. So you should be forward-looking
• 17:50
  and prescriptive but at the same time, it's all about.
• 17:53
  Out the returns. If there's no significant use
• 17:56
  case that fits that will deliver cash.
• 18:00
  Don't do it. I was gonna ask sahil.
• 18:02
  I thought his so you know working search.
• 18:05
  You look in the future of search, right?
• 18:06
  And retrieval and personalization and I'm kind of
• 18:09
  curious like how you think about you know, if one argues not only
• 18:14
  company but our personal data is kind of like the most valuable
• 18:17
  thing we have, right? But we obviously want to
• 18:19
  contribute that data to search experiences that are great for
• 18:23
  For us. How do we think about like
• 18:25
  what's the future look like? And you guide us through that,
• 18:27
  like in terms of like how you may be how you're thinking about
• 18:31
  it at you or maybe how you're just thinking about it.
• 18:33
  Personally, we can detach those two things if you want, but I'm
• 18:36
  very curious what the future of personalized search looks for.
• 18:39
  It looks. Looks like honestly I think it
• 18:41
  looks super excited. Yeah, I think the future of
• 18:45
  search is probably More exciting.
• 18:48
  Now that I think it's been the last last couple years last five
• 18:53
  years, maybe. And I think there's a number of
• 18:55
  reasons why I think a lot of it is obviously the recent advances
• 18:58
  and you know, natural language processing, Etc.
• 19:01
  I think when it comes to personalized search, I think it
• 19:03
  comes down to how like I guess when we think about if we think
• 19:07
  a little bit more into the future, what can we do to really
• 19:11
  allow you to control your search experience?
• 19:14
  So I think that's something that we'll have a lot more ability to
• 19:16
  control. A lot of times now when were
• 19:18
  interacting with some of these models, for example, just think
• 19:21
  of the concept of a system message, so when you're using,
• 19:23
  you know, one of these models. There's this idea where you can
• 19:26
  specify a system message, and it will basically dictate how the
• 19:30
  model is able to adapt to that. So, that's that, that in itself
• 19:33
  is a degree of personalization of AI that we have not been able
• 19:37
  to do in the past. So I think ideas like that
• 19:40
  basically will allow us to really personalize the results
• 19:44
  that we get from searching the future.
• 19:45
  If we comply that more broadly and then also, in terms of
• 19:48
  bringing your own data, I think there's going to be a lot of.
• 19:50
  We have a lot of ideas at least we were thinking about building
• 19:53
  an open platform where other people can contribute their
• 19:56
  data. And essentially what we call
• 19:58
  apps into search and will incorporate that into our chat /
• 20:02
  of results. So, I think there's, you know, a
• 20:04
  combination of open Community efforts that can make your own
• 20:07
  personalized, as well as virtualization enabling
• 20:11
  technology itself. So we're going to go do that.
• 20:16
  We're going to kind of wrap this up, you know, can probably talk
• 20:18
  about this all day. So I'm going to do one quick,
• 20:20
  you know, rotation through everybody.
• 20:22
  And I'm going to ask you to either recommend a tool, it's
• 20:27
  totally fine to recommend the tool that you're working on a,
• 20:30
  you know, recommend resource or kind of give a thought to the
• 20:34
  audience to go chase in terms of like how to be thinking about
• 20:37
  data, privacy and security and trust in these AI systems.
• 20:40
  So I will start with you. Yeah, I guess maybe I was Start,
• 20:43
  I guess one thought would be maybe just you know what you
• 20:46
  currently find these tools to be very useful to you right now
• 20:50
  where I think sometimes it's easy to think of schools as
• 20:53
  useful in abstract or maybe on a couple examples, but what are
• 20:56
  some ways in which you've been using some of these Technologies
• 21:00
  and consistently useful for you and then, you know, from there,
• 21:04
  I think one can imagine what the future is guests will be.
• 21:07
  Yeah, it's a great question. So since we have, like a very
• 21:11
  big audience here and I That not everyone started to like you
• 21:15
  know work and experiment with generative Ai and bring the
• 21:19
  applications for your link chain.
• 21:21
  So I strongly suggest to do that because you'll be very
• 21:24
  impressed, how fast you can have a great results and then you're
• 21:28
  ready, we'll have a good use case, you can order the like,
• 21:30
  you know, double town on the accuracy at likely user
• 21:35
  experience, right? Vin, I would start looking at
• 21:39
  Auto GP T, the whole concept of self-healing Correcting, those
• 21:44
  are some really fascinating use cases.
• 21:48
  There's danger, but a lot of potential in that direction.
• 21:51
  So I would say, look at those tools.
• 21:54
  If there's anything that comes out of this that I think becomes
• 21:59
  an exceptionally powerful construct going forward.
• 22:02
  That's the one to keep an eye on from a forward-looking
• 22:04
  perspective. Got it and Shreya and I expect
• 22:07
  what you're gonna wait, what we get is expected to push here.
• 22:10
  So do it. Totally have more that But
• 22:13
  explain. Yeah, yeah we could do that.
• 22:16
  So I can, I think I'm going to basically talk about guardrails
• 22:19
  and I'm going to talk about it more in the context of this idea
• 22:22
  that I want the audience to think about, and like home
• 22:24
  inspiration and like, why guardrails kind of fits inside
• 22:27
  that. So I think essentially, we like
• 22:31
  these tools are these, these models are really performant,
• 22:33
  right? And we see, like, really
• 22:34
  interesting, use cases and everything.
• 22:36
  But like, are they ready to be deployed into production where
• 22:38
  they can? You know, like work reliably
• 22:40
  work like a hundred percent of the time and not you know, the
• 22:42
  toilet. Offal messages to your potential
• 22:44
  users Etc. Right.
• 22:45
  And so, I think the idea of what kind of want to share is that
• 22:48
  this isn't going to be the. This isn't sufficient enough to
• 22:52
  like actually put a lot of, you know, the applications that were
• 22:55
  building into production and the actual solution would be a
• 22:57
  hybrid of more traditional machine learning more
• 23:00
  traditional like rule base and heuristic based methods, in
• 23:02
  addition to like large language models.
• 23:03
  And I think like that kind of gives us both the performance as
• 23:06
  well as the kind of reliability safety guarantees.
• 23:08
  We care about, and I think that's a very powerful construct
• 23:10
  of, like, ensemble in those two kinds of methods together.
• 23:13
  And so in that context I think they got raises a good tool that
• 23:17
  I wouldn't push. Obviously I'm biased, but I
• 23:18
  think it's a great tool that, you know, allows you to kind of
• 23:20
  like, get a lot of those, a lot of those guarantees Straight Out
• 23:23
  of the Box. Yeah, awesome.
• 23:25
  Well hey thanks all of you for a great panel.
• 23:29
  I believe we're we're here wrapping up then give org
• 23:34
  Sawhill and strea. Thank you so much for taking the
• 23:36
  time. They will all be in the black
• 23:39
  channel for the conference will also share the links about what
• 23:42
  we you know. What was talked about today,
• 23:45
  Demetrius back to you. All right.
• 23:48
  Wow, that was special so I knew it was gonna be good.
• 23:51
  I'm not gonna lie. You get this many hard hitters
• 23:54
  in one place on one virtual room.
• 23:57
  And what do you expect? This podcast is over, but the
• 24:08
  good news is, if you enjoyed it we have another large language
• 24:13
  model in production conference coming up soon on June 15th and
• 24:16
  16th. As we mentioned before, we're
• 24:18
  going to be doing it all over again.
• 24:21
  We are also pairing that with some in-person events live
• 24:25
  events that are happening around the globe.
• 24:27
  So, check out the calendar. If you're an SF, if you're in
• 24:30
  Berlin, New York, even Amsterdam.
• 24:32
  We've got all kinds of fun stuff happening in.
• 24:36
  And with the next virtual large language model in production
• 24:39
  conference that's happening on June 15th and 16th.
• 24:43
  Two days of this. It's going to be incredible.
• 24:47
  Sign up now with the link in the description, it's completely
• 24:49
  free. I'll see you all there and I am
• 24:51
  already taking song requests for the