Latest / Agentic Conversations (formally mlops.community) / From MVP to Production // AI in Production Conference

Transcript

From MVP to Production // AI in Production Conference · Agentic Conversations (formally mlops.community)

0:05
All right, folks, this is a very special episode that we are
0:07
getting into today. We've got a panel discussion
0:10
that happened in the most recent AI in Production conference.
0:15
This was a virtual conference and the panel was all about
0:17
going from MVP to production. You all know hitting the open
0:26
API API and trying to incorporate that into your
0:30
little front end app is nice but go into production.
0:35
Oh my Point in case, we had a blog in the ML OPS community and
0:44
it was written by the Quantum Black team and they showed
0:47
everything that you need for a production at scale rag use
0:52
case. And so it's not just an API
0:56
call. You got to figure out like where
0:58
your data source is, how you chunking it, what are the
1:00
embedding models, how's the data access, where are you looking at
1:04
the input handlers, what are the guardrails to it?
1:07
Configuration stores. So do you have prompt responses?
1:11
How are you getting your operational stores?
1:14
So the feedback database, all of that.
1:17
And then the monitoring and reporting, the usage reports,
1:20
The cost reports, The data reports.
1:23
Oh yeah, I didn't even mention the Elms.
1:27
Do you want to have an LLM gateway so that you can figure
1:30
out which LM is going to be the cheapest?
1:33
Are you gonna have anything in house or are you gonna only be
1:35
using external APIs? There's so much to consider.
1:41
And oh, are you using Pi? Do you need to scrub that?
1:45
What does that look like? There are so many different
1:48
pieces when you are in production that you need to
1:51
think about. So I've got some really fun news
1:56
that frequent listeners will know that we are having our
1:59
first in person conference on June 25th and in the spirit of
2:05
going from MVP to production, the whole conference is going to
2:09
be about AI quality. This feels like the most
2:13
pertinent piece of the ecosystem today.
2:19
One thing is creating a Twitter demo, The other thing is
2:22
actually putting things into production.
2:25
And so we said, what do you need to be 100% confident that when
2:32
you put something into production, it is going to work
2:36
well. You need to ensure that the
2:38
quality is right. That's why the theme of this
2:40
conference is AI quality. I joined up with my buddy MO
2:44
from Colina to make this happen in San Francisco June 25th.
2:50
It's all going down and I would love to see you there.
2:52
I've seen so much support from the community, which is
2:56
overwhelming. It is so cool to see that and I
2:59
would love to see you come and join us on June 25th.
3:05
Let's get into this very special panel discussion from the last
3:10
Virtual Conference and again hit me up.
3:14
If you need a discount code and you're like, I want to come to
3:17
the conference but I can't afford it, I'll make it happen.
3:19
I want the community to be able to join us at this conference
3:23
because I would love to just have the most epic place where
3:28
there are quality AI people talking about AI quality.
3:32
Let's get into the MVP Two production panel.
3:35
See you all later. Everyone, My name is Alex
3:42
Volkov. I'm the host for this panel.
3:44
Our title of our panel is going to be from MVP to production and
3:48
it looks like or almost all of us here.
3:50
I think we're missing one more person.
3:52
And while she joins, I'm just going to introduce myself
3:54
briefly and I'm going to ask you guys to introduce yourself as
3:57
well. As I said, my name is Alex
4:00
Volkov. I'm in the evangelist with
4:02
weights and biases. And I'm also the host of the
4:04
Thursday I podcast, a recording of which I just came 30 minutes
4:08
ago. So I'm still kind of hyped up
4:10
about the everything that happened in the eye for this
4:12
week. And I'm here to host the panel
4:14
on everything from MVP to production and have an amazing,
4:19
amazing panel here. So we're going to start with,
4:22
let's say, I'm going to ask you guys to just introduce yourself.
4:25
I guess we'll start with Eric and then I'm going to call out
4:28
some names, introduce yourself, and then we're going to get
4:30
started. Awesome.
4:33
Thanks for having me here today. I'm I'm Eric.
4:35
Peter, I'm one of the PM leads on the AI platform, Databricks.
4:39
I look after kind of two primary areas.
4:42
First are our model training capabilities.
4:45
So what are the tools that we provide to help customers build
4:48
AI models? That's everything from a simple
4:51
scikit learn model through fine tuning large language models all
4:55
the way up through kind of pre training completely from scratch
4:58
large language models. Second I I look after retrieval
5:02
augmented generation. So how do we help customers with
5:04
that. I know rags the fun buzzword I'd
5:07
really call it more like a I systems it's a lot more than
5:09
retrieval is is what we found out and and we we deliver tools
5:13
there to help folks really build high quality applications.
5:16
I think that's one of the biggest challenges that we've
5:17
heard from our customer base. And in both of these efforts I I
5:21
work really closely with our our academic research team who are
5:26
kind of working on everything from building their own
5:28
foundational models to defining new information retrieval
5:31
techniques to you know new programming paradigms for for
5:34
large language models. So excited to be here and and
5:37
chat with everyone. Awesome.
5:38
Thank you for joining Eric. And then I want to ask Don to
5:41
introduce and say what you're working on.
5:46
Hi, I'm done so I. Work as a.
5:50
No worries. I work as a machine learning
5:52
engineer with process. So yeah, we're working on using
5:58
LM's to create assistance that kind of help portfolio companies
6:03
and now we're going public with a product that acts as a, yeah,
6:07
like an AI assistant. And I'm currently focusing on
6:10
doing that for like very specific use cases within within
6:15
the companies that we work with. German ohh Andrew hey how are
6:20
you man? Good.
6:22
How are you doing? Great.
6:23
Might look sounds. Great.
6:25
From Andrew, I am a co-founder and CEO of Last Molly.
6:29
I used to be the GPM for AI platform at at Facebook AI and
6:35
then did some AI at other companies for that Starlight
6:40
Smile I Last Mile AI is a company's building generally a
6:43
platform basically been working a lot on ML platforms before
6:47
now. What does it generate a platform
6:50
and how do we help solve that? And so actively thinking about
6:53
MVP to production and what's the production mean for general I
6:56
and how does it differ from traditional ML?
6:59
Cool. Nice to.
6:59
Meet you. Awesome, nice, nice.
7:01
Nice to meet you, Andrew and Philip.
7:03
Last but not least, definitely tell us about what you guys are
7:05
doing at Honeycomb. Hello, my name is Phillip.
7:09
I'm on the product team at Honeycomb.
7:11
We are not an AI company primarily not doing AI stuff.
7:15
But what I think is quite exemplary of where we're at in
7:20
this industry right now is you don't have to be an AI company
7:23
that actually do useful shit with this with this tech.
7:26
And my some of the things that I do there is I'm kind of sort of
7:30
the main person who like builds and prototypes things but and
7:33
and to sort of gets them into into production in some way
7:37
shape form or fashion started that last year and got started
7:43
driving a little bit into AI observability or large language
7:47
model observability in particular and really focusing
7:50
on how you can use just information from production to
7:53
directly impact. Well, sorry I shouldn't say
7:57
directly impact directly improve the features that you're
7:59
actually building. And base that not on like
8:02
speculation but on like hey, this is what people are actually
8:05
doing. And this is how we can use that
8:07
to understand patterns and inputs and outputs and and
8:09
meaningfully improve what we have in production, because our
8:12
users are going to want stuff to get better over time instead of
8:14
worse. Awesome.
8:17
So I wanted to maybe use this as a kind of the the, the, the
8:21
first talking point getting feedback from production.
8:23
So it's been it's been fairly clear that once different API's
8:29
were released for GTA, lot of excitement folks boards are
8:32
talking pushing their CEO's like hey, you have to add the you
8:34
have to like what's your AI strategy.
8:36
Everybody started building MP's a lot of like glowing videos on
8:40
on X and and elsewhere and then putting those production and to
8:44
production in the hands of your users.
8:45
Suddenly you get things like well this doesn't actually work
8:48
or this says something that the company is not really aligned
8:51
with. I guess, Philip, we'll start
8:53
with you because you mentioned this first, what are some, what
8:56
are some things you've seen that changed since something that
9:00
you've kind of played in internally and kind of presented
9:02
the stakeholders and then went to production and then the users
9:04
kind of showed that, ohh, there's there's something that
9:08
we have to fix. Did you have a moment like this?
9:11
Oh, absolutely. I I think somebody is quite
9:16
arrogant if they think that they're going to be able to
9:18
predict what their users are going to try to do with what
9:20
they throw into production. It's rather easy actually.
9:23
I would say in in many cases, not necessarily all cases, but
9:26
it's rather easy to get something to the point where
9:28
it's probably good enough to go to production.
9:30
But that's where the hard stuff actually starts.
9:32
You're going to find that, at least in my experience, that
9:37
when you present sort of not even just a blank canvas, but
9:40
just like a way for someone to input what they actually want to
9:43
input, like what's closer to their mental model.
9:45
Rather than them having to learn like the particular gestures of
9:48
like your particular UI or something like that.
9:51
You find that they're going to approach your product just
9:54
differently than they would have if you didn't have that user
9:57
interface in there in the 1st place.
9:59
And so that almost kind of resets expectations around what
10:02
users are going to want to do and what they're going to try to
10:04
do. And that gets you to the point
10:06
where like, OK, if we want to actually keep this thing in
10:09
production, we have to take like what that are actually doing and
10:12
what they're getting back. And if there's a way to get it
10:15
like what they think about that information, like what they're
10:17
getting back and feed that back into development.
10:20
And it's it's like, I mean the software industry always talks
10:23
about like, Oh yeah, we should be shipping code faster and we
10:25
should be like learning from what we're shipping and all
10:27
that. Most teams don't really do that
10:30
very well. But you really like this cranks
10:33
it up. You have to do that very well
10:35
here because otherwise you're just going to shit.
10:37
And like nobody wants bad software that's all janky and
10:41
and doesn't actually do what it's supposed to do.
10:44
And I think like Jenny in particular makes that a lot
10:47
harder because you're you're dealing with things that are in
10:50
some cases, sometimes by design nondeterministic, but then also
10:53
dealing with like practically unbounded inputs that anybody
10:57
could put in that you have kind of no hope of predicting up
11:00
front. I guess I I see quite a few
11:03
folks agree, so I'm gonna ask today then they I I saw you kind
11:06
of go like this for for a few few items here.
11:08
Talk to us about how you guys are dealing with the problem of
11:11
users doing unexpected things, or even following back and
11:14
seeing what users actually did to improve the continued kind of
11:18
deployment the next employment, especially as these models are
11:20
nondeterministic. Ohh, I think I cannot hear you.
11:28
Sorry, I'm using myself. Yeah.
11:32
So it's it's yeah, it's interesting that this was the
11:36
first thing that came up because that's definitely been for us
11:38
this MVP process to production is definitely the place where
11:42
you're finding. Ohh, it looked fine.
11:45
In in it looked fine while we tested and we had, you know,
11:48
done due process. We had test sets, we had, you
11:52
know, checked and retested. But once you give these things
11:55
to users, they just don't behave in the way that you expect them
11:59
to. I think that comes from a place
12:02
of we're designing with the people who have defined the
12:05
problem, but not necessarily the people that use it.
12:07
And then we're also giving people blank canvases, and they
12:13
don't think about these things the same way that we think about
12:15
them. And they don't always understand
12:17
kind of what's happening under the hood.
12:19
And so they treat it as a black box.
12:21
And black boxes are magic, right?
12:24
So it's it is a challenge and I think what what we've done or
12:30
what we're working on is kind of what we think of as an
12:33
evaluation period. So it doesn't have to go from
12:36
zero to 100. You can release in sort of
12:39
phases, so step, step releases and give it to.
12:44
If you have like an end goal user who is like a very general
12:47
user, you don't have to go straight to them.
12:49
You can go to an in between person who has some expectation
12:53
of what it's going to do, but also some understanding of of
12:56
how it's being built. And they they can test it in a
12:59
way that's a bit closer to real life and get feedback and you
13:02
can kind of iterate on that process and gradually work up to
13:06
your final end user. And this kind of evaluation
13:10
period is because also smaller you can evaluate on a much
13:13
deeper level than you could if you're getting hundreds and
13:15
hundreds of new users at the same time.
13:18
And and I think for us that's been really useful to get an
13:21
understanding of of how real users are going to use it
13:24
without risking releasing a product that we don't always
13:29
understand what's going to happen afterwards.
13:32
Yeah, I I wanna, I wanna send it to Eric.
13:36
I see, Eric you're in constant agreement here and getting that
13:39
evaluation and variation is a big thing that I would like to
13:41
ask to to kind of to to stay on for a minute.
13:44
Getting that evaluation back into your like model training
13:47
and continuous like updating whether if you're serving your
13:50
own models whether you're like fine tuning some problems as
13:52
well. Definitely a big piece of all of
13:54
this and in the MVP, at least for some folks who just build
13:57
like a quick demo that's not something folks think about,
14:00
they don't think about. They're like, OK, I wrote my
14:02
problems and put it up. How?
14:03
How have you found this kind of transition, both from the folks
14:07
who are building kind of maybe agents and retrieval systems
14:10
were evaluations hard. But also from putting it out
14:13
there in the actual user's hands and then seeing what they
14:16
actually do. This and then bring it back and
14:18
iteratively improve the pros. Yeah, I mean I think the, the,
14:21
the process on that you have distracted like hey, like giving
14:24
it to a small group of users who are somewhat close but not
14:26
necessarily the, the full breadth of users.
14:28
That's what we see a lot of our customers do.
14:30
We actually call it like the expert stakeholders or the
14:33
internal stakeholders we have four or five experts are kind of
14:35
like, you know, maybe they're not the first person who defined
14:38
the requirements, but they're close enough to actually
14:40
understand the content. Because a lot of the times for
14:43
like our developers and the challenge they run into is
14:45
they're like I actually don't know if this answer is right.
14:48
Like I'm I'm building a bot for the HR team or I'm building a
14:51
bot for the customer support team.
14:53
And like this answer looks good but like I don't actually know
14:57
this isn't my day job. I'm a day the scientist.
14:59
I'm I'm not a I'm not an expert in this.
15:01
And so having that loop and being able to get really fast
15:04
cycles from from that small group, it is really important.
15:08
I think they think there is like having the right tooling to do
15:11
that is incredibly important because the the most brute force
15:13
way to do that is hey here you can come chat with the thing
15:17
I'll write it down on an Excel spreadsheet and give me notes
15:20
and or you know that that that's maybe the the second most brute
15:22
force. The first one is oh just go play
15:25
with it and tell me if it's working or not.
15:27
And what we've heard from a lot of folks get feedback like yeah
15:29
it doesn't work. I hate it.
15:31
They're like, well, well, which ones weren't working?
15:34
They're like ohh was that question I asked about.
15:36
I was like, well, that's not helpful for me.
15:37
And so a lot of what we help our customers with is how do we put
15:41
the make it really, really easy to put tooling in place such
15:44
that every time someone uses it, no matter what, there's a log.
15:48
There's a full capture of everything that happens.
15:51
And there's the ability for the end user to quickly send like a
15:54
thumbs up, thumbs down along with rationale.
15:56
And even as go as far as edit and look at what was retrieved
15:59
along with it. But, you know, I think you know,
16:02
Alex, the point said like, well, logging and looking at the
16:04
feedback is one part, but like how do you actually measure it,
16:07
right? Like it's great if I can go look
16:08
qualitatively through a bunch of things and see that maybe people
16:11
are happy or unhappy, but what are the metrics I can use?
16:14
And so I think that's a really important part and we see both
16:17
emerging, you know, kind of the standard information retrieval
16:20
metrics that people use, but also this emerging kind of area
16:24
of LM judge oriented metrics where you can actually get
16:27
pretty detailed feedback from having a model judge these
16:30
things. And that's not perfect, right.
16:32
So it's not quite the same as a human, but it can give you
16:34
additional signals to to work with and kind of have a suite of
16:37
metrics to look at. And I wanna, I wanna ask Andrew
16:42
as well. Andrew, any any of what we're
16:44
saying resonates with with how you approach figuring out what
16:47
what doesn't really quite work for users in production.
16:51
Yeah. So you might have to dig me out
16:53
if I get too into the weeds because I've been thinking about
16:56
this for so long and we've been working with some customers on
16:59
this that I might get a little bit too deep into it.
17:02
But it's revaluation. It really breaks into three
17:05
possible steps. The first one is you have human
17:09
annotation, human loop style, right?
17:11
Which is like audit logs or manual experimentation where
17:14
you're able to manually annotate whether the results are right or
17:16
wrong. Then you have like your
17:18
heuristic based ones which are like the classic NLP ngram
17:21
algorithms for figuring out what the results produced are
17:24
correct. And then the last one that Eric
17:25
talked about which is LM based or feed it into a large language
17:29
model. Each one has its pros and cons.
17:31
Generally people are thinking more about the annotation style
17:34
or the LM1. The problem with annotation
17:37
style, It's pretty expensive. You know you're looking at going
17:40
to a company and trying to spin that up but the biggest
17:43
difference is it's no longer the annotation that people are
17:45
familiar with which is like you can get a lot of people to help
17:48
annotate now it's like specialised roles.
17:50
I was talking to a company and and they mentioned that they
17:53
have a summarization task and so part of the annotation processes
17:56
who can distil 100 pages of of material into a single paragraph
18:00
and annotate that which is like it's no longer just you know
18:03
someone you find. Now it's like we need an expert
18:06
who can process 100 pages of data and really annotate whether
18:10
this summary is correct. The other approach of LM is
18:13
great, but it's quite costly, right?
18:14
And you're you're kind of feeding GT4 results or any LM
18:18
back into another LM and you're doing some system prompting to
18:21
try to get the right results. We found the most progress with
18:26
actually not using LM and we use more encoder based models which
18:29
are much cheaper. Like when you think about
18:31
evaluators, they're just classification problems and so
18:34
if you can have a really robust classification problem that's
18:37
like one 500th of the cost, why not go with that approach?
18:41
The hardest thing though is it's so diverse.
18:44
You know, I think Phillip and Donnie mentioned it, which was
18:48
like so many people have so many different requirements for it.
18:51
And so you find evaluation is more less of single metric
18:55
driven, more of like a composite metric or many different
18:59
functions people are optimising for.
19:01
And so there's some assertions that go on which is like it says
19:04
valid Jason or marked down, there's some content quality
19:07
which is, is this faithful to what's retrieved in the original
19:09
data set. Is this relevant to the answer?
19:12
And so some composite goes on and it's almost like industry
19:15
base. We've been working with
19:16
different industries and each one is so nuanced.
19:19
It's so difficult to do it for broadly for everyone.
19:22
It almost feels like you need a custom one per industry.
19:25
Hopefully not every company needs a custom one because
19:27
that's not scalable. That's true.
19:30
And I wanted to dig in into the the custom industry one because
19:33
it's going to be impossible to cover all all the possible
19:36
industries and the very generic cancer may not be the Super most
19:39
helpful one for our listeners here.
19:41
And maybe I'll I'll go around and ask like 1 industry specific
19:44
thing that you can think of that that that maybe in evaluations
19:47
like an example that you had to work through in, in how to how
19:51
to actually build an evaluation for, right.
19:53
It's, it's fairly difficult. So whoever wants to go first in
19:56
the industry specific one, Andrew, if you mentioned a few
19:59
feel free. But also, let's go around the
20:01
table just like see how we're differing, like there's
20:04
difference between. A lawyer summarization task, for
20:06
example, and and the the translation for for another
20:09
example. I can start off with one of the
20:12
most classic ones which is for sales.
20:16
NASA actually means North America, South America regions,
20:20
not the NASA that will most likely come up from any LM which
20:23
is our space agency and and and so having that context aware
20:27
evaluation to understand that it's a sales specific system and
20:31
it should actually use the terminology and vernacular
20:34
similar for sales and whatever the acronyms are used.
20:37
And I think all of us who come from those companies that have
20:40
1000 acronyms. Facebook used to have a
20:42
dedicated internal site for acronyms.
20:45
You're going to find yourself needing to do that translation
20:48
and make sure that the the answers are correct as opposed
20:51
to the original training data that is most likely most you
20:54
know, embedded into the LM. So that's the first example that
20:56
comes to mind for me. We we one example from from from
21:01
us is that we've built about for translations and then it started
21:04
translating the the links as well and for example weights and
21:06
biases. You don't want you don't want
21:08
like brand names to be translated because this in
21:10
Spanish does not sound like like what it needs Eric.
21:13
I I see also like I haven't found this.
21:15
Do you have any like industry specific examples of what you
21:18
have to work through in like a specific domain or industry?
21:21
Yeah. I mean I actually just give an
21:23
example from some things we've been doing on our own products
21:25
like our our coding assistant. And so one of the things
21:29
obviously we want the coding assistant to be helpful, right,
21:32
Like it's it's gonna generically helpful, right.
21:34
Well, what is helpful, mean, well, helpful.
21:36
What we kind of discovered was that if you just ask GPT forward
21:39
to evaluate if if the answer was helpful or not, it it
21:43
essentially says close to 100% of the time that what the LM
21:47
generated was helpful. But it's like OK, well like you
21:50
know they asked a question, here's some response.
21:52
And so not not until you kind of prompted the LM with kind of
21:56
very specific guidelines of what is helpful mean and kind of, you
21:59
know, it's all you almost set it up as a few shot problem rather
22:01
than a zero shot type problem. But you know that that's kind of
22:05
one specific example where you do have to give the model some
22:08
context. And once you did that, you
22:09
actually started to see pretty good results in terms of, yeah,
22:12
there are some helpful and not helpful answers, but I think
22:15
that pattern probably applies across other ones.
22:18
Right. Like one, I think one bank told
22:19
me they're like, you know, Eric, for safety for us it means
22:23
something different than everyone who talks about like
22:25
swearing it. It means like specific,
22:27
incorrect infactual information because that's a safety issue
22:31
for us because we might make a bad trade on it.
22:33
And so kind of what these definitions actually mean to
22:36
each person. I I see where the LM judges
22:38
would be going and and like I think what we're gonna take them
22:42
is to make them so that they can start to be tuned with that
22:45
small number of human labels you get and that's a lot of the
22:48
research that we're doing. So ultimately you know Andrew
22:50
kind of made the point of their expensive, well the best way to
22:52
reduce costs is to distil something down to a smaller
22:55
model and if you can while doing that you you can also get
22:58
something that's like a little bit essentially tune the
23:00
classifier head on the end of it you can get a lot better.
23:03
I'll come there. Awesome.
23:07
Don't I want to hear from you as well.
23:08
I saw. I saw that some of this wasn't
23:09
it. Yeah sorry, just like this idea
23:13
of like context is really important for the the, the
23:15
models and to like do the right thing is the context.
23:19
I think for what we're working on right now is is around
23:23
Sequel, but Sequel you know with the actual users data and there
23:28
the issue becomes distorting the context in a way that it can use
23:32
enough of it without losing itself.
23:35
And then I think also again like the challenge of working with
23:41
like consumers directly is a lot of the implied knowledge, a lot
23:45
of what they know is not always available to the questions
23:50
they're going to ask. And how do we then map that or
23:53
or I think the biggest challenge is get the model to recognise
23:56
that it doesn't know that and ask for clarification.
24:00
You know, you know these, these, the abbreviations and the
24:05
terminology that's so specific if it can make an inference that
24:08
well and like that getting it to understand that you don't know
24:11
everything. So please check.
24:13
That's kind of one of the things that's interesting about working
24:17
with a model that can know so much.
24:23
That's true and and I think for many people definitely the
24:27
assurance with which the the the seriousness with which the
24:29
models like yeah this is the right answer and it's not the
24:33
confuses users on the other side because users are used to
24:37
something that it's clear as an answer.
24:39
Philip, have you run into these issues as well?
24:42
What's? What's your take on how to even
24:45
know when the failure mode fails?
24:47
And how to even, like, grab those information and and and
24:50
use them for to actually improve some of these evaluation methods
24:54
and models? So yeah, that's a really
24:57
difficult one. I I've primarily been working
25:01
like when I think of my day-to-day work, Not like user
25:03
interviews and stuff, primarily with like data querying.
25:09
The problem is, data querying is a very generic term.
25:11
There's like a million different databases out there that all
25:14
have like different ways that they're like really good at some
25:16
stuff and really bad at other stuff.
25:18
And your LM does not know the difference between these things
25:22
at all. Like it, It has a passing
25:24
knowledge of like various query languages, but has no
25:28
understanding of LIKE. Is this actually a good query
25:29
for the task that I have at hand?
25:32
I found in in. In my case that like the the
25:36
thing that really differentiates someone is like it feels a
25:40
little bit obvious but like actually being able to run
25:42
something that the LM generates and find a way to evaluate if
25:46
how correct or useful it is in tandem with like presenting
25:52
something to someone in some way.
25:54
So, so like, it's a little bit generic, but but like for
25:58
example with Honeycomb, there is where effectively a data
26:01
querying platform for your observability data.
26:05
A query has a particular shape and there are like rules for for
26:09
that. So like there's levels of
26:13
validation and evaluation even before we go and run the query
26:16
that we can sort of like say, OK, here's like some hard
26:18
criteria if this actually works or not.
26:21
I can be a surprisingly deep thing to go and solve for
26:23
because I guarantee you you're going to have your product in
26:26
like something that like some users kind of want to do that's
26:29
like kind of awkward to do in your product.
26:31
And like in Honeycomb's case, saying the words what is my
26:34
error rate is like a surprisingly challenging,
26:37
difficult question to answer. We're really, really good at
26:40
other questions, but that one maybe not so much.
26:43
And like what is useful query for that trick question?
26:46
There is no useful query for that.
26:48
So but it's still output something.
26:50
So like like now we're getting into OK, while we're giving you
26:53
something, is that something actually helpful and does that
26:57
set you on the right path or does that lead you astray
27:00
somewhere? And like that's I think a point
27:03
that a lot of people are going to be heading down towards
27:06
eventually once they get past the initial hump of like, OK,
27:08
we're generally doing the right thing.
27:10
If you want to be really useful for people, especially the
27:13
people sort of thinking like from a Pareto distribution
27:16
standpoint, like that smaller percentage of of things that are
27:19
a little bit like wonky inputs but tend to be really important
27:22
for a lot of people. That's where you're really going
27:25
to make a break and a good product experience using this
27:27
stuff. And that's where I've not found
27:30
any best practises really. And it ends up being so domain
27:32
specific that there are ways that you can quantify, quantify
27:36
something as good or not. But it's it ends up being so
27:41
like custom almost that like you you have to instrument this
27:45
thing for yourself and you have to bake in some of that domain
27:47
knowledge about like what is generally a useful thing versus
27:50
what is generally a not useful thing.
27:52
And if you don't have that domain knowledge from like
27:54
people on your engineering team, your product team and and so on
27:57
who deeply understands like generally what is helpful for
28:00
customers, even if it's not actually what they're asking
28:02
for, then that that's where you're going to end up
28:05
struggling a lot. Because I guarantee you that
28:07
people with the title of like, VP and executive staff or
28:10
whatever are gonna like, expect this stuff to be able to do
28:13
that. And if it doesn't do that, then
28:16
you're kind of in trouble. So I want to talk about you, you
28:19
mentioned knowledge as well and I want to talk about
28:21
specifically if if you're building a model or you're using
28:25
model in production, just fed up model, you're kind of at the
28:28
whim of the cut off date when opening.
28:30
I decided to kind of stop the knowledge cutoff October,
28:34
December 23, whatever and they keep updating this.
28:37
And if you have a rack system on top of this information
28:39
knowledge system like Eric said, you you still at the risk of
28:43
providing some amount of information, let's say it's not
28:45
like a live API, everything sits in the vector store and then
28:49
that also is kind of, it could go stale.
28:51
So how are you guys thinking about this additional problem of
28:54
data ingestion and updates pipeline?
28:57
How how are you like keeping the knowledge up to date?
28:59
I see Andrew, you're you're agreeing with this.
29:01
And then maybe Eric, let's talk about how we, before we even get
29:04
this to the users, how how do we even make sure that the latest
29:07
version of our SDK appears there or the latest version of
29:10
documentation also gets in there and the model actually knows
29:13
about this? Andrew.
29:17
Before Andrew If. You want?
29:18
Yeah. Big smile.
29:19
So I'll go first. I mean it, it sucks like it.
29:24
Long story short, it's like a massive version control problem.
29:29
We abstracted as much out as possible to try to version
29:32
control everything because there's two things, which is it
29:35
sucks when it comes updated underneath the hood because then
29:39
you see some type of performance or distribution change in terms
29:42
of the performance it was delivering.
29:44
And also sucks when the other side when like you want to keep
29:48
it up to date, how do you refresh it and make sure that
29:51
you understand when to reevaluate things.
29:54
The best that we can do is just like, you know tried to version
29:57
control everything. Actually like I don't know how
30:00
many folks here have like been part of like the recommendation
30:04
systems for some of these companies.
30:05
It feels very familiar like you're trying to version control
30:09
all the infrastructure for retrieval, the data sources, the
30:12
processing pipelines and then the other LM you're trying to
30:15
fix that version and say this is pinned version if we AB tested
30:19
do it with the B. But you've like versioned
30:21
everything and it's like this is getting so gnarly, there's no
30:24
way to roll back because it's going to be so painful head.
30:26
It's it feels the same. And that's The funny thing.
30:29
Maybe, like, I think I listened a little bit into the Talk Four
30:32
and they're talking about how many of the problems seem so
30:35
familiar to just classic MLA's problems.
30:37
It's like it it's funny because it's under the generative AI
30:41
umbrella, but we're, you know, for all of us, you know, we're
30:44
just like, yeah, this is like, it's the same problems under a
30:48
different name and it hurts maybe even more now.
30:52
Yeah. And I think that's that's
30:54
because I was gonna say like we we kind of like fill up.
30:57
You alluded at the idea of like evaluation sets and like
30:59
actually curating for you kind of like what's a good and bad
31:02
answer. And like, it's funny because I
31:04
think like maybe a year ago was a lot of people.
31:07
Jennifer is Eyes magic. It's going to change it.
31:09
We can just put this thing out in production.
31:10
It's gonna be awesome. And I think, like, I think it's
31:13
kind of funny to see now the world shifting back to like what
31:15
ML practitioners have been doing like for you know, not hundreds
31:18
of years, whatever, some very large number of years, which is
31:21
OK. When we started our task, we're
31:23
going to go curate ground truth evaluation set.
31:26
We're going to decide what metrics working with hill climb
31:29
on and and now like the rest of the world everyone is doing
31:32
generative. And I was like oh wow.
31:33
Like actually like now I get why why why why that's necessary.
31:37
And so it's interesting to see if you're if energy point like
31:40
these paradigms that like have been around for many years
31:42
coming back and like it's a much harder problem now because hill
31:46
climbing on our regression model with a ground truth set is is
31:49
like that ground truth that's relatively straightforward to
31:51
collect. But hill climbing on a, you know
31:54
a nondeterministic English language input output is is a
31:59
much harder problem. But you know Alex answer your
32:03
original question like keeping things up to date.
32:04
Right. Like, you know there's obviously
32:06
you could say like I'm going to retrain my LLM everyday or every
32:09
night. Like I don't think that's
32:10
practical And what we've seen is like that that is why people
32:13
start using these systems that are like for rag or you know
32:17
augmenting the prompt with information.
32:19
But like in our view when it comes back down to is its data.
32:23
Where's your data? How good is your data.
32:25
Is your data available. Do you have a robust
32:28
infrastructure diversion control it.
32:30
Do you have robust infrastructure to deploy it.
32:33
And at least for us what we see is like This is why like a lot
32:35
of our customers like yeah. It makes sense to build our
32:38
generative AI stuff right on top of our data platform because
32:40
guess what, Like that's what we need.
32:42
That's what we have to start. We have to get our vector
32:44
databases In Sync with our source systems.
32:46
And so I think just having those pipelines, having that right
32:49
like becomes even more critical of these days, you know whether
32:52
you build on Spark or you build it on you know choose your
32:55
favourite data pipeline builder and poison.
33:00
Guys, this has been full of insight.
33:03
One of the takeaways that I have from this is just OK.
33:06
This new slogan. LM OPS.
33:08
Same problem, new name, but it hurts a lot more.
33:12
This is my, this is my takeaway. Thank you guys.
33:16
I think we have a couple of questions from the chat.
33:19
One of them is from Jeffrey. What would be the right tooling
33:23
to capture and annotate user feedback?
33:26
We've been looking at our gala, but interested in what else is
33:29
out there. Any ideas?
33:33
I think we can all do our shameless plugs.
33:36
Go for it. We have.
33:37
We have great tools for that at Databricks biassed answer.
33:46
Have you guys mostly just built these things in house?
33:48
Do you use just like off the shelf?
33:50
Yeah. It it at Honeycomb it's all it's
33:52
all in house. I mean so I mean granted since
33:55
we're an observability company, we have a really good
33:57
observability for our product and user feedback is is
34:00
basically like a column on an event and and so because we have
34:05
that that sort of rich instrumentation we can just
34:07
slice and dice based off of user feedback and splay out like OK
34:11
thumbs down result. What was unique about inputs and
34:14
outputs? Or like we know this to be a bad
34:17
output given an input. What's unique about user
34:19
feedback? Related to that, there's
34:22
probably some more tools that can do that out there.
34:24
But like frankly, if you have pretty robust like application
34:28
observability, it's quite straightforward to add that and
34:32
then and then use it within within another tool.
34:35
Just don't do it. Maybe with one that doesn't
34:38
handle high cardinality data very well, like Data dog
34:41
otherwise. Or at least be careful about
34:42
doing it. They have a great product, but
34:44
that's a way to get your build to explode, so watch out for
34:47
that one. It always comes down to the
34:50
shameless plug. It's like the anti plug.
34:58
Any other answers? I mean a quick plug, if we're
35:02
doing shameless plug, everything in waiting biases is stored and
35:06
like evaluated within within biases we have reports, we have
35:08
an internal bot called want bot. You guys check it out it's on
35:11
slack. I think it's now on the part of
35:13
the custom GPS as well. We can ask and get like a
35:15
precise answer of how to set up everything where some biases end
35:18
to end and one bot has and with open source one bot and it's
35:23
evaluations and auto evaluations and also reports how we built it
35:27
and how we're doing the data pipeline ingestion and auto
35:30
evaluation as well. And we mentioned some automatic
35:33
evaluations like GP LED evaluations and all all that
35:37
data and all of the ability for us to know whether a small
35:40
change in the vector database or embedding dimensionality or even
35:44
GPT 4 and different model of this kind of like happens and is
35:48
tracked obviously with America. So you can take it from the
35:51
model training all the way to production and then kind of see
35:53
in end to end what changed and how it how it, how it affected
35:57
their users eventually. Nice.
36:01
Thank you very much guys. By the way, in New York City we
36:05
ran a workshop the MF community in New York did of how to build
36:08
the one bot. So that was pretty fun.
36:13
Thank you very much everybody. Please stick around in the chat
36:16
in case people have more questions, and it's been a
36:20
pleasure having. You thank you.
36:21
Thank you everyone was a great panel.