Latest / Agentic Conversations (formally mlops.community) / Balancing Speed and Safety // Panel // AIQCON

Transcript

Balancing Speed and Safety // Panel // AIQCON · Agentic Conversations (formally mlops.community)

0:09
Today, we're going to. So we're very lucky to have
0:14
Godrell's AI CEO here on stage and Yahoo's director.
0:20
We're going to talk about AI safety and how to really balance
0:23
this with speed. But before we get there, I'm
0:25
just wanted you to introduce yourself.
0:29
Go ahead, Shreya. Yeah, hey everyone.
0:31
My name is Shreya. As Rami mentioned, I'm the CEO
0:34
of Godrell's AI. Guardrails AI is primarily a
0:37
company focused on building platforms to add AI reliability
0:41
and controllability to any generative AI applications that
0:44
you're building. We're primarily an open source
0:46
company with an enterprise offering as well.
0:49
Before starting Guardrails, I have worked in machine learning
0:52
for 10 years across, you know, research and computer vision,
0:55
classical AI and planning, etcetera, self driving cars for
0:59
a number of years before that. And then I guess like like
1:02
everybody else here, torn ML OPS as well.
1:05
Awesome. Hi everyone, My name is Erica.
1:09
I'm an engineering director at Yahoo.
1:11
I manage a team that owns recommendation systems and other
1:13
AIML applications at Yahoo specifically focused on news, so
1:18
content recommendations. I've worked in this industry
1:21
almost 15 years back when we trained machine learning models
1:25
ourselves. I've worked across media and
1:28
e-commerce, mostly on consumer products, and excited to be here
1:32
and chat about yeah, speed and safety.
1:36
Awesome, thank you so much. I'll give a quick introduction
1:38
about myself too. My name is Remy I'm I'm leading
1:42
growth at Vectis we're documentation platform for
1:46
machine learning and AI and I also lead a community of AI
1:50
leaders of over 1300 director plus yeah executives all across
1:58
enterprises in the US specifically in AI.
2:01
So today I, I'm super excited about the fact that we get both
2:07
on the field experience about about the topic and also a
2:12
vendor here, which at the same time has kind of this broadview
2:15
of all the companies they've been working with.
2:18
So I'm very, very excited about hearing about this
2:21
complementarity of the two profile here.
2:22
I want to start by defining AI safety.
2:26
I think starting off with definitions is always a great
2:29
way to start. So yeah, Erica, if you want to
2:32
start with that, what, what does that mean for you?
2:34
Sure. And I would love your, your
2:36
answer as well. So I actually, I asked for us to
2:39
start this question with a definition because I think this
2:42
term means different things to different people.
2:45
And I, I guess I wanted to start out with the, the call out that
2:50
this is not AI safety conversation like the robots are
2:53
going to take over the world unless you want to talk about
2:58
that. But I, I don't really want to
3:00
talk about that. I don't think it's really
3:02
relevant to us living on Earth right now, but I do think that
3:06
there are real risk risks to these products.
3:10
There's always risk to any sort of company.
3:12
There's certainly risk to tech companies.
3:14
And as we add more and more complex technology quickly,
3:19
there's sort of more and more risks and we have to be
3:20
thoughtful about it. So I would say broadly risk.
3:22
And the other thing I wanted to say is that it's contextual to
3:25
the industry that you're working in.
3:26
And certainly if you're working in self driving cars, there are
3:29
very large risks, as we heard this morning from the CEO of
3:32
Cruise. But there's risks in other areas
3:34
as well. I mean, I work in media right
3:36
now and there's a real risk if you're auto generating
3:39
summaries, auto generating content or headlines that you
3:42
might give people misinformation about the election or something
3:45
like that, right? Like if you tell people that the
3:47
polls are closing at a different time than they are actually
3:49
closing and you send that out to a bunch of people in Wisconsin
3:53
or Philadelphia, that has real risks.
3:54
So I think safety and risks are not just confined to industries
3:59
where you could cause like physical, physical harm.
4:02
Yeah. Yeah, I really, really want to
4:06
underscore that that safety, AI safety especially is very, very
4:09
context dependent. I think like a widely or a very
4:13
commonly used, you know, definition of that is something
4:15
that, you know, the Senate bill in California, as we 141047
4:19
talks about, which is essentially can you like, it's
4:22
very specific to, you know, very specific to models and it's very
4:26
specific to very concrete harms in terms of, you know, is this
4:29
model going to make it very easy for an average person to look up
4:31
how to create like, you know, mass destruction or bio warfare
4:36
or, you know, like increased cyber security risk, etcetera.
4:38
So that's a very like specific narrow definition of safety.
4:42
But, you know, often how you're using these models, right, is
4:44
like within the context of an application.
4:47
And so safety becomes like, OK, I have like a Chad board or I
4:50
have a new summarizer, right? Is this news that I, that
4:53
somebody, you know, who maybe doesn't understand the nuances
4:57
of like a specific AI system, if somebody like that comes onto my
5:01
platform, are they just going to verbatim trust everything I
5:03
produce? And then as the generator of
5:06
that news, then, you know, what's the responsibility on me?
5:08
So factuality, you know, not adding any types of bias into
5:13
the, into your predictions, etcetera, is like kind of what
5:16
that definition evolves to if you look at it outside of the
5:19
model and within the context of an application.
5:21
And then that really varies upon, you know, what your
5:23
application is, what your domain is, etcetera.
5:26
Amazing. Thank you.
5:28
So now I want to ask you a question.
5:31
Erica, in your experience, you you come, you've been working at
5:35
many enterprises and you've been going from traditional machine
5:39
learning to LLMS. And I'd like to know what's what's changed in
5:43
terms of approach to safety? Yes.
5:48
So I think that the thing that's changed predominantly is that
5:52
executives want to launch things quickly.
5:56
I felt like in my early part of my career, it was a lot about
5:59
convincing people who had never heard of this to use it and that
6:03
it could work. And I was constantly pitching
6:05
people on projects and they were like, I don't know, is this
6:09
really important? I don't think it's going to
6:10
work. And now there's all this top
6:12
down pressure, right? So we used to be able to
6:14
actually be much more measured and we had to be, we had to make
6:19
a case for it working. And now we have to make cases
6:23
for not launching it because it doesn't work.
6:26
And that has been a real shift in the approach and the
6:30
landscape. And so I actually just today, I
6:33
wasn't working. I wasn't working as I was here,
6:37
but I, I killed a project successfully that I thought was
6:41
a really, really bad idea. So normally I was on the other
6:43
side of that argument. And now I'm on this side.
6:44
And that's been the biggest change for me.
6:47
Yeah. And, and as you were building
6:50
those projects now we, we touched on the, the
6:53
communication, the drivers behind those initiatives, what
6:58
on the field, the way you develop models, the way you you
7:01
designed the systems, What has changed?
7:05
Yeah, I mean to state the obvious, we used to have to
7:07
train the models and get the data and have expertise in
7:11
machine learning. I mean, that's what my
7:12
background is in, in statistics and convex optimization.
7:17
And you don't need that anymore. You just need to call an API.
7:20
And so it's completely, as you say, democratised, opened up who
7:24
can build AI applications. There's good things about that,
7:27
There's bad things about that. But yeah, any, I mean, anyone
7:31
can do it. And in many ways, people who
7:32
have no machine learning background do it much more
7:33
quickly because they don't even think, they don't even know that
7:36
you can tune certain parameters. They, they just plug it in and
7:41
launch it. So that's the, that's the main,
7:44
that's the primary thing that's changing.
7:46
I mean, we don't, you don't need to own the models, but it's not
7:50
that everything has moved that way.
7:52
I mean, many things. I own a big recommendation
7:54
systems for content like those are still trained on our own
7:57
data that you don't want to on every single request go and call
8:00
open AI or something like that. But yeah, it's opened up a whole
8:04
new area of, of applications that work and you can do, you
8:08
know, fairly cheaply. It's cool.
8:12
Yeah, I guess just to interject there, I think like what's from
8:16
a safety perspective, what's novel for this current like era
8:20
of machine learning, which is very LLM Gen AI driven.
8:23
Is that the the surface area of you know what, what your task
8:26
means is just so much broader than what it would be in the
8:29
past, right. Like for recommendations, it
8:31
would be this very, very kind of like well defined, well scoped
8:34
out task, which is, you know, here's like a list of things and
8:36
then like rank them or something, right.
8:38
I'm over simplifying obviously a bit.
8:40
I think now you have, I'm sure a lot of people here are from a
8:43
labour like Devin, which is this AI junior software engineer.
8:48
And it can give it an abstract task and it'll go inside your
8:50
GitHub repo and write up a PR for you, right?
8:53
That the surface area of everything that that task
8:56
touches is so broad. And that means that what is,
9:00
what correctness means or what it means for, you know, this AI
9:03
system to do this task well is just like very ill defined,
9:06
right? Like it's a very complex kind of
9:08
like function to define that well.
9:10
And that means that like now to figure out like, is this
9:13
autonomous system that I'm trusting, is it doing the right
9:15
thing is much, much harder to solve compared to, you know,
9:18
like the previous generation of machine learning, where like
9:21
tasks would be much smaller. So, so now that the, the
9:24
ecosystem changed and like there's a whole new domain that
9:28
opened, how do you keep track of all the new things that come up?
9:33
How, like, what do you try? What do you not try?
9:37
How do you keep, how do you manage the FOMO?
9:40
So I'd love to to get your, your input, Erica 1st and and then
9:45
Tria also like kind of from a vendor perspective, how do you
9:47
help companies kind of navigate this, this new space?
9:51
Yeah, I guess it's the FOMO is real.
9:53
Things are moving so quickly. Personally, I go through phases
9:57
of digging in into the details and then building.
9:59
And it's really hard to build things while also digging in.
10:03
So I can't, I can't personally focus that much on both things
10:05
at the same time. I'm in the very much applied
10:09
space. A lot of the activity right now
10:12
is going on in. Yeah, you call foundational
10:15
models, scaling these models, the hardware super interesting,
10:19
totally irrelevant almost to my day-to-day job.
10:22
And so the truth is that you don't need to follow all of the
10:26
the twists and turns. And if you kind of don't pay
10:29
attention for a few months, a company could be started,
10:32
bounded, funded and failed before you even know that you
10:36
had to pay attention to them. So like, I think there's just a
10:40
lot of noise. There's a lot of noise.
10:42
I have, I asked a friend of mine who's very balanced.
10:44
He's a machine learning engineer, very competent, but
10:46
very balanced, not into the hype.
10:48
And I asked him last week how he deals with the FOMO and how he
10:52
pays attention to the industry. And he says there's one
10:54
newsletter every Friday. I open it, I post two things
10:58
that they post into our company, Slack.
11:00
And my manager loves me. So I I do think that that is
11:04
probably sufficient. Maybe that's bad advice, but.
11:11
Yeah, it's fine. I hope the newsletter is the ML
11:13
Offs newsletter. Yeah, I guess I so I guess I do,
11:19
you know, work in this area where you have this like fire
11:22
hose of like new developments all the time.
11:24
And it's my my work is very like LM focused.
11:27
And it is, it helps to have a short attention span, you know,
11:30
and like be able to kind of like split attention across like a
11:32
bunch of different things. I think it's like there's no
11:36
really good answer. I think there's like a few
11:38
newsletters that I really like. There's a few, you know,
11:41
LinkedIn accounts, Twitter accounts I really like.
11:43
There's a few like company blogs that I watched like a hawk.
11:46
Some subreddits that I think are really high value shouted to our
11:50
slash local llama, which you know, I, I kind of watched like
11:52
a hawk, which is fantastic. I think there's a lot of like
11:56
just being keyed into the right communities and honestly, ML OPS
12:00
like is like a great place to, you know, like do that.
12:04
Yeah, that's kind of my answer. There's no, there's no secret
12:07
sauce. You just watch like some people
12:09
that are really high signal and then you know, kind of like
12:11
focus on those. Amazing.
12:15
So I want to touch now a bit on how do we take care of safety
12:20
first very quickly upstream of the model with your Erica and
12:25
then we can like dive much deeper into a downstream of the
12:29
model. So Erica, what what do you what
12:32
do you put in place? What do you pay especially
12:34
attention when it comes to the data like we we talked earlier
12:37
in this conference about rack system etcetera, like the data
12:41
that is fed or that the model has been trained on etcetera.
12:45
Like how do you manage the the risk associated to this?
12:49
Right. So, right.
12:52
So I'm for LLMS, I'm not training LLMS, to be clear,
12:56
we're training traditional machine learning models and
12:58
neural networks. We're using LLMS. I have no idea
13:00
what these things are trained on.
13:01
Lots of copyright data likely. So I, I think that when you're
13:10
thinking about like risk or safety, you, you need to define
13:15
the harms, right? You need to think about what
13:17
could possibly go wrong and essentially like do a pre mortem
13:21
on what the application is and think outside the box.
13:24
Like if you are a reporter reporting on this company and
13:28
this feature and you want to try to like poke a hole in it and
13:31
write a tell all investigative piece.
13:33
What are you going to, you know, start looking for what are the
13:36
edge cases like people who work at newspapers, who write all
13:41
these stories or who work at the markdown or something.
13:43
They're not about how all these systems fail.
13:45
Like they're not, they don't have some special sauce, right?
13:47
They just like think and try it and think about the worst case
13:50
scenario. And so there's no reason you
13:52
can't do that. And so it's shocking to me how
13:55
many companies get caught like left footed on these
13:58
applications because there's no secret sauce.
14:00
You can also sit down and think about how these applications
14:04
could go sideways. I just like anyway, how Google
14:08
does not get people to do this correctly.
14:10
It just blows my mind. And yeah, that's what you have
14:13
to do to be really, really thoughtful.
14:14
And I think you need people in the room that have different
14:18
backgrounds than you. You need open brainstorming
14:20
sessions. You need an environment where
14:22
people feel comfortable to raise concerns at any point in the
14:27
project. It's not, there's no like right
14:29
or wrong answer about the data that goes into it.
14:32
It's about how you structure the system and how you integrate the
14:36
kind of AIML component. And I'll just give one quick
14:38
example, again, contextualised safety, right?
14:41
So this is again, not killing people, but I used to work at
14:44
the New York Times and I worked on comment moderation and the
14:47
New York Times used to moderate all their comments by hand.
14:50
So there's a team of people, they moderate the comments.
14:52
If you write a comment into the New York Times, someone would
14:55
read that comment before and would have to approve it before
14:58
it got posted to the website. So like pre moderation and we
15:02
wanted to they had we had a great data set to train these
15:05
models on and we wanted we trained a model and we
15:07
implemented like AI augmented system and we were thinking
15:10
about how to design it. And the biggest concern, the
15:12
biggest risk to us was that some voices of our readers would be
15:17
automatically declined, automatically rejected, and we
15:20
would not know about it. That the system would maybe
15:23
learn that comments about race or comments about, you know,
15:31
homosexuality or something like that had a higher likelihood of
15:33
being toxic. And we just automatically reject
15:35
them even if they weren't inappropriate and that those
15:38
voices would be silenced. So we built the system such that
15:41
it would never auto reject. It would only auto accept.
15:45
And if something got through that was not good, it would be
15:47
flagged by people and then it could be taken down.
15:50
But that was the IT was about the mechanism of creating the
15:53
system and not about the data specifically was trained on.
15:56
We did see in the end that some of that data was biassed and we
15:59
were able to correct for that, but we thought about it ahead of
16:03
time and that was the key thing, yeah.
16:05
So, Shreya, if we look at downstream of the model, like
16:10
how would you categorise the main risk that that people
16:13
experience? Yeah, I think it's, it's
16:16
actually weirdly parallel, right?
16:18
Which is that how do you even think about like, so the risks
16:22
are very different, like depending on the application,
16:24
depending on the domain. If you're a financial
16:26
institution doing like customer support with AI chat bots, your
16:29
risk profile is going to be really, really different than,
16:31
you know, like a consumer tech company that's building like an
16:34
internal, you know, employee knowledge QA bot thing, right?
16:38
So depending on that, the risks are really different.
16:41
But I think like the key challenge if you're thinking
16:44
about risks is still the same, which is that somebody has to
16:47
really like, understand the system, understand, you know,
16:51
you're downstream, like stakeholders and use cases and
16:53
domains and then come up with like, you know, some opinionated
16:58
manner of like, these are the things we care about.
17:00
Like hallucinations is, you know, like almost like a
17:03
buzzword. Like people care.
17:05
People talk about hallucinations so much as the failure mode of
17:08
large language models. But for a lot of use cases, you
17:11
know, hallucinations might not be bad or like an organisation
17:13
has to decide if hallucinations are bad for them and, you know,
17:17
what can they really do? So just kind of like sitting
17:19
down ahead of time and thinking about how bad is it for us if
17:23
our system or chatbot or something hallucinates versus
17:27
how bad is it for us if, you know, we accidentally leak some
17:30
information that we're not supposed to.
17:32
Somebody has to come up with that determination.
17:35
There's a bunch of different frameworks, you know, that help
17:37
you do this. I'm think, I think the NIST
17:41
AIRMF is like one of the ones that comes to mind.
17:44
ML Commons. The same guys that do, you know,
17:47
ML perf, like the ML perf benchmark are also coming up
17:50
with an AI safety, I forget like AI safety benchmark or taxonomy
17:54
or something. So that is a good, you know,
17:57
full set of everything that can go wrong and that you have to
18:00
kind of like think about we, you know, at guardrails, we released
18:04
this thing called a guardrails hub, which is basically for all
18:07
of the different types of AI risk.
18:08
These are the guardrails that you care about.
18:10
And we thought, OK, it's the actual like guardrail or asset,
18:14
you know, the code asset that verifies your risk.
18:16
That'll be interesting for people.
18:17
But actually it's just the enumeration of, you know, all of
18:20
these types of risks that product developer, AI developer
18:24
should think about that, you know, people end up finding
18:26
really interesting. So yeah, I think it's like
18:28
mimics a lot of what you see pre like upstream of the model.
18:32
I'd also add that I think about risk as risk the whole company.
18:37
Like how much risk is the company entirely taking on, not
18:40
just the AI systems, right? So if your company has a
18:43
negative impact on somebody in a, in a real material way, they
18:46
do, they do not care if there was like an LOM in the code path
18:49
that resulted in that. They don't care if there's AI in
18:52
that code path. And I think that we can also
18:54
use, and this is not generally part of the conversation, but we
18:57
can also use AI to de risk the companies in general, right?
19:01
Like it doesn't always need to be adding risk.
19:03
It can also be taking risk away. And so one example of that is I
19:07
used to work at Etsy. Etsy you can people, it's UGC
19:10
essentially, they can upload, anybody can upload items and
19:13
sell it on Etsy. And they have a real problem of
19:16
people uploading items that are against policy.
19:19
One one thing is that people will there's this category that
19:24
we refer to as baby bling. So people would like create toys
19:29
and bottles and whatnot and bejewel them and then sell them
19:33
on Etsy. And this is really bad.
19:36
Don't do this because it's a choking hazard for children.
19:39
But, and so there was a team that would try to find the baby
19:43
bling on Etsy, they would be much more aided if they had some
19:48
image models to help them. And so you could actually use AI
19:51
to de risk the entire company. So I do.
19:54
I do think we should think about it on both sides of the
19:56
equation. And and shreya in your
20:00
experience like what are the main?
20:03
Like use case that you've seen where it's been applied and and
20:06
where like people really put like metrics and frameworks in
20:11
place. Could you have a couple of
20:12
examples to share? Yeah, Yeah.
20:14
I think that's a great question. I think use cases though again
20:19
like I guess my, I'm talking from the perspective if you know
20:23
the current era of machine learning, which is very, very
20:26
massively driven by LMS and not about a lot of the work that I
20:30
used to do in the past about self driving and other kind of
20:33
different sets of ML problems. I think if you're specifically
20:36
thinking about LMS, most of the use cases that are successfully
20:41
deployed end up being very constrained use cases.
20:45
So use cases that are typically behind, you know, one layer of
20:48
like protection. So for example, a lot of people
20:50
have, you know, support chat bots which are guarded by an
20:54
actual human that gets the final say of like, you know, yes or
20:57
no, Is this OK or not to kind of like send forward?
21:00
A lot of those use cases end up kind of like making it too
21:02
broad. I think like some of the other
21:04
ones are very internal, you know, question answering, like
21:09
asking questions about your HR policies, etcetera.
21:11
You know, so employee question answering in terms of like the
21:14
metric, like the evals in the metrics that people care about.
21:17
I think you're kind of like circling like what the big
21:20
unsolved question in LM land. I don't I haven't counted, but I
21:24
don't know how many talks today were about LM evaluation.
21:28
So I think it is a it is an evolving conversation.
21:31
I think like people have like some ideas of what works well or
21:33
not. I think one thing we know for
21:35
certain is what word for, you know, the ML two point O
21:38
landscape, which is very deep learning driven and you know,
21:40
very like kind of static metrics, etcetera, is not as
21:43
applicable in LM land because, you know, even the like, even if
21:48
you have ground truth and you can like even comparing ground
21:50
truth with like prediction is really hard because the tasks
21:53
are really, really hard. So some general principles for
21:55
evaluation that work well as, you know, setting up your system
22:00
so that you are accurately collecting human feedback and
22:03
then, you know, looping that in and essentially, you know, using
22:07
it to update like specific parts of your pipeline like retrieval
22:10
or generation or, you know, prompting, etcetera.
22:13
But I think evals are, I think they are the kind of like big
22:17
problem over times, yeah. Erica, earlier during the
22:21
conversation you talked about kind of killing projects which
22:25
were not not suitable. We talked about balancing speed
22:29
and safety here. So when speed exists that the
22:32
product has not been killed, how how do you put the cursor?
22:36
How do you select like how much work to put, how much time to
22:42
put into into the safety compared to just shipping the
22:47
the product as soon as possible? Yes, it's a good question, good
22:53
question. I think a lot of these
22:54
applications you can get something a demo going really
22:57
fast. So might as well get a demo
22:59
going really fast and you might as well it's good to get as many
23:02
eyes on it as possible to start to define a rubric for how to
23:06
evaluate whether it's working or not.
23:07
And having a rubric for what is really bad, right?
23:10
So I think one of the one of the less, one of the themes that I
23:16
think is emerging from all this evaluation work is that it's
23:19
good to have tiers of your test cases.
23:22
Not every test case is the same, right?
23:24
So you really want it to work on the easy examples and it better
23:27
be really, really good on the easy examples.
23:29
And then there's a hard or the wishy washy people don't agree
23:32
on them. And then there's like the really
23:33
bad examples or it would be very bad.
23:35
So you need to start, but you really need to get in the data
23:39
deeply and have a lot of people looking at it, not just the
23:42
engineers. So bots that that push the
23:45
responses to Slack channel send emails every night or something
23:48
to get people looking at it. This stuff is so cool and looks
23:52
so magical and it's so tempting. I think for people like us who
23:58
have worked like in the trenches of this industry for so long,
24:01
when it didn't really work, to be excited to hand pick the
24:06
examples that are working and be like, oh, like CTO, how cool is
24:10
this? Like we can do this now, make a
24:13
demo with these hand picked examples and then they're like
24:15
great, ship it. So but actually that was one in
24:20
10 and the nine other examples don't really work right.
24:23
So I think we have to temper our enthusiasm a little bit and be
24:27
really honest when we show people examples.
24:30
So like the project I killed, so I keep mentioning this, but this
24:34
project I killed today I I was like totally guilty of this.
24:38
I found some really cool examples.
24:40
They wanted to like auto write some headline and auto generate
24:43
some images and personalise it for everybody about news
24:46
stories. And it works great on a few
24:48
examples with a few hand tuned prompts with a few hand chosen
24:51
news stories. And then I was like, you know, I
24:54
really should look on this on like 100 examples randomly
24:56
chosen, look through it. And there was a bunch of
24:58
examples where the summarization misstated the headline and it
25:03
said like, so and so cancelled the tour.
25:06
Willie Nelson cancelled his tour because he's sick or something.
25:10
And then the summary was that he didn't cancel the tour and he's
25:13
having like a cancer tour or something, something really bad
25:16
and really wrong. And I put that that's how I
25:19
killed the project because, you know, there's no way to test
25:21
that. And these models are really bad
25:25
at news, breaking news, misinformation, because they
25:28
don't know what's like happening right now.
25:32
So anyway, so I think if, if, if the project is really not super
25:37
risky and you feel comfortable with it, then just like get your
25:40
get your hands in the data and come up with a rubric for
25:42
evaluation. Tria, how do you guide your
25:45
partners, customers on that balance in between safety and
25:50
speed journey? Yeah, Yeah.
25:53
I think it's a great question. I think a lot of my view, and
25:57
this might be, you know, the unpopular view, but
25:59
unfortunately a lot of this, a lot of this trade off, right, is
26:02
like a it's, it's almost like a product trade off and there's no
26:05
one right answer for what's safe for one person versus another.
26:09
So in general, like what I really recommend is like as
26:13
you're kind of like, you know, when you're, when you're doing
26:15
the scoping of what this product should kind of like look like,
26:18
right? It's like, OK, what are the
26:20
inviolatable conditions for me as somebody who owns this
26:24
feature or this application that wants to wants their users to be
26:28
successful with this application, What are the
26:30
inviolable constraints for me? Like what behaviour do I want to
26:33
guide my user towards? And then being able to like
26:37
ground that criteria in data, right?
26:41
So not just like picking, OK, I tried it five times and it kind
26:44
of works five times. But we're in this era where, you
26:46
know, like you really have this very, very, very long tail.
26:49
And that is kind of the key problem with machine learning,
26:51
which is so many tasks are very hard to capture in like small
26:54
data sets. So you really then like take
26:56
that criteria, ground that in data and then really think
26:59
about, you know, this inviolatable condition that I
27:02
care about a lot. How many times does it get
27:05
violated with the understanding that it's never going to be 0?
27:08
You know, you're always, you're in like nondeterministic land.
27:10
You're always gonna have some sort of failures.
27:12
But what's your susceptibility do that, which really varies for
27:15
people, for everyone. Yesterday I was talking to
27:20
somebody who kept asking me, when you talk to your customers,
27:24
what is the hallucination rate that they see and what is the
27:27
hallucination rate that's acceptable to them?
27:29
And you know that that question, right?
27:31
Like what's a hallucination rate that's acceptable to people?
27:34
Like it's, it's an impossible to answer a question like even for
27:37
me versus, you know, for a financial institution that uses
27:40
guardrails versus, you know, for somebody who is doing an indie
27:44
hacking project, hallucination rate acceptability criteria is
27:47
vastly different. So depending on like in terms
27:52
of, you know, how you given this like post metrics land that
27:56
we're all in right now, like how do you, how do you still execute
28:00
really quickly while, you know, balancing safety is to like
28:03
think upfront about like, what does success look like for me?
28:07
And then once you kind of like build it out, like try to ground
28:10
that success in some data, you know, so is 5% hallucination
28:13
bad? Like tie that to some KPI or to
28:16
some, you know, product metrics that's important and then kind
28:18
of build accordingly. I I'd also add to think about
28:21
the reputational risk to your company.
28:23
Like if, if your company is building something that's like a
28:26
fun, I don't know, fun, goofy consumer social media, like
28:33
maybe you have a it's, it's fine.
28:34
It's but if your company's brand and trust and differentiate
28:41
differentiator is having authority and, and being
28:45
trustworthy and being reliable and being accurate, like maybe
28:48
this is not the technology for you.
28:50
Like skip it. I don't know.
28:52
And I think of the New York City government had a chatbot that
28:58
gave advice about business permits and for, for small
29:04
businesses, etcetera. And it started giving
29:05
misinformation, incorrect information, really bad, very
29:09
bad. Like, I would say not at all
29:12
worth building that because you want the government to be
29:15
trustworthy. You do not want your, your, the
29:18
people living in your city or living in your country not to
29:20
trust the government. I mean, we people don't trust
29:22
the government, you know, very much already.
29:26
So I don't know. It's always a choice not to use
29:29
it. And yeah, I don't know, you
29:33
don't have to get on the bandwagon.
29:35
So I see that everything is very particular to the use case.
29:40
But what I'm wondering is, is there any complexity triggered
29:44
by the technology, the audience, the industry?
29:48
So maybe let's start with the the technology.
29:50
Like if we look at pure like just prompt engineering or fine
29:55
tuning or some rag applications like are there some very
29:59
specific risk associated to, to those technologies that needs to
30:03
be taken, taken care of? Yeah, I think like I think not
30:09
owning the model and the models kind of being this black box
30:13
that is hidden behind an API for, you know, this like
30:16
completely different organisation.
30:17
I think that introduces a lot of risk.
30:19
So essentially, you know, forever.
30:22
I think a lot of people here might use LMLM as a judge style
30:26
of evals, right, where you essentially throw, throw stuff
30:29
at an LM and ask it is, does this look right or not, yes or
30:32
no, right. I think if you run that
30:34
experiment multiple times without changing any input, etc,
30:38
you'll kind of see that you just get different results each time.
30:41
And so if you were to aggregate it, that's not like reliable.
30:44
So essentially like if you look at it with how it used to be
30:48
before, right, which is like, what's the delta with LMS
30:51
earlier, even though models were black boxes and you had like
30:54
these small models that were neural networks and weren't
30:56
interpretable, you would still have a lot of consistency.
30:59
Like you own the training data, you could set the seed, you
31:03
could set, you know, like the inference stack, etc.
31:04
And that meant that even if it wasn't this magical answer
31:08
anything model, you still kind of like knew where you stood
31:11
with that model. Now that a company, you know,
31:14
you don't even up own like model updates, like you know, your
31:17
model provider, like maybe under the scenes, like ships a new
31:19
model, you know, for the same like GPD 3.5 version or
31:23
something, right? That means that it makes it
31:25
really hard to understand like what you're really signing up
31:29
for when you build an application.
31:30
And so the fix for that right is to like, this is what I kind of
31:34
suggest, if any, if you're interested in shipping this,
31:36
which is when you're in that land.
31:39
And if you think that you can benefit from using LMS for any
31:41
stack. The key thing to focus on is
31:43
like, what is the inviolatable criteria?
31:46
That is important to me, you know, as an application builder
31:49
and thinking about whatever you build from that criteria
31:52
perspective, rather than, you know, just like from a, you
31:56
know, this looks good on, you know, like a handful of examples
31:59
or in demo or something. And so it's it's kind of good to
32:01
go. Yeah, I, I also think that
32:03
there's a middle ground that you can take.
32:05
And I'm curious if any of the people that you work with do
32:07
this of using LLMS to annotate data and then and then train
32:11
more traditional machine learning models on that data.
32:14
Neural networks are powerful models and in many of the
32:18
applications that I see people using could be solved very with
32:22
very high accuracy with just a plain neural network.
32:25
And so if you have the capability of training your own
32:28
models, that's a way to get consistency, much more sort of
32:32
observability into what's going on.
32:35
You can then go edit the data up, sample the data, etcetera.
32:38
Yeah, I think. I just, yeah, I guess so we do
32:42
see that, you know, a bunch and we've done a bunch of work on it
32:45
ourselves as well for, you know, like guardrails, for example,
32:48
under the hood RML model. So we do this for a bunch of
32:51
things. I think it's consistent and you
32:54
can shape your data and you end up getting the results that you
32:57
want. But it's not, you know, it's not
33:00
as robust as on an outer distribution example as an LLM
33:04
would be. So let's say you train some
33:06
data, right? That data looks a certain way.
33:07
Now in production, you end up getting a data point that looks
33:10
completely different. It's not reflected in the data
33:12
set that you have. You are, if I had to, if I were
33:15
a betting person, if I had to make a bet, I would bet on an
33:18
LLM with like you know, nicely outlined instructions to handle
33:22
that outer, outer distribution example better than your small
33:26
consistent, you know, fine-tuned model would.
33:29
And that's, you know, that's where that balance kind of has
33:31
to come in. So.
33:34
We're getting quite tight on on time.
33:36
So I just want to touch on one last topic, which is the, the
33:39
people perspective on risk. Who do you think who are you
33:43
involving at the moment when it comes to to risk?
33:47
Like which departments, which role are involved in those
33:49
discussions? And and also Shreya, who do you
33:51
recommend and who do you see in other companies?
33:57
Yeah. So I think it's the, so it's the
33:59
product owner. If you're a large enterprise,
34:02
you know, also the risk and compliance teams typically end
34:05
up getting involved in the later, later stages of
34:07
development. And then finally, it's the, you
34:09
know, data science or dev teams that implement or evaluate how
34:14
much that risk is actually present in their data.
34:17
So those 3 stakeholders, product owner, risk and compliance teams
34:20
and then, you know, the actual developers that enforce like
34:23
whatever risk mitigation strategies.
34:26
Yeah, I agree with that. Legal, I mean, you depending on
34:29
what industry you're in, you have to know what regulatory
34:31
framework you lie under. And even though tech broadly is
34:35
not regulated in the United States, we do have lots of
34:38
regulations of other industries and there's technology companies
34:40
that work in those industries. And so that is a hard constraint
34:44
if you're rolling these out. So you should know that and you
34:46
should talk to your friendly neighbourhood lawyers about
34:49
that. I work maybe not that applicable
34:53
to a lot of you, but I work in media.
34:55
We actually have an editorial arm and I find that people,
34:59
journalists are really good at poking holes in questions,
35:03
defining good questions and poking holes in things.
35:05
So I actually always involve my editorial partners in these
35:08
types of questions about defining.
35:10
We actually have like contracts. We we write our constitutions of
35:15
how the product should work with very specific language.
35:18
And it's very helpful actually to be to have people with that
35:22
kind of background involved in the process.
35:24
Well, Eric, Ashley, yeah, thank you so much for joining us today
35:27
and thank you so much for so the audience see everything filling
35:31
as we talk and that was wonderful.
35:33
Thanks so much.