Latest / Agentic Conversations (formally mlops.community) / Extending AI: From Industry to Innovation // Sophia Rowland & David Weik // #247

Transcript

Extending AI: From Industry to Innovation // Sophia Rowland & David Weik // #247 · Agentic Conversations (formally mlops.community)

0:00
My name is Sophia, I'm a senior product manager at SAS and I
0:04
just like my drip coffee black and I can drink almost a part of
0:09
it in a day. I am David, I'm a solutions
0:12
architect at SAS and I like to mix things up so I don't get to
0:17
become a coffee snob. What is happening in the Labs
0:20
community? We are back with everyone's
0:23
favourite podcast. I am your host as usual,
0:26
Dimitrios, and today talking with Sophia and David from SAS,
0:31
we got into so many different use cases and so many different
0:35
things that they've had the pleasure or displeasure of
0:38
seeing out in the wild and how you can hopefully take steps to
0:44
not fall into the same holes as others have done.
0:49
There were a few horror stories. We talked about zombie models
0:53
being just out there and companies not even realising it.
0:58
I've heard about this. We talked a few podcasts ago to
1:01
someone who actually had that very same story and Sophia
1:06
talked about how she has met people with no governance in
1:10
place. So let your ears perk up and
1:13
hopefully you get a good laugh out of that or a good cry
1:16
because ideally you're not in that situation.
1:20
But if you are, I understand, I empathise and you can have your
1:24
cry. So this was an awesome breakdown
1:28
about some of the best practises that they've learned over their
1:33
years implementing ML OPS in different industries.
1:37
There's a plethora of use cases out there for machine learning
1:43
and generative AI. We talked a lot about many of
1:46
them and what the downfalls or pitfalls are with these
1:51
different use cases. Let's get into it with Sophia
1:53
and David. All right.
1:55
And a huge thank you to Sass, our sponsor of this episode.
1:57
If anyone is curious on what they are innovating on these
2:01
days, you can find a link in the description.
2:04
Go check it out. They're doing some really cool
2:06
stuff and Sophia and David attest to it in this episode.
2:11
Oh, and by the way, like subscribe, whatever, leave us a
2:13
comment, share it with a friend, you know, all that good stuff.
2:16
Have some fun. We'll see you on the other side.
2:23
So I've got a confession to make and this is going to tell you a
2:29
lot about the vibe of my day today, which is I said, you know
2:35
what, my normal dose of vitamins, I'm just going to
2:40
double that because I knew I was talking to you both today.
2:43
So I hopefully AM extra super powered right now for this
2:48
conversation. So David, I know you're in
2:51
Columbia right now, you're travelling, Sophia, you're at
2:54
home. I presume both of you are doing
2:57
some really cool work when it comes to working in the weeds on
3:03
ML OPS and different AI products.
3:07
I would love to know, like what are you getting your hands dirty
3:13
with day in, day out? And maybe Sophia, you can kick
3:15
us off. Thanks.
3:18
So I work currently as a product manager and I'm focused on our
3:23
solution for Enelops. And I get to work with a wide
3:28
variety of different customers in a variety of use cases.
3:31
So some of my things that I get to do is speak to our
3:35
executives, our users and figure out what are they trying to do
3:39
with ML OPS, what are their gaps, what problems are they
3:43
trying to solve? And from those conversations, it
3:46
gives me a really great insight on the space and where people
3:50
want to go. And I can then take that,
3:53
translate it to requirements and start working alongside our
3:57
developers to bake that into one of our products.
4:00
So I get to work with dozens of customers across industries
4:04
worldwide of different sizes to really start to map out what the
4:09
ML OPS landscape looks like to these different organisations.
4:14
Love that. How about you David?
4:16
What do you get in your hands dirty with?
4:18
I think my my attention is currently kind of split one is
4:22
on the I would say like ground level envelope stuff where we
4:28
are talking about real time integration of models into a
4:31
fraud process where we yeah, I mean, if you're standing at the
4:36
Crozier, you don't want to wait for your transaction to be
4:39
processed. So there needs to be real time
4:41
or even stream processing, and we need to make sure that we
4:44
don't charge you or have you as fraudulent when you clearly
4:49
aren't. And on the other hand, I talk a
4:52
lot about how do you put generative AI into processes and
4:57
make sure that they run great in production, right?
4:59
So it's kind of a real split that I see right now where we're
5:03
still talking a lot and trying to face the challenges of
5:06
traditional envelopes and then trying to adapt how these new
5:10
technologies fit into that as well.
5:13
Yes, And that is a little bit of a path that I wanted to go down,
5:18
which was a, what is traditional MLML OPS problems and challenges
5:23
look like? What are the new Gen AI problems
5:26
and challenges look like? And then what do the use cases
5:30
where you have both of them stacked on top of each other?
5:33
I know that's becoming more and more common these days.
5:36
So I would love to hear what you're seeing in that realm
5:40
also. Now before we get into it, let's
5:44
talk a little bit about what some of these use cases are.
5:48
You mentioned fraud, David, Are there other ones that you're
5:52
seeing commonly as like, no, this is a very clear ML problem.
6:01
I can think of a few off the top of my head like loan scoring.
6:04
I think we have recommender systems.
6:08
Those are very clear. There's different pieces that I
6:14
imagine you're seeing. What are they?
6:16
Yeah, I do. I'm think one of the classics is
6:19
still churn prediction, right. It's cross industry.
6:22
It's a topic you always will out and the question there
6:27
especially in ML OPS is starting to change.
6:29
Where it was a lot of batch focused in the past and it's
6:33
moving more and more into real time processes where also the
6:37
data scientists now have to talk to a different set of people
6:41
where before they talk maybe with the data engineers and they
6:44
knew them already because that's where they got the data from and
6:47
they knew kind of how to build a process together.
6:50
And now they are talking to application developers all of
6:54
the sudden and they want things like open API free spec or they
6:59
tell them, yeah, you have to integrate into my Java
7:01
application or even worse sometimes like COBOL
7:04
applications or whatever. And kind of navigating that
7:08
space. How do you go from batch into
7:10
more? This application focused world
7:13
is really something that a lot of data scientists just get
7:16
thrown into and they need help along that way because they
7:20
might have traditionally not worked in that space.
7:23
Oh, what a great point. That is something that I've been
7:29
seeing a tonne. And we actually were just
7:32
speaking a few days ago to Catherine who wrote a book
7:36
geared towards data scientists on software engineering best
7:39
practises. And it was very much that like
7:42
she wrote the book because she did not feel that she was
7:46
equipped as a data scientist with the software engineering
7:49
best practises. It sounds like you ON one thing
7:54
from this book is that she said the biggest takeaway is
7:59
understanding the common language.
8:01
A lot of times data scientists don't even know what an API is
8:05
or CICD and why it's very important.
8:08
So have you seen ways that these data scientists, now having to
8:15
think about it in real time as an application, have been
8:19
successful? So for me it was always the kind
8:23
of tool based approach to to show them how they can translate
8:27
the batch based approach into an API based approach and what kind
8:31
of difference that makes for them and trying to help them
8:35
understand. So maybe building a small
8:38
JavaScript application together with them.
8:41
And for that, that's a great use case for Jenny.
8:44
I right, to just help it pull something together and getting
8:48
them to test it to see how it feels like and getting them to
8:52
understand that every additional input they require in the
8:55
process is additional input field for the front end
8:59
application. It's not like in batch where
9:01
just everything is available or just an SQL query away in the
9:06
API from the front end developer.
9:08
It's much different than if you require 100 inputs into your
9:12
model, that might be a hurdle to getting it into production.
9:16
And when they see they have to fill out each individual fields,
9:20
so making it really explicit, they start to rethink and like,
9:23
Oh yeah, let me go back to variable selection.
9:26
Maybe I need to trim some of them out because they don't make
9:29
any sense, right? And that only becomes apparent
9:32
if you make it a paint for them. So that has been a great thing
9:38
to showcase and talk about. And I mean, just to jump in from
9:43
even the organisational perspective, we are seeing
9:46
shifts in how data scientists begin to work within the team
9:50
because I mean, you've really hit home for me there with your
9:54
earlier point. When I went through my master's
9:57
programme in data science, what we were taught was the age-old
10:02
adage of let's throw the model over the fence.
10:05
It is not our problem once we've trained it, and our capstone
10:10
project was a great example of where this went wrong because we
10:15
went in and we were working with a local healthcare provider and
10:19
to date that was the most depressing data set I've ever
10:21
seen in my life. It was 300 individuals who where
10:28
2/3 of them had passed away unfortunately and we were trying
10:32
to figure out who would survive a particular procedure.
10:36
And so we went through all of this effort.
10:38
The data was so messy, 1200 columns, 300 rows put forth all
10:43
of this effort doing the cleaning and what did we hand
10:46
off to this healthcare organisation training code.
10:50
We weren't taught to give them a trained model object or the
10:54
scoring code. It was here's your training
10:56
code. Best of luck.
10:58
And I sit with that regret to this day about how awful what we
11:03
did was for that organisation and moving into actually my
11:07
career and figuring out that that's not acceptable in a lot
11:11
of organisations. Like you want your data
11:12
scientists to be integrated inside that team where we're all
11:16
working together on that sole focus of how do we get this
11:19
model in a form and location where it can be used to really
11:24
affect that decision making? Because it's not just the data
11:26
scientist, it's not just the MO OPS engineer or IT resource.
11:30
We we need to also work with our data engineers, with our
11:33
business, with our end users, with model risk so that we can
11:37
have a efficient system that is actually valuable for an
11:41
organisation so that we are moving past hopefully throwing
11:44
the model over the fence. Yeah, yeah.
11:47
But now these days I'm guessing you take a different approach.
11:54
Absolutely. I've taken a hard curve left
11:59
ever since I started working with customers.
12:01
Before moving into product management, I worked in a
12:04
similar role as David. I focused on helping financial
12:08
service customers with their advanced analytics needs.
12:11
From that customer advisory perspective, where I focused a
12:15
lot on leveraging open source models, text analytics,
12:19
optimization, and ML OPS became one of my unique niches where I
12:23
got to figure out how to take what the data scientists are
12:27
doing and putting it into production, where it can be
12:30
used, where it can be monitored, and trying to solve the problems
12:34
that they encounter. It's very interesting.
12:37
As you talk to these customers, it feels like there's a quote
12:42
where every happy family is happy in their own way and every
12:46
unhappy family is unhappy in their own way.
12:49
So everyone started bringing up all of these very unique
12:53
problems at the organisation. I started to kind of batch them
12:58
down into a few different ones. But you still see folks who
13:01
recode their models for production.
13:03
The data scientist develops it in one language and to
13:06
production allies, it has to be in something else.
13:09
And of course that's an opportunity to introduce errors
13:12
with. The data scientist isn't the one
13:14
doing the recoding. You have to test that your
13:17
outputs match from the original model.
13:21
You have dependency management. So if you've developed your
13:23
model using one version of Python And one version of
13:26
packages, and that's not what's available in production, you get
13:31
some very interesting errors. I've had, I've gone in and I've
13:34
done, I've worked quite a bit, a lot when the development
13:38
environment doesn't match the production environment.
13:40
And you'll have to start to interpret those yourselves.
13:44
But that's funny you mention that, because there is another
13:47
person that came on here recently and he was talking
13:50
about how he was working at Yandex and one of the big
13:54
projects that he did was go and analyse all the models that were
13:59
in production and cut out like 10% of them.
14:03
And he said he was able to save so much money because they were
14:06
just basically like zombie models hanging out there.
14:10
They weren't getting hit, but people were afraid to take them
14:14
offline because they didn't know if they were actually driving
14:17
revenue or not. And so you think about those
14:20
scenarios and you're like, wow, that feels like they were
14:22
missing a piece of the circle. You know, it's like you put it
14:26
out there and then you just kind of forget about it and you don't
14:29
recognise if you're getting ROI on it.
14:31
You don't have any clear metrics on what that models doing.
14:35
And this seems like it's even a step further.
14:38
You don't even know the model exists and it's out there.
14:41
Exactly, exactly. There's been, I think there's a
14:44
very interesting number of customers that you'll find that
14:47
or in organisations that just have models hanging out in
14:51
production. And of course that's going to
14:52
incur cost if no one's using them.
14:54
You're paying for the compute for the storage you're opening
14:58
your, you're opening yourself up to potential security
15:03
vulnerabilities or attacks by just leaving them hanging out
15:06
there. And so it's I've started to add
15:09
a little reminders in some of my presentations about the
15:12
analytics life cycle to say, hey, check your models in
15:15
production. Are they actually being used?
15:18
If not, you are allowed to retire those.
15:22
Oh, that is classic you. Yeah, we're giving you
15:24
permission right now. Yeah, go on.
15:26
It's OK to take them offline, right?
15:29
Anybody listening, you do not need that model out there,
15:32
especially if it's not actually bringing any value to the
15:35
company. And people will probably thank
15:37
you for it because like this situation at the index, it'll
15:43
save money. And it also saves you the time
15:46
your talent invests in keeping those models alive, right?
15:49
They could be doing something much more productive or
15:52
interesting to them, right? Keeping things on health support
15:56
is fun for a time, but if it doesn't drive anything, you
16:00
don't get any great feedback on it.
16:02
It will also lead to you maybe looking elsewhere where you get
16:07
more interesting challenges to tackle.
16:09
So you also need to think about how do I manage all of these
16:13
models in the productive ones to keep the talent that can create
16:18
them. Be engaged in my company and
16:20
look for a new use cases of think about how they can improve
16:25
the models that are now in production and driving value
16:28
before they leave or kind of go back to doing something
16:33
completely else. Yeah, it's, it's funny you
16:37
mention that because I was reading the evolution of
16:40
Michelangelo's blog post. And one of the big things that
16:45
they said when they went from Michelangelo one point O to
16:49
Michelangelo 2 point O was they had certain use cases that were
16:54
first class citizens because they were driving so much
16:57
revenue. And what they had to do is
17:02
beforehand any ML model that went out in production, any use
17:06
case was treated the same. So you would have someone who is
17:11
an engineer babysitting a model that may or may not be actually
17:16
driving revenue. And then they have the same SL
17:19
as on a model that is driving millions in revenue.
17:23
And so it just doesn't make sense at all.
17:26
Yep. And this is going on at any
17:30
scale of company, right? It's not just the big hitters
17:33
like Uber that have these types of problems.
17:36
I was working with a company and they had a student work on a
17:40
project for them and the model was actually quite good.
17:43
And then the student left, of course, after some time.
17:46
And how did they operate this model in production before we
17:50
came along? And trying to solve that pain
17:52
point for them, they had to import acsv file, send it off to
17:57
that specific laptop where the environment was all set up.
18:01
Then they rerun the Jupiter notebook and exported the CSV
18:06
file, send it for e-mail back to the production system, and then
18:10
they were good to go, right? I was like, I can't even imagine
18:14
doing any of these steps. I love hearing about these
18:20
horror stories. It's painful sometimes.
18:23
Yeah, it just shows you the lengths that we'll go to and the
18:28
the capabilities that we have to hack something together.
18:33
It's amazing. Yep.
18:35
Humans are resourceful, that's for sure.
18:37
So Sophie, I wanted to ask you, I know that you have been
18:43
getting into some other use cases that we talked about
18:47
before we hit record. Yeah, there's been some very fun
18:50
ones, some very helpful ones. So there's been in my chats with
18:55
folks throughout fast. There's been folks who've been
18:57
working with on a worker safety use case.
19:00
So there's this large shipping and logistic company that
19:02
they've been working alongside and they've had a problem with
19:06
worker injury. They've seen individuals injured
19:11
on the line, on this assembly line and they're trying to
19:14
figure out a solution. How do we make sure that people
19:16
follow proper procedures? Typically when, for example, if
19:20
you're working on an assembly line and you're trying to repair
19:23
it or take it down for maintenance, you have something
19:25
called a lockout tag out procedure.
19:27
And this ensures that no one can start that assembly line while
19:31
you're working on it. Because should they get started
19:34
while you're in there, you really are liable for a lot of
19:37
risk. And so they've been working with
19:40
have the computer vision models that are running on the cameras
19:44
in this warehouse to identify specifically worker safety risk.
19:49
So making sure, for instance, no one's run like leaning over an
19:54
assembly line that's running, no one's putting their hands on
19:57
parts of the assembly line where it shouldn't be.
19:59
And also ensuring that that lockout tag out procedure is
20:02
followed where you physically cannot start that assembly line
20:05
until someone comes in and undoes that lock.
20:08
Because that's where they've seen a lot of workers injured.
20:11
And so they're just having these computer vision models run
20:14
alongside the cameras in this warehouse so that they can
20:18
identify and alert a manager or someone on the floor whenever a
20:22
potentially dangerous situation or scenario is occurring to,
20:28
again, you know, prevent people from hurting themselves on the
20:31
assembly line. There's another much more
20:34
positive one that we've been working with one of our
20:36
partners, Clearblade on. So Clearblade, they work in IoT,
20:40
edge, AI and digital twins, and we work with them in solar
20:44
farms. So we are looking at how we can
20:47
make solar farms more efficient, making adjustments as needed
20:50
based on where the sun is, based on where the clouds are to
20:54
maximise the power that we get from the sun.
20:57
And I've seen a very interesting LinkedIn post from one of the
21:00
folks who are working on this project because recently in the
21:04
last few months, we had a partial solar eclipse here in
21:07
North Carolina. And from here, like you couldn't
21:10
see it without the glasses and you could only see part of the
21:13
sun covered. But he actually had the data to
21:16
show where that drop in the distribution was.
21:19
And it was a perfect correspondence to how much of
21:21
that sun was blocked during that partial eclipse.
21:25
So he pulled it right from the solar farm data from our
21:28
headquarters, because we have a solar farm Outback that's a lot
21:32
of fun and you can see the little we have sheep that mow
21:34
it. So it's a really nice perk about
21:36
being in HQ, seeing the sheep mow the solar farm.
21:41
Oh, that's awesome. Now, going back to that first
21:44
example, not that I don't want to talk all day about solar
21:49
farms and sheep, but the first example that you're mentioning
21:53
seems like there's some challenges with integrating the
22:00
vision model with the camera and then the camera being able to
22:03
alert people. And is it that the system cannot
22:09
turn on if there is some dangerous scenario, or is it
22:15
that it just gets alerted? And yeah, break that down a
22:18
little bit more because it seems.
22:20
Yeah. So with the lockout tag out
22:22
procedure, it is a physical like I go in and there's like a
22:27
something over the assembly button that I pull down and I
22:30
physically lock. So if I'm going to go on the
22:33
align, I physically lock it so no one else can start it.
22:36
But they're seeing that workers aren't following that procedure
22:39
and physically locking it, which means someone can walk by and
22:42
see that, hey, the assembly line is off, the buttons open.
22:45
I'm just going to press the button.
22:47
So because the lock is physical, all you can really do is send
22:52
out that alert. But with the way that this team
22:56
has been working is that you can actually start to stream
22:59
together different pieces of information inside of that flow
23:03
for how you start to know when is something addressable.
23:07
So you can say I only want to look at things in this part of
23:10
the screen. I can look and see, compare that
23:13
with that computer vision model to see is this appropriate based
23:16
on the other information in the scenario.
23:19
And from then I can combine that with logic to send that alert
23:22
down to the line. What are some of the hard
23:25
challenges that you face with that?
23:27
Well, with computer vision models, you do have several
23:30
different frameworks that you are working with.
23:33
So a lot of different people have their own different
23:36
preferences from storing in an Onyx framework or from building
23:40
out their Tensorflow pie Torch or even their SAS models.
23:43
So there's a lot of different models coming in that, you know,
23:47
data scientists develop based on their preferences.
23:49
And from there, you have to compare to see what is the most
23:52
efficient model that you want to use for the framework, what
23:55
gives you the most accurate results and be able to actually
23:58
have the infrastructure and the ecosystem to run that model
24:02
regardless of what it was developed in.
24:05
But making sure that we can put that into where we need it to be
24:09
in just a few clicks. With IoT, you'll also see folks
24:14
do things like develop models for a specific machine.
24:18
So this is one model for this one machine or one model for
24:21
this one sensor. And from that they get thousands
24:25
of models that they have to manage.
24:26
So the scale becomes really crazy.
24:30
We have one organisation that we work with, they are a
24:33
manufacturer and I think they're probably the most analytically
24:38
mature organisations that I've worked with.
24:41
And they're currently running 10,000 models in production or
24:46
their different machines and they're hoping to scale that up
24:50
to 20,000 really quickly. So they are really chugging out
24:54
those models. And with that scale, it's really
24:57
hard to identify errors without proper monitoring, without
25:00
alerting, to know when to focus your attention because you can't
25:03
hire 20,000 people to each sit and watch this single model.
25:08
You have to have some way to know hey these are the rules and
25:12
the thresholds for what an error would look like and hey send
25:16
that alert so that someone can take action as needed.
25:20
But at least it's semi standardised, like you don't
25:23
need custom metrics for each of the 20,000 or do you?
25:28
Not for each of the 20,000. So it's pretty similar metrics
25:32
for similar machines. It's just they want to be very
25:35
specific for each one. So this is the machine for this
25:38
one. So for some of these
25:40
manufacturers, they are working with machinery that are decades
25:44
old and they are seeking out ways to be more efficient with
25:49
the machinery they have available.
25:50
Because buying a new machine might be 10s of millions of
25:53
dollars. How much is a model cost to
25:56
monitor that machine, to look at predictive maintenance for it to
26:00
understand when it's time to maybe switch some of the
26:03
adjustments on that machine so that you get value out of it
26:07
without having to actually replace that million of dollar
26:11
machine that's running their manufacturing plants?
26:14
100%, yeah. And actually we were just, I was
26:18
just talking to Richard on here a few weeks ago too, about doing
26:23
ML in the industry, like heavy industry.
26:26
And he was saying how if you make the wrong prediction and
26:30
you say that a machine needs servicing and it's off the line,
26:34
but really it doesn't, then you're losing out on that too.
26:37
And so the cost of error is very high.
26:41
Exactly. And I've been working pretty
26:44
closely with our model risk team here at SAS, and they are
26:49
experts at knowing how much a bad model costs because it's not
26:54
just monetary value. Sometimes a bad model causes
26:59
reputational risk or perhaps it even causes harm to individuals.
27:04
And so how do you quantify what that bad model costs so that you
27:09
know, is it worthwhile? Can I await that in my cost
27:13
analysis for the value of this analytical project?
27:16
Am I even considering that? Because a lot of organisations
27:19
don't think they need to worry about model risk or for when
27:23
things go wrong, unfortunately. AI models causing reputational
27:28
harm. That is a perfect segue into our
27:31
next topic that I wanted to get into, which is Gen AI and how
27:37
people are utilising it or going what I feel they are very quick
27:44
to put something into production and I would love to hear what
27:48
you've been seeing out there. David, I want to bring you into
27:49
this to just like where has it been successful?
27:55
Where has it not been? What like as people are trying
27:58
to bring Jen AI into the fold, what does it look like?
28:02
Fake. To me still the coolest thing is
28:04
how many new people it enabled to play with AI in general,
28:10
right? Because now that we have
28:12
multimodal models as well, how easily they use it for like a
28:16
traditional computer vision use case and they use like Omni for
28:21
that. They are not even aware that
28:23
they could maybe train a traditional computer vision
28:26
model for that, but they just found this tool and they can now
28:31
validate their own use cases and ideas without having to find
28:35
someone in their data science team that they can get excited
28:39
to work with them or they don't need a budget really.
28:42
So that's where I've seen a lot of new brainstorming and ideas
28:46
come into the data scientist teams is from business users
28:51
that were that they just get an e-mail like, so I tried this and
28:55
this was my prompt and here's the result.
28:57
I really liked it. Can we put it into production?
28:59
Right. And then the then the discussion
29:03
really starts to become more interesting.
29:05
But just seeing people get enabled by a technology like
29:09
that is so cool to me. And it's still amazing.
29:12
Like what kind of crazy use cases business users can come up
29:17
with because we just don't know what their challenges are,
29:21
right? Sometimes it's something like
29:23
churn that everybody knows about.
29:25
But often times it's the smaller issues where they say in, in
29:30
Germany, for example, there is a very this everything is
29:34
standardised, right? We get all these forms from the
29:37
government that we have to fill out and often times print out,
29:40
and then somebody has to retype them.
29:43
And we would think, yeah, let's train a computer vision model
29:47
for that. They don't even think about it
29:49
in the business units. They're just like, let's put the
29:51
image in and see what it gives me, right?
29:53
And then they come back and look how great this is.
29:56
So a lot of the work I also find is helping them to understand
30:01
which things should we rely on Gen AI and kind of take these
30:06
additional investments that we have to make as an organisation
30:11
to introduce them into our architectural stack in general.
30:14
Versus where could we maybe shift that use case to more
30:17
traditional machine learning application just because they
30:21
didn't know about them or thought it was a crazy high task
30:26
to do, right. So that's kind of my my first
30:29
point on on Gen AI. And then I would say the thing I
30:33
see most is really people trying to utilise large language models
30:39
in either this typical document Q&A style thing, which they
30:44
mostly use internally and the other one is trying to drive up
30:49
dark processing. So how do we automate our
30:52
processes more with the help of LLM?
30:55
So how can we maybe give it data and then let it decide on the
31:00
best next action or ask follow up questions, decide where to
31:04
route, for example, emails, right.
31:06
You might have info at your company and then you just get
31:11
anything in there. And traditional machine learning
31:15
or natural language processing might have failed there because,
31:18
I mean, especially in Europe, but also in the US, you get 10s
31:22
of languages in there, right? So you first have to understand
31:25
that and then you have to classify them and then route
31:28
them to the right. And LLMS are just great at that.
31:31
And because in this routing example, you also always have
31:36
kind of a human in the loop. The, the person that gets the
31:39
e-mail that they then can decide on, Yep, that was correct.
31:43
Or Nope, I know how to send it along and we can track that and
31:47
understand how it's behaving. So that's kind of the two main
31:50
areas, I would say where I see companies trying to invest into
31:55
this. But I also have to say that I
31:58
see a lot of nuance fought around it.
32:02
So of course, everybody starts out using the Anthropics, the
32:07
open AI, the Geminis of this world, right?
32:10
Because it's an API you can easily integrate and try out.
32:14
But then the follow up questions come around cost about wait,
32:19
what if open eye changes the model version right.
32:23
I just use the the default endpoint and they change the
32:26
model version. Do I now have to retest
32:28
everything or what happens or back in February when they
32:33
change something on the GPU configuration?
32:35
You only got googly gog back from chat GPG.
32:39
These are the types of risks if you really want to start relying
32:42
on it, can you? So often times we start to
32:47
figure out which large language models as an organisation they
32:51
want to use and introduce kind of a common repository of large
32:55
language models. And then from there in the
32:59
actual use cases start to discover which language model
33:03
fits best and kind of have like a cost to performance metric
33:07
attached to it. So usually the big models they
33:11
are the best, but maybe if I become invest more into prompt
33:16
engineering or invest more into RAG, I can get down to a lower
33:21
scale model and the save cost or save on round trip time.
33:25
I think round trip time is something that a lot of people
33:29
don't talk about it. If you have an application that
33:32
you want to feel interactive, the response has to come back to
33:36
you in less than 800 milliseconds.
33:40
The average call to the big players out there is around the
33:45
2nd that might feel sluggish to people, right?
33:48
And then you introduce Rag for a party API called like you
33:52
introduce more and more stuff to it and feel sluggish.
33:56
So even they're hosting your own open source models can become a
34:01
value add. Even though it's a quote unquote
34:04
worst model, if you look at all the benchmarks, it might be
34:08
better for that specific use case.
34:10
So that's what I see a lot of discussions around.
34:16
How do you integrate it, right? It's not like there's one clear
34:20
use case winner. To me, it's really about the
34:23
question, OK, we have a lot of ideas, but how do we get them
34:28
into production and keep them in production?
34:31
Great points. The so many tangents that I want
34:36
to go on off of that what you were saying, because this idea
34:42
on how it's like the easiest and most common way to start is just
34:47
by hitting an API. And you can have even beyond
34:51
that before that, like you were saying, you have business side
34:56
of the house using ChatGPT and just prompting.
35:01
And so there's an easy win. The problem I think I've also
35:06
heard people talk about is not only the cost, but when they
35:10
really start looking into it, they go back to that governance
35:13
piece and say, wait a minute, our team is doing what and
35:17
giving what data to who. So I know that's come up quite a
35:22
few times. And if you're in a big
35:25
enterprise, it's a much bigger problem than if you're in a
35:28
small fast moving startup that doesn't necessarily care that
35:32
much about their data or it's not like they have PII or
35:36
anything, any sensitive data. But if you're in these
35:39
enterprises and you're using it to help you with spreadsheets or
35:42
ask questions on things and it's not been sanctioned by the
35:45
company and it's not it's own like enterprise instance, then
35:49
you can get into some. Hot water even for startups, I
35:55
would say if you think you don't have privacy issues around the
36:00
data, then the the question almost becomes do you have
36:04
valuable data, right? Is it just something anybody can
36:09
scoop up then or replicate your use case?
36:11
So it would always be careful on that front.
36:14
But also having good privacy policies even as a startup can
36:19
be a differentiator for you. So the big enterprises, they
36:22
have to do it for reputation sake and just because they are
36:26
very mature and you would get slandered much more easily in
36:31
the press. But even as a as a start up, I
36:33
don't think you should be too LAX with that.
36:38
I think if we look at the AI incident database that's like a
36:42
public database source. There's also more and more
36:45
start-ups showing up in there because of that one next kind of
36:50
work and feel with data. And I don't want my data to be
36:55
just out there, right? Might make me hesitant to try
36:59
the next startup product. So thinking of that is always
37:03
critical. Yeah, a few months ago I was
37:05
talking to a healthcare system who they were diving into
37:10
ChatGPT at large language models cuz they wanted to better
37:14
support their doctors by reading through all that documentation,
37:17
all of the patient records. But they had so many security
37:21
concerns. They couldn't simply use an
37:23
externally hosted large language model because of those privacy
37:27
concerns. They had to look and see what
37:29
can we host internally, What can we firewall around?
37:32
What can we do to make sure that the private personally
37:36
identifiable information for our patients do not leave our local
37:40
system? And even with some organisations
37:45
I know where my husband works, he's not allowed to use, he's
37:49
not allowed to start to start asking questions about the code.
37:53
He can't use Stack Overflow to address questions because
37:55
they're that concerned about their code getting out.
37:58
And so folks like that, how do you use a copilot?
38:01
How can you ensure that Copilot keeps your code private?
38:04
Because even people are protecting their code because
38:06
that is their IP that they want to save.
38:10
How do you think the governance in Gen AI compares to governance
38:18
with like traditional ML? I think it's still emerging.
38:21
I think folks are still getting their head wrapped around what
38:26
they actually need to do for generative AI purposes.
38:30
I think because trustworthy AI is still something that I think
38:34
is a growing field, especially here in the US, we don't have a
38:38
lot of trustworthy AI regulations on the books.
38:42
We're still merging through what are the standards and back
38:45
practises. Our National Institute of
38:47
Standards at Technology fairly recently just released their
38:51
their AI risk management Framework.
38:54
And so that is our first kind of go into saying here are some
38:56
standards around how you can build trustworthy AI systems.
39:02
And a lot of it does apply to generative AI as well.
39:06
But there are unique considerations.
39:09
I know there's a lot with hallucinations.
39:12
Generative AI can sometimes be confidently wrong.
39:15
And I think this is just an emerging societal issue where if
39:19
someone has a platform and they say something with confidence,
39:22
someone out there is going to believe them.
39:26
And we just aren't really at a place where we are saying, hey,
39:29
that that's wrong. You need to not blindly trust
39:34
everything that not just a machine learning model tells
39:37
you, but also that you you can't blindly trust everything that
39:40
you read on the Internet. So there's definitely a need for
39:43
literacy and some or analytical literacy and some understanding
39:48
around that. But I think, you know, there are
39:50
some unique considerations. There are some general
39:53
considerations in the trustworthy AI space that still
39:55
applies. But it's definitely still
39:58
emerging as people move along. On the technical side, you have
40:04
simple questions like now we want to introduce a vector
40:07
database, right? Who's going to do the SLA for
40:10
that? How do you going to do the
40:11
backup? What if we have two departments
40:15
sharing the same one do? How do we ensure the data
40:18
doesn't cross pollinate or even if you go away from embeddings,
40:23
go to something like where you want to enhance the RAC from a
40:26
data catalogue or an SQL query, right?
40:28
How do we make sure that the calling application can only
40:33
retrieve the data that it's supposed to and not just
40:36
everything within the organisation and just leak
40:39
stuff? These are like questions we have
40:42
been dealing with in the envelope space, like we know how
40:45
to document training data and how to move from one step to the
40:50
next. But in the Jenii space, it's
40:53
much more Wild West right now. And then also on the monitoring
40:58
side, how do you monitor just text, right?
41:02
Do you use like a LLM that then scores that?
41:08
Or do you try to break it down like most chat interface if like
41:12
it thumbs up, thumbs down. So you break it down into a
41:14
classic classical ML kind of binary target variable or do you
41:20
have like actions that a user can take based on what the LL
41:24
recommends and you track that, right?
41:26
How do you kind of translate this hard to grasp text and put
41:32
it back into something you can understand and track over time
41:36
to see the model is actually performing in a way you want it
41:40
to perform even outside of all the ethical concerns, right?
41:44
Just from a pure technical aspect, how do you monitor
41:49
random text basically? 100%, I think the evaluation
41:55
piece is so unsolved right now. And what you get is you're
42:04
seeing a lot of companies pop up because there's this gigantic
42:09
gap. Nobody really has it figured
42:11
out. And so you get companies that
42:13
are coming up and saying, all right, we do evaluation.
42:17
And we've done a few surveys in the community that have been
42:21
about evaluation. And what I've seen is sometimes
42:26
you'll get people who say, yeah, evaluations important to me, but
42:31
I'm not doing it. Like I have more important
42:34
things on the priority list that I need to do before I do that.
42:37
And then other times you just get the thumbs up, thumbs down,
42:41
which is OK, but that's not really that good of a signal.
42:46
And so that it feels very flawed in so many different ways.
42:51
And then also you're probably doing batch evaluation.
42:57
So you're not really doing it in real time.
43:00
So you can't tell if somebody just got a horrible answer.
43:04
You, you can see that maybe the day after or a few hours after,
43:08
whenever you're doing your batch, maybe a week after, who
43:11
knows. But I, I was recently in San
43:14
Francisco, funny enough, and talking with one of these people
43:17
that created an evaluation company and she was saying, and
43:21
her name was Brooke. And she was saying that for her,
43:25
it's very important to understand what the metrics are
43:28
that you're trying to evaluate on.
43:31
And she comes from the autonomous vehicles world.
43:34
And she was saying that you want to look at these evaluation sets
43:40
as like corpuses of data as opposed to a one to one type
43:44
thing saying, oh, was this answer good or was this answer
43:47
bad? No, you want to try and see, OK,
43:50
what is our target answer or what are we trying to go for?
43:54
And then and can we make that as like pointed as possible?
43:58
And then can we see all of the answers and view them like, were
44:03
there anomalies or are there statistical outliers on that?
44:07
As opposed to what I think a lot of people are doing right now is
44:10
saying like, OK, here's the output, here's what I was going
44:14
for. I guess that looks all right.
44:18
And then you bring up another one.
44:20
Here's here's the output. And, and just going off of
44:22
vibes, right? But the.
44:25
Vibes one is even worse when people go off the standard
44:28
benchmarks that are out there, right, and they just pick like
44:31
the winner in three benchmarks and it's so random.
44:36
Yeah, it's it's a wild space and it's so use case dependent, as
44:42
you mentioned, right? You really have to trill it down
44:44
into your specific use case. And then also it becomes the
44:48
question, are you also evaluating your embedding model
44:51
right or you just evaluating your LLM?
44:55
Yeah, exactly. Are you evaluating the
44:58
information that's retrieved from the vector database?
45:00
Sorry, what? That you have to evaluate things
45:03
at every step along the way. And you always have to those
45:08
free chat bot questions I would say in mind, which is kind of
45:13
they are traditional, right? Even when we were doing the more
45:17
like old school rule based chat bot, it was always, it's my chat
45:20
bot talking about something that it doesn't have an idea about.
45:24
Because marketing started a new campaign.
45:26
Now users ask the chat bot and has no clue about that new
45:29
campaign, but it will answer anyways, right?
45:33
Or is it something where the user asked the question, the
45:36
chat bot should have the answer, but it didn't deliver it?
45:40
Or is it deferred in kind of the worst category where we think we
45:45
delivered the correct answer, but the chat bot and the user
45:50
didn't accept that answer and didn't take it or didn't do
45:54
something with it. And how do you keep those free?
45:58
Kind of. The first one is rather easy I
46:01
would say, but the other ones become more and more complex as
46:04
you have a mature system. And then you start to wonder, as
46:10
you get deeper into needing that need for truth, at what point is
46:17
the number of guard rails you put around the large language
46:20
models output just start to become its own roles based
46:25
approach? We're almost, I've seen examples
46:28
where individuals like we are chatbot is what our large
46:31
language model output is 100% true and we know it because
46:34
we've compared it against this and this and we know that it is
46:38
correct. So there's no hallucinations.
46:40
And at that point it's you've almost just rebuilt a rules
46:43
based approach to confirm that my chatbot will never be RAW.
46:49
Yeah, that is that is a great point where you just have to
46:54
hard code in all of these different pieces or all these
46:57
guard rails and you end up realising that maybe it was
47:01
easier if we just changed the interface so it wasn't a chat
47:05
bot and it was just point and click.
47:09
And then you wouldn't have to even deal with all of the
47:12
unreal, the actions of a large language model.
47:16
That's one of the things that I I don't know about you all but
47:20
art and honest question, are you bored of chat bots yet?
47:24
Because I 100% AF. It's definitely everywhere.
47:29
I think everyone's scrambling to find a way to make business
47:33
value out of generative AI and they're throwing every use case
47:36
that they have at it. And sometimes it's just a
47:40
traditional machine learning or roles based approach would have
47:42
given them much more lift. But there are, I've been
47:46
chatting with so many users who are scrambling to say, hey, my
47:49
executives want a generative AI system.
47:52
This is what they need. And I'm trying to figure out how
47:55
to implement it based on this particular use case.
47:58
And you you're looking at it like, that's a little bit of
48:00
overkill. I'm so glad to see that so many
48:03
people are interested in AI, but I want to tell people that it's
48:07
not just generative AI. That's not the only AI there is.
48:12
I've seen on TikTok where someone's claiming that they're
48:16
older than AI. It's like this 20 something year
48:19
old's like I'm going to tell my kids I'm older than AI and
48:22
everyone's just like you're, you're not.
48:27
Yeah, they need to hear about the AI winner.
48:31
The the worst offender I have seen is where the basically
48:35
build out the chat bot to ask for variable inputs that could
48:39
have been just a slider or a button, right?
48:43
If you could change around. But they were like, yeah, you
48:45
can now talk to them. I'm like, no, just give me a
48:47
slider. I just the values from 1:00 to
48:51
10:00, like just give me that slider range.
48:54
I don't care. 100% that's what I feel like you're.
48:56
Making it worse. Sometimes, yeah, the just
49:01
because we can use AI doesn't mean we should.
49:05
And a lot of times, and I guess that's what happens at the
49:08
beginning, people got really excited with the technology and
49:13
the potential. And so they thought, wow, this
49:16
is going to. And also you have a lot of
49:20
onlookers saying this is going to change the world.
49:23
This is going to change how we do business itself.
49:25
This is going to change the world of work.
49:26
All of that type of narrative is not helping one bit.
49:31
And So what happens the by product is that you have
49:36
developers putting Chen AI into whatever they can.
49:40
And I think the epitome of that was somebody commented in the
49:45
community how Meta now has like Meta AI, right?
49:50
And Instagram and WhatsApp. And somebody was saying in the
49:54
community Slack, this was just some PM having to hit their
50:00
KPIs, like, let's all be honest about it because nobody's going
50:04
to meta and being like, all right, Jen, AI search type
50:07
thing. All right, I, I don't know who's
50:10
using that. If you are, please tell me and I
50:13
would love to hear about your use case, but I definitely am
50:15
not. Yeah, I mean I turned off the
50:19
like Google and Bing features for search.
50:22
I just just give you a straight up search please.
50:24
Thank you. Yeah, I, I often wonder if am I
50:29
the a weird one because I did the same and I'm like, am I
50:33
missing out on something here? When I want to do the AI stuff,
50:37
I'll go to perplexity, but when I want 10 links or I just want
50:41
links in general, then I go to Google because half the time
50:45
it's that I need to navigate to a website and do something on
50:49
that website. Like for example, today I've, as
50:53
you know, David being a German resident, Telecom is one of the
50:59
mobile phone providers and I had to go to telecom.de today.
51:04
I don't need AI to tell me what is on that website.
51:08
I if, if in the future we can have agents ideally that will go
51:15
and deal with all the crap that I had to deal with with Telecom,
51:18
that will be splendid. But for now, I don't think we're
51:21
there. And so that's kind of like the
51:25
segue into my next question. Have you seen people trying to
51:29
implement agents and have you seen any success there?
51:34
I I've seen some interesting use cases with agents.
51:37
Some of them are very well guardrailed to the point that it
51:43
does seem very rules based. I know that is something that
51:48
some folks are exploring for how to leverage inside software
51:53
that's for advanced users so that it's a little bit easier
51:56
for folks to figure out. So I don't have to know how to
51:59
use this advanced software. I just type in, this is what I
52:03
want to do. It's done it for me.
52:06
It's given me some suggestions for what I should do next.
52:09
So I've seen folks trying to use it to make things even more user
52:13
friendly since we all know how to type in questions as Google's
52:17
really good for that, for teaching us that piece.
52:19
But it seems to still be ongoing and it seems to be very for
52:25
specific tasks. So it's almost like you have to
52:29
think ahead of time to understand what an individual
52:31
might want to do and what we want to make easier for that
52:34
individual. So I have to think as a newbie,
52:38
as someone who's never seen this software before, how do I make
52:41
this an easier workflow for them so that I can think about it
52:45
ahead of time? Almost like a new way or new
52:47
user interface into using some software I was in.
52:51
Talks with a customer where they implemented again in kind of
52:56
trying to automate more of their processes around incoming mail
53:01
that they have and they reverted back to a simple agent that just
53:07
had the choice between like two or three actions and then did
53:10
those actions and then was done. They had a much more
53:13
sophisticated approach that had like multi agent that handed off
53:17
things, but it just broke too many times and they were like we
53:22
can't babysit this thing. So when when you're like into
53:26
multi step agents or multi reasoning things, then at least
53:31
in the the things I've seen in actual production, it just broke
53:35
down once there was some complexity introduced.
53:39
I had one demo built internally that worked great for me because
53:43
I understood how all the agents were laid out, how they could
53:47
interact and when I was chatting with the system it worked every
53:51
time. Florida State.
53:53
I handed it over to a colleague and he came back to me 5 minutes
53:56
later and I was like this thing is garbage, it doesn't work at
53:59
all, you have no understanding of how to talk to that thing and
54:04
then I realised maybe I made the wrong thing.
54:09
Yeah, just go hand it over to Reddit to stress test it.
54:11
They all come back with a lot of great.
54:16
Yeah, that's what I think about. It's of course in this walled
54:22
garden. It works beautifully with the
54:26
user end of one, right? And then you give it to the next
54:30
person. And since it is open-ended, I
54:34
think that's where a lot of times we fall flat because it is
54:37
so open-ended. And that's why I like the fact
54:40
that there's a lot of people trying to change the interface.
54:43
So it's not a chat bot. It's more point and click and
54:48
then potentially you have those very clear verticals or task
54:54
lists that the agent knows what to do.
54:57
And like you were saying, Sophia on almost like making a very
55:00
complex programme a bit more user friendly is it feels like
55:11
it's just a little bit more advanced help button it's.
55:18
Almost like we're bringing back Clippy.
55:20
Yes, good old Clippy, we totally should have.
55:24
That is so funny. But I do see the need in that
55:28
like especially for me when I was trying to learn how to use
55:32
Photoshop years ago, I would have an idea on what I wanted to
55:36
do. But Photoshop is so powerful
55:39
that you can spend hours just flicking on different ways to or
55:45
different effects that you can add to the photo.
55:50
And I would end up going and having to type into Google or
55:55
YouTube, usually YouTube and get a tutorial on how to do XYZ.
56:00
But the hard part wasn't really figuring out how to do it.
56:05
The hard part was trying to put into words what I wanted to be
56:10
done. And so because that was the hard
56:13
part, this new interface that we have or this new way of dealing
56:18
with it, if it's through natural language and a chat bot, it
56:22
doesn't solve that hard question of like, I have this idea in my
56:25
head. I don't know what it's called.
56:26
I don't know the special language for it.
56:28
I just want this to happen. And because I can't reach into
56:33
the screen and like shape it how I want, it is still going to be
56:39
very difficult with the most advanced form of LLMS out there.
56:45
Often kind of like a brainstorming thing, kind of
56:48
trying to get it. You give me the thing I want to
56:51
then go look up, right? What's that specific industry
56:55
term for X? Give it to me please.
57:00
Totally, 100%. Now that we've discussed a few
57:06
of these challenges, I would love to know, just like when
57:11
you've seen people that are excelling on their AI or ML
57:17
journey, are there common themes that come up again and again?
57:25
Like have there been some things that have cropped up And you've
57:28
said, wow, I noticed that this tends to happen in very mature
57:35
teams or very high performing MLAI teams.
57:39
I think what I've seen is excellent lines of
57:43
communication. They, they're working together.
57:46
They aren't tossing things over the fence saying this isn't my
57:49
problem. They are really working as a
57:52
team to bring in these different individuals to share their
57:57
knowledge. You know, hate to use the
57:59
corporate lingo, but breaking down those silos so that we do
58:02
have a cohesive team. They are experimenting.
58:06
They are doing proof of concepts.
58:08
They are looking to see what has potential value.
58:13
How can we start small and bite off something that we can
58:16
actually chew and become really good at that and then start
58:20
moving into seeing how can we make this something that has
58:24
business value? What are the next steps?
58:27
They have really, well, nice processes that are flexible but
58:32
offer some levels of standardisation.
58:34
So they have a series of steps that they feel comfortable
58:37
following, but they're happy to adjust those steps over time to
58:41
become to meet specific needs. And so they're not building all
58:47
of that out at once or but they are trying to grow over time.
58:51
They're not automating everything all at once.
58:53
They're looking to see what are the phases we need to do and how
58:57
do we grow over time instead of just biting off everything at
59:01
once. And they are looking at their
59:04
successes, celebrating it, but continuing to look on and see
59:07
what's down the line. What else can we try?
59:10
What other experiment can we move forward with?
59:14
And sometimes, you know, it doesn't work out.
59:16
We don't have to sync, stay attached to that sunk cost.
59:20
We can take those models out of production and retire them if
59:24
they're not valuable. Nice call back to the beginning
59:27
of the episode. I like it.
59:29
I think Sophia already hit on the major points for me.
59:32
One, one that I kind of always see is when the data scientists
59:38
work closely with the data engineers, right?
59:41
And they think of the not just the cool model that they're
59:44
going to build, but how they can actually then run it in
59:48
production and have those points where they if they re have to
59:53
retrain the model that it's not like they have to start from
59:56
scratch again, But they have a process for that in place.
1:00:00
If they are scoring the model in batch that they have a point of
1:00:05
integration with the data engineer.
1:00:07
So it's not something completely novel, but there is a clear
1:00:10
point where. This hand off happens and where
1:00:14
the integration then actually works in production.
1:00:17
So if those two teams, they fit on the same floor or share kind
1:00:22
of a common team structure, then I always see them be successful
1:00:28
across like one or two use cases, but then really scaling
1:00:32
up to more of them without getting Hanford down because
1:00:36
they have to reinvent the wheel every time.
1:00:38
More of like a kind of like a platform approach that they take
1:00:41
to these problems and they have clear points where those teams
1:00:46
interact and then deliver. Well put well, thank you David
1:00:52
and Sophia for coming on here. This has been awesome.
1:00:55
I love chatting with you just to hear all of your diverse
1:00:59
experiences and then also hear that it feels like we have been
1:01:07
talking to the same people almost these these problems that
1:01:12
I've been hearing a lot of people having.
1:01:14
They are not unique. You all see it out there and we
1:01:18
get to triangulate them on this call.
1:01:20
So it's been a pleasure. Thank.
1:01:23
You for having us.