Latest / Agentic Conversations (formally mlops.community) / Who's MLOps for Anyway? // Jonathan Rioux // #261

Transcript

Who's MLOps for Anyway? // Jonathan Rioux // #261 · Agentic Conversations (formally mlops.community)

0:00
So my name is Jonathan Ryu. I'm a managing principal for
0:06
artificial intelligence department at Epam Systems.
0:10
I take my coffee black. I, I, you know, recently got
0:15
wean from sugar in my coffee. I no longer like it.
0:19
To me, the important thing is that it needs to be hot.
0:21
I cannot do lukewarm or cold coffee.
0:25
ML OPS Community Podcast is back in action.
0:28
I'm your host Demetrios and talk with Jonathan today.
0:32
We did not conflate. We did not ML OPS and business
0:38
value. That was one of his key
0:40
takeaways that I am going to probably be repeating time and
0:45
time again. Just because you have ML OPS
0:47
doesn't mean your ML projects are going to be successful.
0:54
In other news, I tend to talk about on this podcast how you
0:59
don't see fraud detection as a service type companies out there
1:05
because oh it's too hard or the data's too messy or this I was
1:10
completely wrong. I just found out today there is
1:13
a fraud detection as a service company.
1:15
It is called Cubiant cubent maybe is another pronunciation
1:20
of it. Stock ticker is KBNT.
1:22
It was so successful it I PO D and everything was on the stock
1:27
market until last year when it got delisted.
1:29
So yeah, I was wrong. There we go.
1:35
Let's get into this conversation with Jeff.
1:44
We got to start with this because we were in California
1:49
two months ago, three months ago.
1:50
I can't remember exactly when but we sat down.
1:54
You guys had a very nice place rented out.
1:58
Our man cave. Yeah, I appreciate you inviting
2:01
me there. I felt like AVIP, it was the
2:03
only, it was the only meeting that I actually took when I was
2:06
at that conference and I got treated like royalty.
2:09
So it was awesome to see you in your element.
2:12
And you just told me now that you don't think envelopes is
2:17
sexy. So I have now gone down on the
2:20
People sexiest man of the year charts, apparently because I am
2:26
very attached to the ML OPS term.
2:30
But why don't you think it's it's sexy?
2:34
If I define ML OPS as not being sexy, it's it goes in the same
2:38
vein. That's like in dev OPS not
2:40
really being sexy or, or software engineering as a whole
2:43
not being extremely sexy. I feel like a lot of individual
2:48
have coined the term ML OPS as kind of a guarantee of revenue
2:52
or a guarantee of ROI on their ML initiative.
2:56
It's a different ball game. You know, if you get to think
2:59
about it, or at least my definition, I really see ML OPS
3:04
as a set of principle, a set of process, you know, tooling that
3:09
are all going to work together to facilitate like how quickly
3:15
you can take an idea and get it to production and maintain it in
3:19
production without having crashing on you.
3:23
It does not give you ideas. I mean it I I was discussing
3:28
with a few of my colleagues a terrible idea implemented using
3:32
ML OPS principle is going to yield a terrible product, but
3:36
faster and more reliably. With the with the DevOps
3:44
principles in hand. And that is hilarious.
3:48
So huge thing you just said there is that do not conflate ML
3:53
OPS with ROI. Just because you have very
3:57
strong ML OPS practises and processes in place, that does
4:02
not mean that you're going to have a successful ML product or
4:07
product in general that leverages ML or AI or whatever
4:11
magic bullet you think is going to save you. 100% I mean,
4:16
you're, you're looking into this.
4:19
I, I find it absolutely hilarious to see organisation
4:23
that say, you know, were ML OPS compliant, whatever that means,
4:29
But then they're data scientist or ML engineers are working in
4:34
kind of a shared service and becomes that kind of arbitrator
4:38
of like what is good enough to become an ML product.
4:41
The way like a couple of years ago we were talking around, you
4:46
know, dev OPS for ML before ML OPS became kind of bonafide
4:51
accepted. And to me, it was always about
4:56
putting, you know, business forward people to technology
4:59
forward people. And in the case of ML OPS,
5:02
analytical and, and ML forward people to be able to create
5:06
something. Where did we get lost in the
5:10
process? Saying that, you know, you need
5:12
a centre of excellence that's going to control all of the
5:15
intake for ML and it's going to evaluate it with an arbitrary
5:21
framework to be able to determine the value, which I
5:26
think in practise is whoever screams the loudest or you know
5:31
who, who gets to own the budget. But I feel like this way we're,
5:37
we're going further, further away.
5:40
Pardon me from from the the original piece which was infused
5:45
embed ML where it makes sense into the product development.
5:52
And and I think it's because a lot of organisation don't really
5:55
know what to do with ML. So by putting somebody who's
5:59
accountable on it, I think the natural tendency is to want to
6:03
centralised everything. It's a little bit the same thing
6:06
with data organisation. If you have a data team who
6:09
doesn't understand how the data is being produced and who has no
6:12
control in fixing the source, how can you hope to have data
6:17
quality? Same principle.
6:20
So bit of a hot take there on organisations don't know what to
6:25
do with ML. But before we dive down that
6:28
rabbit hole, you sparked a gigantic thought.
6:34
I may have my next business idea thanks to you which is ML.
6:39
OP certified.com just bought that domain while we were on
6:43
this conversation. Now I am going to go sell that
6:47
to these businesses and say you know what you need, you need
6:50
some ML OPS certifications and I will look at your processes and
6:55
tell you if they are in line with the best standard practises
6:59
that we are seeing out there. Hey, you don't know at one point
7:04
you need to make your. You need to make rent.
7:09
My daughter needs diapers. That's it exactly.
7:11
So let's go down in this rabbit hole of what makes you say that
7:15
folks do not know what to do with ML.
7:18
I think this has been magnified and I mean, we can talk probably
7:23
about the two last hype cycle, you know, deep learning.
7:27
Couple of years ago, I, I actually joined EPAM, my current
7:33
employer when at, at the hype of, I think it was around the
7:38
time where they were saying like blueberry muffin versus dog
7:42
using computer vision. Like that was kind of the, the
7:45
thing that a lot of people were going around.
7:47
And now we're having large language model and generative
7:51
AI. And, and to me, I think there's
7:54
a, there's a big discomfort because we're back to use case
8:01
chasing and a lot of conversation are oriented in the
8:06
fact of, hey, I have this new technology, where can I apply
8:10
it? And in my opinion, this stems
8:16
from a, a misunderstanding a little bit of the technology,
8:20
which is totally normal. I think there's a lot of hype in
8:24
in what it can and cannot do. I think people have incentives
8:27
to be able to vote from a personal standpoint, but also
8:30
some companies are over advocating the capabilities of
8:34
of generative AI, let's say in that kind of circumstance.
8:37
But, but I think it leaves a lot of business leader kind of
8:41
flipping the script of instead of focusing on what they know,
8:46
which is their business and seeing what are the problems
8:50
that can be solved using this technology and, and looking at
8:53
your portfolio from a holistic fashion.
8:57
We're seeing the opposite where people are starting with the
9:00
technology, which they understand partially and then
9:06
trying really, really hard to think about compelling business
9:09
problem or even worse in certain cases, like, oh, what kind of
9:15
white space innovation we can do where the biggest benefit that
9:20
we see companies that I mean, our company as, as an example,
9:23
like our, our biggest usage of generative AI is Spermi in the
9:27
grey space innovation. Like what are the process that
9:30
are cumbersome? What are the process that are
9:34
tedious, repetitive that we can kind of do micro optimization if
9:39
you allow me to like blend 2 words together.
9:44
But for us, it was always evident you, you know, if you're
9:47
in control of the road map and the problem that you have and
9:49
you have a good grasp on what are your opportunity for
9:54
improvements. You don't really need to go into
9:58
that, that all out panic. And you add on top of that, you
10:04
know, adding a generative AI centre of excellence, which
10:08
means that your data scientist who already have permeated the
10:12
industry now are kind of working double shift because, you know,
10:16
you have your, your regular job where you're part of a product,
10:19
you may follow ML OPS process. Some companies are lucky enough
10:23
to be able to have data scientists that are horizontally
10:26
laid out. And then you have that centre of
10:29
excellence, which means that you're, you're uprooting how
10:33
your company is working just to be able to adopt that
10:36
technology. So I, I think a lot of companies
10:39
are not AI ready and a lot of companies that were, you know, a
10:43
little bit more AI mature Gen AI is kind of disrupting them, but
10:48
but not because the technology is so promising and wonderful.
10:51
It is promising and wonderful, but because I think the natural
10:55
reaction is decentralised to control, to limit and and
10:59
ultimately it it sets you back. So basically it's like you're
11:04
saying, folks are taking their eye off of the ball and instead
11:09
of spending time and energy and cycles in what they know best,
11:13
which is product or just their business line and what the
11:17
business is actually focused on now, they're thinking, oh, this
11:22
AI stuff is so cool. What if we put it into here?
11:25
And what if we put it into there?
11:27
And mind you, nobody's actually asking for that.
11:32
Nobody's thinking, you know what, I would love chatbot that
11:37
actually responds to me when I pop up onto your website, stuff
11:41
like that. It's not that they are going
11:44
from those principles of what is the customer want and then is AI
11:50
the best way of solving it? It's more thinking we've got AI
11:55
now, we might as well use it and especially if you're in another
11:59
one. This might be my ultimate dream
12:01
job. I want the title of AI Centre of
12:04
Excellence at some company. That sounds like an incredible
12:07
job. I will be Chief AI Centre of
12:11
Excellence if you annoyed me. That title sounds like the most
12:17
bullshit job ever, but I agree. So.
12:22
So anyway, if it is if if that is my title, I am going to of
12:28
course champion for all these different use cases of how we
12:32
need to infuse AI into all these processes, right?
12:35
Because it's in my best interest.
12:37
Yeah. But then you get into the very
12:39
real problem of you're, you're ultimately centralising all all
12:45
of that innovation for an entire company, which means that you
12:50
need to acquire more than a working knowledge on every
12:55
business function that is coming to you with use case.
12:58
Because otherwise it's very difficult for you to understand,
13:03
appreciate and empathise with the value that this can bring.
13:08
There's also a tremendous amount and this is something that we
13:12
see quite, quite a bit is separating, you know,
13:18
optimization versus potential revenue generation, calibrating
13:22
some of the risk that you can have.
13:23
So there is a place for deeper knowledge on, on what is the,
13:28
you know, the application of generative AI and having a
13:31
little bit of a more technical bias.
13:33
But this is doing in conjunction with, you know, your, your, your
13:39
business functions and you, you, you can't remove them, the
13:43
agency, you can't prevent them from innovating otherwise.
13:48
We're you know, we talked a lot about shadow IT, but we're
13:51
seeing a lot of like shadow AI, shadow Gen AI and and to me,
13:55
this becomes the that is step one in, you know, having a very
14:03
unfortunate thing that's going to happen being a word.
14:06
We're seeing a tonne of use case that, you know, public chat bots
14:11
that are making promises, whether we're not supposed to
14:13
make promises, data leak, those kind of thing.
14:17
And and I think it is related. I I'm not involved in those
14:22
thing. I like the thing that, you know,
14:24
I keep a tidier ship than this, but we I, I think it's easy to
14:31
explain is you're you're in a business function, you really
14:34
want to be able to launch something.
14:36
There's someone somewhere else in a company that's saying this
14:38
is not really a priority. You're looking at this as an
14:42
opportunity to out innovate the innovation lab.
14:46
You go forward. Yeah, and then you sell cars for
14:50
$1.00. You sell cars for $1.00 or, you
14:53
know, in, in, in Canada. We, we had Air Canada promising,
14:58
you know, refunds where there were no, no refunds that, that
15:01
were supposed to be done. And we're, we're seeing, we're
15:05
seeing a lot of funny things. I mean, internally we experiment
15:09
quite a bit with like some patterns that we hear.
15:12
We, we test a lot of LLM. We have some pretty funny
15:16
stories. We definitely dog food a lot of
15:21
of what were like try to come in front of a client and and have a
15:25
compelling point of view. So I I failed a lot with LLM but
15:33
you don't usually behind closed doors.
15:37
It's yeah. So you protect your ass.
15:39
That's my way of doing it. Just yesterday I read about how
15:42
Slack rolled out their AI feature and they didn't realise
15:48
it, but you can prompt inject it so that it gives information
15:53
from private channels into the answers of any questions that
15:58
are asked to the Slack AI. So it's like there's stuff that
16:03
are in those private channels that may want to be kept
16:07
private. And this is Slack.
16:10
Like, we've had really great engineers and data scientists on
16:14
this podcast from Slack. They're incredible.
16:17
Yeah. And they're still not able to
16:19
fully think about all the different repercussions that
16:22
this can have when it comes out. And I think we're going full
16:25
circle on that frame is to me, this is, this is a gap in your,
16:32
let's call it LLM MOPS pipeline because I, I see this as
16:37
building a tower. You know, you, you have your dev
16:40
OPS practises, you know, your version control, good coding
16:43
hygiene. Are you using, you know,
16:46
extension that can augment your productivity and so on and so
16:50
forth, like personal development, but also team
16:52
management. You have your data OPS, which is
16:55
where you know, data as a product, what can be used when
17:00
clear governance. There's still a lot to explore
17:04
there. I, I think that we're, we're
17:06
doing a pretty bad job at weaponizing data properly.
17:13
We moved to ML OPS, which is your entire life cycle model
17:16
management, but also your entire lineage, making sure that you
17:20
know, you can have reproductive reproductible builds when, when
17:26
a model is put in place. And, and then you go with LLM
17:28
OPS, But you get to think about it like to me, a data leakage.
17:33
I, I don't know exactly, you know, how they've architected
17:37
the product, but somewhere at some point there, there's an
17:41
access control that is failing. And I'm not saying that it's
17:45
easy and, and they're under a tremendous amount of pressure,
17:52
but it becomes like it, it's easy to understand, at least
17:56
conceptually, where it failed. And it's a it's a good reminder
18:01
that even the the strongest company, the company that
18:05
employs the best engineer and that has the most budget is not
18:09
immune to those lapses in in their process.
18:16
So you mentioned how centralising things with this
18:20
centre of excellence is not the ideal scenario because the
18:26
centre of excellence now has to understand on a deep level each
18:31
aspect of the business, almost more than the actual people that
18:38
are doing that. Because they have to understand
18:41
what AI is capable of and where, where the rub is for the
18:44
accountants or for the sales people or whatever it may be.
18:48
How do you recommend going about it so that it isn't centralised
18:54
but it's also successful? I mean, I like to you say the
19:01
expression like are you a left-leaning data scientist or
19:04
right leaning data scientist? So like are are you looking
19:07
into, you know, you want to be able to centralised by
19:10
competence, which means a little bit what we've mentioned and I
19:15
think we see that it does not work in practise.
19:19
But I also think that the complete opposite, which means
19:23
you each business function is is responsible for the entire data
19:27
MLLLM stat. It is also out of reach for most
19:32
organisation. You can't have, you know, one
19:38
data scientists or a series of data scientists across multiple
19:42
functions and it becomes really hard to encode best practises
19:46
and you end up with, you have a tonne of data scientists all
19:51
around the organisation, their governance become difficult,
19:54
dissemination of best practises becomes hard.
19:57
This is where you get, we're seeing sometime, you know hand
20:04
different environment. With a different set of
20:07
governance, it's, it's really, really tough to control your,
20:12
your enterprise risk on, on that front.
20:16
So to me right now, like in, in August of, of 2024 or I think
20:24
September of 2024, when this is going to, to go live, I, I think
20:29
that the, the best model that we can have is, is some sort of hub
20:34
and spoke where but a little bit of a, a different approach where
20:41
you're, you're not seeing the centre of excellence as a
20:44
governing function, but more of a support function.
20:49
And, and I think that that's what's hard for a lot of people
20:52
because they see the centre of excellence as a way to augment
20:56
or profile. But you know, you're really
20:59
supporting, you're, you're making sure that everybody is as
21:03
the tool that they have a, a little bit kind of a, you know,
21:07
a proverbial scrum master, your job is to facilitate and make it
21:11
easy to do the right thing. So I think right now this would
21:15
be the model that I have a tendency to recommend.
21:18
We do have a tendency to go in between, like as LLM is going to
21:22
become more commoditized, easier to understand, more permeated
21:28
into the industry, I think that we're going to go more and more
21:31
decentralised and then something else going to come in and you're
21:34
going to see a huge shrinking that or we have a huge skill
21:38
gap. We don't have a lot of people
21:39
that understand this. We're going to create a centre
21:41
of excellence that's going to govern whatever new hype cycle
21:44
is going to come and we're just going to repeat that process
21:47
again and again. Again, anybody out there that's
21:52
hiring for Centre of Excellence chief.
21:57
So that is exactly that hub and spoke model is exactly what I
22:04
think Uber did. When I was reading their blog
22:07
post on how they've evolved Michelangelo, which is arguably
22:11
one of the most famous ML/AI platforms out there.
22:16
They said that and they mentioned how you have very
22:21
experienced and advanced folks that are are kind of pulling
22:27
features or needs or capabilities out of the
22:31
platform. And what they had to do is
22:35
recognise what use cases they were trying to create with these
22:42
feature requests and see how much money that was making for
22:48
the business. And they, one thing that I
22:52
thought was fascinating, and I repeat this quite a bit, is how
22:55
they were able to figure out which models, which use cases
23:01
were generating what kind of ROI for the company.
23:04
And so they had different tiers of babysitting, we could say,
23:10
for different models. And you see that.
23:13
And it kind of is, is what you were saying, because they have
23:16
the Michelangelo platform and it helps support 99% of the use
23:21
cases. But if there are people that
23:23
want to do their own thing, they can.
23:26
And Michelangelo just tries to make it as easy as possible for
23:30
those people to. Then when they're done doing
23:33
whatever it is they're doing, they can bring it into the fold
23:37
and run it on Michelangelo. I, I, I love that example
23:43
because to me did this. This exemplifies perfectly that
23:50
support model that, that we're trying to get it.
23:53
And I mean we're even looking at this and Uber is an example of a
23:59
very advanced company. But I think a lot of, a lot of
24:03
individual are working, whether it is banks, insurance company,
24:07
pharmaceutical publishing, they, they, they probably don't have
24:13
the, the VC funding that to, to invest or nor should they.
24:18
Like I, I think we have to acknowledge that Uber made a
24:22
very risky bet in creating their bespoke architecture and they
24:26
were one of the first to, to do so and to be very public about
24:30
it. I think the market is catching
24:32
up. I mean, Uber has open source
24:35
significant portion of their tooling.
24:38
There's other players in the market, but I think you have to
24:43
think of it as the jobs to be done.
24:46
So what what are the process that you want people to follow
24:51
up? Where are your your source of
24:54
truth? And you know, the way, the way
24:58
that we look at this, you know, we talked a lot about ML apps
25:02
around people process platform. And I think a lot of people are
25:05
are just throwing this around, but people follow process and
25:10
the platform enables those process.
25:12
But at the end of the day, you can have the best platform in
25:15
the world, you can have the best people in the world.
25:19
But if there's no clear way of doing things or as you said,
25:22
when you need to go off script, how do you plug in and plug out
25:27
of of that that process and, and identify those 95 to 99%
25:36
activities like you, everyone's going to need to host a model.
25:41
How do we make it as simple as possible?
25:43
What is, you know, the SLA that we need to provide?
25:47
Do we want to create this in house, you know, deploy our own
25:50
model in Kubernetes? What's going to be the scale
25:53
gap? You know, a lot of people are
25:56
going and they're saying, oh, you know, we don't want to pay
25:58
for, you know, either a third party tool or, you know, hosting
26:02
is very expensive. So we're going to try to do it
26:04
ourselves. What you have to acknowledge
26:06
that if your team of data scientists are not, you know,
26:12
cloud competent, it's going to be difficult for them to go and
26:18
they will be frustrated because as long as it's going to work,
26:21
it's going to be wonderful. But the day that you're going to
26:24
have to troubleshoot something and, and the abstractions going
26:29
to fall, you, you also need to be mindful that everything is
26:35
good as long as it works. But, but, and I, I talk from
26:40
experience, like my previous organisation, we, we made the
26:44
choice of, of deploying our, our models on, on ETS and it was
26:52
wonderful. It was really fun.
26:54
I mean, it's very gratifying to be able to deploy.
26:57
We got some problem where our, our scaling was not working
27:02
properly. It was very interesting and
27:04
fulfilling like intellectually to be able to learn about this.
27:08
But ultimately I look at this from from an, an OK R
27:12
perspective. I was unable to work on any
27:17
other thing. My team was 100% mobilised to be
27:19
able to fix this, which basically means a month later,
27:23
there's nothing new, no model refresh, no nothing kind of
27:26
coming into place. So it's, it's all a question of
27:29
cheques and balances. And I I like to use and remind
27:32
everyone of what needs to be done.
27:35
Make it as as easy as possible, make it as friendly as possible.
27:39
And if people want to be able to go off script, at least they
27:42
have a set of convention that it can hinge into.
27:45
God, just talking about that reminds me of a conversation I
27:48
was having with my buddy last week and it was a little bit
27:54
about how you have these companies that are spun out of
27:57
the what quote unquote top tier companies like the Ubers or the
28:03
Linkedins or the Googles or Facebooks, Meta, all of that.
28:07
The engineers at these companies say, oh wow, I have this
28:12
problem. And they create some open source
28:14
product and then it gets a bit of traction and boom, you know
28:17
the playbook, they go raise a bunch of EC funding and it's off
28:21
to the races. And my buddy was saying, man,
28:25
but what a lot of these folks don't understand is that not
28:29
everybody is a top tier engineer.
28:32
So they've built this product for their friends at the company
28:39
that they used to work at, knowing the skill level that
28:42
they were at. And they're not building it for
28:44
me. Like I am not that type of
28:46
engineer. I, and I'm being very candid
28:49
about it. I'm not going to be working at
28:53
Uber in their engineering department anytime soon or
28:56
Meta's engineering department, except these products are built
28:59
like I should know how to work it and I should know how things
29:05
need to function. And it's like you're saying,
29:09
like if you can just bring it down a few notches, dumb it down
29:13
for me, that would be incredible because you're going to have
29:16
much more success. That's going to be mass market
29:19
and you're going to have people that love your product because
29:23
of that user experience. I'm, I totally agree with you.
29:28
I, I think it's also like, I, I, I want to excuse a little bit
29:33
the behaviour of those folks because I, I also think that
29:36
it's, it's just very natural. Like I, I think anybody who's
29:40
been working in the space for a long period of time, especially
29:43
surrounded by their peers, sometime there's kind of a big
29:48
dichotomy like on, on one part you're looking around and you're
29:52
like, man, we're so late and everything.
29:55
But also, and, and I'm 100% guilty of this.
30:01
We have an overwhelming tendency to assume that everybody knows
30:06
as much on the topic we love as as ourselves.
30:10
So I mean, I'll use kind of very funny anecdote, but we had some
30:16
interns this summer and very eager to learn and, and very
30:20
excited. And they've done a lot of data
30:24
science, you know, in Jupiter notebook kind of analysis.
30:28
And then we get them, you know, kind of set up into the
30:32
environment and me and my team, at least for personal project,
30:37
like we have a pretty old school stack.
30:39
Like I'm still a big, big fan of make files with all of its
30:42
warts. I I just feel like, you know,
30:44
it's kind of a good thing that he can hinge ourself onto.
30:48
And they were all like, if you're talking about one of the
30:52
low lights of their internship was like, figure out all of
30:56
those tool version control, CICD, make files, automation.
31:01
They were just like, it's so much, so much.
31:04
To take all of no the first months so.
31:09
But to me, I was like, oh, what? Well, I've, I've learned this.
31:12
Like, you know, anybody can learn it, but you get to think
31:16
about that. Like, I'm happy to go on the
31:18
record that I, you know, it took me, I mean, a couple of years to
31:24
be able to adopt version control properly and then to be able to
31:29
understand CICD in a way that works for me.
31:36
Like not necessarily all of the intricacy.
31:38
It took me, you know, a little bit of time afterwards.
31:41
So I, my, my expectations of adoption are like, Oh yeah, man,
31:48
that should be enough. It's not like you're, you're
31:51
trying to change habits and it, it's one of the challenge.
31:57
You know, I, I, I do a lot of, of ML OPS consulting and
32:03
implementing the stack, suggesting a process map, doing
32:07
an analysis of like what is versus what should be doesn't
32:11
take a lot of time. You know, there's, there's good
32:14
internalised practises that you can go.
32:17
We, we take pride in nuancing a little bit, like not everybody,
32:23
you know, I'm, I'm not going to go and sell the same product to
32:25
everyone. Not everybody is in the same
32:27
boat. But to change management and,
32:30
and getting data scientists to adopt tooling.
32:34
And we have a lot of conversation down downstream
32:38
where, you know, we do a check in and they're like, well,
32:40
they're, they're like, I don't understand, like, you know, CICD
32:44
is not, not everybody's using it.
32:46
And it's like, take, it's going to take time.
32:48
You know, people need to, I almost feel like you need to be
32:54
frustrated enough with the current state so that you can
32:57
adopt the the change. But but ultimately everything
33:03
you know from you decide as a company to become more agile, to
33:07
put some automation to streamline your process.
33:10
This is going to cascade into individual team how they're
33:12
going to adopt this and then it's going to go to individual
33:15
developer and, and that process is like it can take months,
33:21
years, depending on how far you're you're going.
33:26
And I, I don't think we appreciate it.
33:29
Like, you know, you and I, we, we did all of this.
33:32
So now it's kind of like everybody should do it.
33:35
Exactly. So great points.
33:37
All right, real quick question for you.
33:39
Are you a Microsoft Fabric user? Well, if you are, you are in
33:42
luck because we are introducing SAS Decision Builder.
33:46
It's a decision intelligence solution that is so good it
33:51
makes your models useful because, let's face it, your
33:54
data means nothing unless you use it to drive business
33:57
outcomes. It's something we say time and
33:59
time again on this very show. But wait, what do you mean by
34:03
nothing? Well, SAS Decision Builder
34:08
integrates seamlessly with Microsoft Fabric to create
34:12
effortless business intelligent flows.
34:15
It's like having a team of geniuses you manage in your
34:19
pocket without all that awkward small talk.
34:22
With Decision Builder, you'll turn data into insights faster
34:25
than brewing a double espresso. And you know how much we like
34:29
coffee on this show? Visually construct decision
34:32
strategies, align your business, and call external language
34:36
models. Leverage Decision Builder to
34:39
intuitively flex your data models and other capabilities at
34:43
breakneck speeds. There's use cases for every
34:47
industry, including finance, retail, education, and customer
34:50
service. So stop making decisions in the
34:53
dark. Turn the lights on with SAS
34:56
Decision Builder. Yes, I did just make that joke.
34:59
Watch your business shine, SAS Decision Builder, because your
35:04
business deserves better than guesswork.
35:06
Want to be one of the first to experience the future of
35:09
decisions? Well, sign up now for our
35:13
exclusive preview. Visit sas.com/fabric or click
35:19
the link below. Let's change gears for a minute.
35:22
I want to talk about this idea that we touched on where you
35:28
mentioned how you're doing all kinds of POC's with folks in the
35:33
Gen AI sphere. And all of a sudden, six months
35:38
later, you got that POC up and running and people are coming
35:44
back to you saying, OK, so now what's the ROI on this?
35:47
You're looking around like, wait a minute, we weren't talking
35:52
about ROI. We're just trying to get
35:53
something up and running. You're explaining that a little
35:56
bit more, put a little more colour on that.
35:57
Oh, for. Sure.
35:59
So maybe like we can contextualise a little bit like
36:04
the the situation you're, you're an enterprise and I mean this
36:09
could be my company, this could be any companies that that we do
36:11
work with. One of their primary goal is to
36:15
get familiar with the technology.
36:17
So they're, they're selecting use cases, they're prioritising
36:21
use cases and, and as part of the prioritisation, no talk or,
36:26
or very little of what's going to be the return on investment.
36:29
And I'll use the example of the HR chatbot because I think it
36:34
was identified for a lot of company as it's kind of the
36:38
Goldilocks use case. You know, the data is well
36:41
understood because everyone is at least has a high level of
36:47
familiarity over what HR is, the type of questions that you can
36:52
ask, the type of answers that you can expect.
36:55
Not saying it's a very easy use case.
36:58
There's a lot of of, you know, finesse into it, but
37:03
conceptually everybody can understand it.
37:06
Your data's captive. You don't need to do third party
37:08
data acquisition. You don't need to, you don't
37:10
even need to understand your data landscape.
37:14
You know, every company has an HR function, everybody takes
37:17
vacation, everybody gets paid like it, it, it almost kind of
37:21
perfect on that front, but it's, it's one of those automation use
37:27
case. So, so the real question that
37:29
you have to ask yourself is if, if you want to turn out an ROI
37:32
is are you going to be able to limit your hiring?
37:37
Is it going to, you know, relieve people from doing more
37:43
productive work or revenue generating work?
37:49
And it, it's not a use case that's going to drive I, I don't
37:54
think for most organisation in HR chatbot is going to drive a
37:58
tremendous amount of return on investment.
38:01
Because on top of it, one of the prerequisite to be able to have
38:04
that is to have your, your documents very well organised,
38:10
very well structured, which is kind of a, a big important of
38:14
the. And by the time you're done with
38:15
this, then an FAQ becomes a very easy thing to implement.
38:21
So you do it, you implement it, you do the POC, you get it up.
38:26
The costs are very real because you know, you're calling the
38:29
API, you have your application, you can have a cloud bill or you
38:33
know, the salary of people that are coming into.
38:35
Usually it's going to be a time, let's say a team of four people
38:39
for a number of weeks. And then you keep one or two
38:41
people to be able to maintain it and continue updating the
38:45
knowledge base. And those jobs might be part
38:47
time. So you're engaging the cost and
38:50
then you're going to be asking yourself, OK, what's going to be
38:52
the return on investment? And we see a lot of people whose
38:58
like their usage statistics are not super great.
39:01
A lot of people don't want to use the chat bot or the answers
39:04
are not super relevant. We've seen people have like chat
39:08
bots getting confused by the country that you're in because
39:11
you don't necessarily have the context so on and so forth.
39:16
But then they're like, OK, so what's the return on investment?
39:18
And, and you have to acknowledge that the true return on
39:21
investment that you had is well, you undertook, you know, doing
39:25
your vendor selection for the technology that you want.
39:28
You, you got a preferred Gen AI, but vendor you, you know,
39:33
internalise some of the design pattern.
39:35
You got to see some of the failure mode from Gen AI.
39:39
There's a lot of endemic knowledge that kind of got into
39:41
it. You're better positioned to be
39:43
able to prioritise those use case.
39:45
But somehow all of that learning, nobody cares about it.
39:48
Everybody is like, OK, you know, how do we turn an ROI on this
39:53
specific initiative? And, and I feel like it's, it's
39:55
a little hypocritical to to do that because, and I understand
40:02
where it comes from. But ultimately, you know, you
40:05
have to acknowledge that you took that use case because you
40:10
wanted to get familiar with the technology.
40:13
You need to keep this as part of of your narrative because it is
40:20
probably. Like 90% of the reason why you
40:23
pick that use case. Otherwise, if you really wanted
40:26
to have something that drive return on investment, if you
40:30
have to do this at the inception, you have to do this
40:32
as you're prioritising initiative and, and especially
40:37
APOC is, is meant to prove return on investment.
40:41
So you're, you're validating. Like to me, the concept that
40:44
you're trying to to validate is if I use an LLM for this
40:50
business process, A isn't going to work.
40:53
We've seen occurrences where you don't have access to the data,
40:57
you don't have a complete picture.
40:59
You're looking at automating the process, but you need to insert
41:02
the human into the process at the very awkward places, which
41:07
basically means that attempting to automate the process makes it
41:11
slower and more cumbersome. That's what you want to
41:14
validate. So to me it comes a little bit
41:20
as as a weird positioning. We're seeing less of this now.
41:24
I know we talked about this a couple of months ago.
41:26
I think organisations are getting a little bit better at
41:29
this. And, and we see, you know, some
41:32
of the organisation that we do work with are, are actually
41:35
quite unapologetic around selecting use case and
41:39
identifying revenue. And what they have in common is
41:45
their, their business people took a very keen interest in
41:49
getting familiar with the technology, understanding not
41:53
necessarily like, you know, the intricacies of it, but they have
41:56
a good mental model of how it works.
41:59
And there is a very tight collaboration in, in creating
42:03
those products. And we're seeing more and more
42:05
of this. I find this very encouraging.
42:09
Do you feel like those learnings that you talked about in the PO
42:12
CS, they can be salvage and almost like you pull a rabbit
42:18
out of the hat and you say hey the HR chat bot is shit, We all
42:22
know that. Even.
42:24
But now that we've got all this stuff, we actually have an idea
42:28
for something that will bring ROI.
42:30
Does that give people pause? Because now they're thinking
42:35
this again, We got to go through that whole cycle, even though
42:38
it's going to be a bit more streamlined, there still is.
42:42
Isn't that kind of buy in as there was before?
42:47
I, I think it depends on like what, what's your appetite to
42:49
invest and, and, and ultimately how, how the market is
42:55
responding to, to those claims. Like, I think it's important to,
43:02
to, to remember that it's, it's very exciting to, you know,
43:09
pursue generative AI and her and, and I mean the, the market,
43:14
like for a lot of public company, the market has rewarded
43:17
companies that have been very vocal about adopting Gen AI.
43:21
So what do you do with that power?
43:26
I I think it's, it's the, the broader question that that you
43:29
need to ask yourself. Some companies are, you know, a
43:33
little bit burned because you know, they, they, they, they
43:39
don't really have that risk, like that notion of risk that
43:46
it's not guaranteed. You know, you're, you're putting
43:49
a hypothesis in on top of that. Some people are, are not
43:53
stopping their POC early enough. I'm, I'm, I'm seeing sometime
43:58
people doing a proof of concept and they're engaging full UI
44:03
complete build. They want to be able to do user
44:06
testing and it is expensive. If, if you're very confident
44:14
that your, your backbone that like there's going to be, you
44:18
know, a fit and it's going to be used and you have a good notion
44:22
of, of where it's going to go. I, I think it's fine.
44:25
Like there's no one size fit all.
44:27
But I would, I would postulate that a lot of company could use
44:31
in reducing their, their POC's and you know, trying to distal
44:39
them down to a series like one, maybe up to three questions that
44:45
you're trying to validate. And one of the thing that we're
44:49
implementing on a lot of the PO CS that we do is we start with a
44:53
scorecard. So, you know, you want to do
44:56
I'll, I'll use a very common example that we see, but you
45:00
have a tonne of unstructured data that is coming, you know,
45:02
fax, emails, PDF and, and you want to extract, you know,
45:08
relevant information on it. I think at this stage the PO CS
45:12
been validated. I think industry wide, we
45:15
understand it's a very good use case for generative AI.
45:18
Is it going to work with your document?
45:20
We'll boil them down to a very simple question.
45:23
Work on the ML portion, use those result to be able to build
45:27
a business case. You don't necessarily have to
45:29
engage, you know, a massive team for a long amount of time.
45:33
It it's your, your exactly. You're, you're proving a
45:36
concept. If if you want to go straight
45:39
into a pilot, we're seeing a lot more about that happening
45:42
because, you know, the concept's been proven and then you can go
45:47
you can start asking questions that are going to come on to it.
45:50
But to me, this is like, this has very little to do with how
45:55
you do data science as an organisation.
45:57
It's very much how do you develop internal product or even
46:00
external product? Yeah.
46:02
And and how do you like, how do you innovate as an organisation?
46:07
Yeah, I was thinking that same thing.
46:09
It's more about how do you run a proper POC it?
46:13
If it's in Gen AI, great. If it's in some other random
46:17
aspect to the business, great. You still got to know how to
46:20
properly conduct these POC's. Yeah, there there's another
46:25
thing too that as I was thinking it almost is, I don't know if
46:33
it's, it's counterintuitive, but it is like going back to what we
46:37
were saying earlier on your job is the centre of excellence and
46:43
you now have to validate those PO CS and you have to figure out
46:50
where is the ROI. It makes for potentially sticky
46:55
conversations. And so like you were saying,
46:59
you're coming in more in the line of there are certain use
47:04
cases that have been proven. We've had ChatGPT for almost 2
47:09
years now, right? And so it's pretty clear that it
47:13
works really well with these five things.
47:16
Choose one of these five things and you'll probably be able to
47:20
see some ROI from it. If you go down this route.
47:24
It's cool, it's fun, you'll learn stuff.
47:26
But ROI? Don't bring that into the
47:30
conversation. Well, I mean, we're consulting
47:34
organisations, so we've, we've created a series of like handy
47:39
dandy tools for, for those kind of endeavours.
47:43
It's funny that you say those five things.
47:45
We, we actually have like I use that mental model a lot of time,
47:49
like the five pillars of Gen AI, like like what are the verbs
47:54
that generative AI is very good at?
47:56
It's like, you know, kind of summarise, synthesise like it,
48:02
it gives, I think it's a great way to do outreach, you know,
48:06
with people that are closer to business function.
48:08
And it also helping them build a healthier mental model of of the
48:15
technology, whether it is, and I mean, generative AI is what's
48:20
going to be the tip of the spear.
48:21
But, you know, traditional machine learning or deep
48:25
learning or even plain old automation has also a place and,
48:31
and usually they, they work together.
48:34
Like, you know, you have a lot of compound system.
48:36
A lot of people talk about, you know, LLM agents, which I prefer
48:41
to see it as that synergy between software, ML, deep
48:45
learning and, and, you know, generative AI.
48:50
But you're by by doing that outreach and continuing to to
48:55
maintain those relationship with within your organisation, you're
48:58
in a much better position to be able to identify use case and do
49:02
back of the envelope POC. But one of the thing that we do
49:06
quite a bit is, you know, hey, one afternoon we're going to
49:10
validate this a handful of document like see, this is a
49:13
result. This is, this is what's good.
49:15
This is what's bad. Are we in a stage where we feel
49:20
pretty comfortable? And then as you design, you
49:24
know, your, your flow like what you're trying to, to optimise or
49:27
what you're trying to develop, understanding what are the
49:31
different failure modes? I, I think that this is, this is
49:38
a very exciting way to me to develop new products and to be
49:42
able to, to generate what I would say like differentiating a
49:50
product. Like once in a while we see a
49:53
couple of companies and most of the time it's not a company that
49:56
you would expect, but they naturally like couple of people
50:01
are gonna converge, they're gonna hack on an idea and you're
50:03
just gonna be like, wait a moment.
50:05
That's really. Smart.
50:09
What have been some of those that you've seen that surprised
50:11
you? I would say to me, it's
50:17
something that we, we've seen very recently was to how do you
50:25
create a chat bot, let's say with content that is not that
50:35
needs to be curated like in the sense of your, your
50:39
hallucination tolerance is 0 like none whatsoever.
50:45
So it's kind of a tough predicament to say we're just
50:47
going to put in LLM. No amount of prompt engineering
50:50
rag fine tuning anything. Is this going to make up for the
50:55
fact that ultimately you will get inconsistency.
51:00
So one of the thing that we've developed as, as a, as a
51:04
blueprint, which came from numerous conversation that we
51:08
had across the, the industry was to, well, what if we didn't use
51:14
an LLM for that stage? What if we just use a mere
51:18
classifier, present the information in a way that you're
51:22
not compromising. Like, you know, when you ask a
51:25
question, you do, you know, a classifier to like what is a
51:29
question that is closely related and present that.
51:33
And if there's a little bit of inconsistency, that's just
51:35
become part of your, your knowledge loop.
51:38
But then use the large language model, the LLM loop in the
51:41
content creation, creation and review to streamline the work,
51:47
which is mandatory work. At one point someone needs to
51:50
create the content that's going to be served up.
51:53
How can we accelerate it? So we ended up creating a bona
51:56
fide workbench for optimising the creation of that content,
52:02
the ingestion of of the relevant material to be able to create
52:06
going even to a point you're ingest APDF.
52:09
What are like this is the style of questions that we know people
52:13
are going to ask. What type of question does this
52:15
document answer? And we're ultimately we're
52:20
seeing the same benefit. You have the chat bot, the chat
52:23
bot is presenting the information that is supposed to
52:26
the chat bot is not hallucinating.
52:27
There's nothing sexy there. Like, you know, it's something
52:30
that you use previous generation technology, but we're, we, we
52:37
look at the process, see where the friction was.
52:39
And ultimately it wasn't a content creation and curation.
52:44
And because this is something that the human is very tightly
52:46
in the loop. Well, LLM can play in that
52:50
arena. And, and to me, it was one of
52:53
the thing that like, at first when I was looking at this, I
52:57
was a little bit puzzled because I was like, this looks overly
53:00
complicated. But then you start decomposing,
53:03
like, what are the non negotiable?
53:06
And then you're like, well, traditionally you would say
53:09
like, oh, you can't have any hallucination.
53:12
Oh, well, so LLM is not a good use case.
53:14
We're going to go to the next, Oh, wait a moment.
53:17
This is actually something that is very valuable.
53:21
You know, a lot of contents. It's a big pain in the ass for a
53:24
lot of of people. Can we pay attention to it?
53:30
And then it was, you know, in our case, you know, we had some
53:34
subject matter experts, some people that are creating the
53:36
content, a team of data scientists to be able to develop
53:40
the model. And it works.
53:42
I'm I was, I was legitimately surprised it works.
53:46
Wait, so I I wasn't clear there was no LLM involved or it's just
53:51
involved later on it's. It's involved ahead of time.
53:55
It's like you're basically, you know, you're, you're like you
53:59
have kind of a knowledge base and you're doing, you know,
54:04
you're using natural language understanding to be able to
54:06
match the question that is proper.
54:08
But when you're creating those articles and the metadata that
54:13
makes them referenceable, you use large language model to to
54:19
generate that. Like the embeddings, when you're
54:22
creating the embeddings, you have your embedding model and so
54:25
you're using the embedding models still.
54:28
It's, yeah, it's, it's kind of a, it's kind of a reverse rag,
54:31
like. You just created a whole new
54:34
type of rag. Get out of here.
54:38
So now hold on, hold on, hold on.
54:39
Yeah, we've got graph rag, we've got advanced rag, we've got
54:42
naive rag. And you heard it here first.
54:45
You we now have reverse rag. I will.
54:48
I will take absolutely no credit for this.
54:50
I, I do not want to, there's something that a lot of people
54:55
say like it's been unvented, like, you know, it just happened
54:59
to be the pattern at hand. I, I, I, I don't want to be, I
55:06
don't want to, I don't want to go and create a paper on reverse
55:09
rag and I'm going to get shredded on LinkedIn.
55:14
This is how I'm going to go and get my Centre of Excellence job.
55:17
I'm going to go write that paper on reverse rag of an I.
55:21
Can imagine this in your ML OPS certification thing.
55:24
You've never heard of reverse Rag?
55:26
And then people start searching and they see your name as the
55:29
primary author. Like self-service at there we
55:33
go. LinkedIn top voice.
55:36
That's what I'm going for baby. All right so you reverse ragged
55:40
it basically, which means you and there's a classification
55:44
model. I'm.
55:44
I'm still not completely sure how this works out but it sounds
55:48
really cool because any time anyone says these magic words to
55:52
me my brain and my attention perks up, which is we decided
55:58
not to use an LLM. When you say that to me in 2024
56:02
I go wait, wait what? Yeah, yeah, that is
56:05
unconventional. And so you're using a
56:08
classification model but. But I am not.
56:11
So you're using the embedding model still?
56:13
So I'll, I'll, I'll give you an example like let's, let's say
56:16
for, let's say, for instance, that you're working for an
56:22
insurance company and you want to be able to have some
56:26
information around your policy. So you're, you're going to ask
56:31
her questions around like, you know, what is my deductible if I
56:35
have sewer backup? Well, there there's actually
56:40
like you, you don't need an LLM in order to be able to serve
56:46
that information. There was a couple, it's there's
56:49
a couple of classification that you're going to have to do
56:51
where, you know, you live in, you know, state X, you're in
56:57
this city. This is your policy.
57:00
You're going to have to extract all of that information, which
57:02
for the most part is structured like you, you don't need to
57:06
like, you know, start stuffing some embeddings and creating a
57:09
massive rag of all the policies and so on and so forth.
57:12
Because it's it's in the CRM exactly.
57:15
You have that information about the customer and this is
57:18
basically a customer support question.
57:20
Insurance can't be telling you policies that are not your
57:24
actual policy. So it can't go wrong exactly.
57:27
And and then the, the article like, you know, as a matter of
57:30
fact, your, your insurance policy was written with a set of
57:34
articles. So if you can just say, you
57:37
know, you're going to take this article, that's going to be
57:39
attached to your policy, that's going to be attached to who you
57:42
are, and and then you can craft that answer.
57:45
So in in that case, you know, you just resurface the piece of
57:50
content that you want and you can just add like some
57:54
deterministic fluff of like, Hey, Dimitrios, like based on
57:59
this information that we have with you, this is what's written
58:02
in your policy. If if you have more questions,
58:06
you can, you know, ask more question and then you can
58:08
escalate with it to a human. But serving that initial piece
58:12
of content, there's absolutely nothing stochastic with the
58:17
exception of taking your question and translating it to
58:21
the appropriate article. But if you use an LLM, there is
58:25
a chance like that is whatever it might reshuffle something.
58:30
And we're not talking like in that case, we're not talking to
58:35
people that are like, I would rather eliminate hallucination.
58:39
We're talking around people that are either scared shitless or
58:45
they're constrained either by the government or, you know, by
58:51
they're afraid to get sued if they get some bad information.
58:54
I mean, think of it for a moment.
58:58
Companies that are trying to do chatbot for.
59:02
Sign in. Drugs, drug usage, where when
59:06
you get information like it's, it's really easy for this to get
59:09
out of hand. Like let's discount the legal
59:12
framework for a moment. If you ask questions around like
59:16
interaction between two medication and the LLM, for some
59:21
reason there's there's something that magically escape.
59:26
Well, you you could be in trouble.
59:29
So, so how do you redesign this and find alternative pattern?
59:35
And it's not just data science, it's how you're going to present
59:39
the information. We did a lot of of analysis like
59:44
human centre design, UX development to be able to see
59:49
how it is going to go, a lot of experimentation internally
59:53
because it, it's a problem that we decided was we're pursuing.
59:57
It's a pattern that we're seeing across multiple customers.
1:00:01
And it, it's, it's an interesting problem.
1:00:04
And I think that taking that approach also demystifies that
1:00:08
all or nothing approach to like either you put LLM at the centre
1:00:12
of everything or you go home. There's a there's a lot of
1:00:18
things that you can do with a more nuance and thoughtful
1:00:22
approach. Wait, so isn't this just
1:00:26
basically what we were doing in 2017 with chatbot?
1:00:30
Yeah, but not every company is not every company is is is Peter
1:00:36
was ready for that or for a lot of company generating the
1:00:40
content was tedious. So, and, and this is one of the
1:00:47
place where in that case and in the content generation or the
1:00:51
the content curation or collection or in in that case,
1:00:55
like we've, we've done some work with a client where we've used
1:00:58
large language model to generate the questions that could be
1:01:03
answered by that document and do Q&A pairs, which is a lot easier
1:01:08
for someone to review then have to generate it themselves.
1:01:11
So it becomes more of a productivity play rather than
1:01:14
the like your user with an approximate chat bot.
1:01:18
I see. Yeah.
1:01:20
And this is this is fascinating to me because I know a few
1:01:25
companies that are actually trying to do this as a service.
1:01:29
And so I think it is one of those things that is immensely
1:01:33
valuable as a use case if you can.
1:01:36
The support team at any company that has a large volume of
1:01:42
customers is just inundated constantly.
1:01:44
And we know that because if you've ever tried to call or get
1:01:49
somebody on the phone of whatever your phone provider or
1:01:53
your for her local whatever, you know that it's a pain in the
1:01:58
butt. And so if you can lift a little
1:02:02
bit of that burden, it's really helpful.
1:02:05
What I always have trouble with is a lot of times there are
1:02:13
questions, but then there are other times where I need
1:02:17
something done. I need to talk to you because I
1:02:20
need something to happen. And I'm.
1:02:23
So it's almost like I am very sceptical that I'm going to be
1:02:28
able to get a machine or a support bot to actually do those
1:02:32
things in a way that I can trust that thing to happen.
1:02:36
And I mean, it's, it's also kind of The Dirty secret of, of, of a
1:02:41
lot of like ancient workflow. We like a lot of people use the,
1:02:51
the verb like function calling, you know, for, for an LLM to be
1:02:55
able to perform an action, the function need to exist.
1:02:59
Like, I find it funny to say this.
1:03:02
And, and, and I know like I'm, I'm I saw your smile when when I
1:03:08
I started. But the if the, if there's
1:03:13
nothing to call, then everything fall like a House of Cards.
1:03:16
So, you know, you need to build that kind of tower.
1:03:20
Like let's say you use an example like I want to close my
1:03:23
account. Then you talk to a chat bot
1:03:26
because you're going to your bank.
1:03:28
Well, there's a pretty big SOP that like standard procedure,
1:03:33
standard operating procedure that needs to be followed.
1:03:36
Sometime people are going to try to retain you.
1:03:39
So it, it, it actually like if if you look at this, it usually
1:03:43
looks like a decision tree with like, you know, Inflexion point
1:03:46
and and decision making that you have to do.
1:03:50
But let's say for an example, that like the action close the
1:03:54
account involves 3 application, 4 screens and then a letter that
1:04:01
you have to mail to the person. Cool.
1:04:04
Your very first step should be to automate that and, and, and
1:04:09
then you can start growing your process and maybe, and maybe the
1:04:15
agentic LLM is, is going to be your North Star, but start with
1:04:21
those fundamental gain the ROI as it's going to go.
1:04:24
What are the things that because you're not necessarily going to
1:04:28
want to the, the LLM to call for function one after the other.
1:04:32
If that step happened, you might want to, you know, do something
1:04:36
that's going to abstract it. You might still be working with
1:04:41
like an AS400 type system, which means that you're going to do
1:04:45
RPA or God forbid, you know, put an LLM to interpret the screen
1:04:50
and then click on those things and.
1:04:52
And there we go. Yeah, but, but, but I mean, it's
1:04:55
it, it all has to do with your system design and and an LLM
1:04:59
application is also an ML application, which means that
1:05:01
all of the consideration that you have with ML also applies to
1:05:05
your LLM app. And all of that is also a
1:05:07
software product. So if you so you have to
1:05:09
remember that all of the responsibility of developing a
1:05:12
software product and nothing disappears because you ignore
1:05:15
it, but it also gives you that complete array of tool that you
1:05:21
can leverage. And a lot of things need to
1:05:24
happen sometime before you can use an LLM competently.
1:05:29
And a lot of, a lot of companies are I, I, I think we're seeing
1:05:37
this right now where everybody's doing agent base LLM and they're
1:05:41
all doing LLM hosting and they're all doing like, you
1:05:44
know, LLM development into the ecosystem because it's new, it's
1:05:50
interesting, it's fresh, But you, you also need to remember
1:05:54
that's just part of the story. Like what are you going to do
1:05:57
with the rest of the stuff? Do like you want to be able to
1:06:03
do a rag and you need embeddings?
1:06:05
Cool. But if your data is on a
1:06:10
OneDrive somewhere and you know it's getting updated by, you
1:06:16
know, Penny from finance every Tuesday, well, you need to be
1:06:22
able to have a pipeline that's going to do this.
1:06:25
It's not an AI, but it needs to happen still.
1:06:31
It's funny because in 2020 we had Luigi Petrino on this
1:06:36
podcast, and I wrote a post after talking with him because
1:06:40
one of the main themes that I took away from the conversation
1:06:43
was start manual 1st and then automate.
1:06:47
Yeah, and you just added an extra little bit to it.
1:06:50
It's start manual first, then automate, then AI fi or then
1:06:56
make it AI agentable, whatever it may be that the next step is.
1:07:02
So if you want to function call it or make it agentic.
1:07:07
It's almost like that manual automate agentic.
1:07:11
And you can't just jump over any of those steps because your
1:07:16
agent is going to not be efficient.
1:07:22
And I really like how you said that it's probably not the most
1:07:25
useful for the agent to have four different function calls in
1:07:29
there. Maybe you can extract abstract
1:07:31
something and make it a lot easier for the agent when it
1:07:37
comes time for you to make it agentic.
1:07:40
And as you design your experience, you know, you have
1:07:44
to remember that even though it's an LLM, like it depends on
1:07:48
how much control you want to give to the machine.
1:07:50
Like if, if you're one of my favourite thing, like sometime
1:07:54
we do design of like, you know, customer service bots or, or,
1:07:59
you know, internal productivity bots.
1:08:01
And, and you ask, OK, what should happen at this point?
1:08:05
And everyone around the room, it's like, well, it depends.
1:08:08
And it's like, cool, we're going to have to break that down
1:08:12
because ultimately you're relinquishing your decision
1:08:18
making to either an open source LLM or a third party LLM.
1:08:24
And and you also need to be able to design fail safe.
1:08:28
And this is very similar to how you design for human.
1:08:31
You know, humans errors are creeping and how do you correct
1:08:34
them there? There's a lot of process that
1:08:37
you don't need a second person to go over it because it's
1:08:41
highly sensitive. So, so you have to design those,
1:08:46
those workflow. And I think we're, we're getting
1:08:49
the grasp of it. One of the, one of the thing
1:08:53
that I use a lot is every single time there's a decision point
1:08:56
that you want to use machine learning and you want, or you
1:08:58
want to use generative AI. I'm always saying the question
1:09:02
like, let's assume that's 70% of the time it's going to do the
1:09:05
right thing, but 30% of the time it's going to do something
1:09:08
wrong. Is this making or breaking your
1:09:12
use case? And I, I had some occurrences
1:09:16
where people were like, well, we need 100% accuracy still to this
1:09:21
day. You know it, it was the same
1:09:22
thing when we were talking about like linear regression tree
1:09:26
based models, you know, even neural network.
1:09:30
And now with LLM you know your your definition of OK.
1:09:36
How many errors are you willing to accept if your number is 0?
1:09:42
Probably not the right use case. Probably not the right use case,
1:09:45
but also are you really that certain that the humans that are
1:09:48
doing the job are? Also, making zero mistakes.
1:09:52
Turn the tables. Yeah, totally.
1:09:56
Because anybody who answers that honestly is going to go, yeah,
1:09:59
of course we're making mistakes all the time.