Latest / Agentic Conversations (formally mlops.community) / Building in Production Human-centred GenAI Solutions // Mohamed Abusaid & Mara Pometti// #177

Transcript

Building in Production Human-centred GenAI Solutions // Mohamed Abusaid & Mara Pometti// #177 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Mara Pometti, Associate Design Director at McKinsey as
0:05
part of Quantum Black and I like to take my coffee like info
0:10
espresso, very Italian, possibly macchiato.
0:15
Hi, I'm Mohamed Adarseid, Associate Partner in Quantum
0:18
Black and McKinsey, and a senior solution architect.
0:21
I like my coffee black. Either a long black or a double
0:27
espresso. Hello and welcome everyone to
0:30
another ML OPS Community podcast.
0:31
I am your host Dimitrios and today I am joined by none other
0:37
than my amigo Stefan. What's up bro?
0:41
How you doing? Hey everyone, I'm doing
0:43
fantastic. I'm doing amazing on this day.
0:46
So we just got off of a pretty incredible conversation where we
0:50
went down so many roads with the Quantum Black crew, MO and Mara.
0:58
And I feel like there were so many cool takeaways that I had
1:03
one. I'll, I'll try and break down
1:06
this conversation into parts, right?
1:10
One major theme that we talked about was the different ways
1:13
that enterprises think they need AI and these three maturity
1:19
levels or ways that they interact with AI are the takers,
1:23
shapers, and makers. Keep that in your mind for later
1:26
and I'll let, I don't want to spoil anything.
1:29
So we'll get into what each of these mean in just a second.
1:32
And then there was the idea of how you can treat AI more like
1:39
code. And this definitely is
1:42
something, it's a theme that's been coming up over and over
1:45
again. How do you treat the new
1:47
paradigm, especially when working with LLMS, more like
1:50
software engineering? I know that we've been talking
1:53
about this for years, so it's almost like we're beating a dead
1:56
horse when it comes to that. But ML OPS is all about that
1:59
software engineering dev OPS practises.
2:01
Now we're bringing it to LLMS. What is that that we need to
2:07
think about? Like what does it entail?
2:09
And so we talked about prompts and prompt tracking and all
2:12
that. Testing your prompts, all that
2:14
good stuff. And then the last little Rd we
2:17
went down I feel like was on the private AI, self hosted AI and
2:24
open source sections. Love the discussion about the
2:28
open source part, actually, and like all those licences and how
2:32
important it is, you know, like, and we all talk about safety and
2:37
there's so many people talking about safety with AI, like how
2:40
AI is going to take over the world and blah, blah, blah.
2:43
Suddenly with LMS and I'm like, yeah, with open source, you
2:45
know, people can actually check it.
2:47
People can actually make sure, you know, also the part that
2:50
Barrett touched on, which was, you know, education, how to
2:53
educate people about AI and to make sure that, yeah, LMS are
2:56
cool, but remember what's happening behind the scene a
2:59
bit. It's funny how she talked about
3:03
the spectrum of people that we're using open AI or some off
3:09
the shelf API until they realised and they got educated
3:14
on all these different ways that things can go wrong.
3:17
And then they reassessed from their risk and their appetite
3:21
for risk. And so they brought things in
3:23
house. And that's one thing that I've
3:25
been fascinated about too, is just how many people are truly
3:29
bringing these LLMS in house as compared to the people that are
3:33
using Open AII think Open AII say, And I, I mean like Open AI
3:39
or any of the other APIs out there.
3:41
Yeah, it's, I mean, I also see it, you know, like in our
3:45
company, for example, in a lot of companies that would I talk
3:47
to in Berlin, they always like, they all tried to to like have
3:51
the other labs and everything. And then they're like, Oh yeah,
3:53
maybe privacy wise, like how do we delete the data?
3:55
And then they're like, OK, maybe we'll maybe we'll hold on a bit.
3:59
Especially there's that added layer of complexity being in
4:03
Europe, and then I think there's an extra added layer of
4:06
complexity being in Germany because they follow.
4:09
The rules. To the T you I, I think I've
4:13
told this before, but when I went to the Frankfurt ML OPS
4:17
community in person meet up, it was the most quintessential talk
4:23
that you would expect to hear from a company in Germany.
4:27
It was all about regulation and how they're fitting into the
4:31
regulation and how they're using ML LS practises despite the
4:36
regulation and how their ML LS practises are helping them fit
4:39
better into regulation. It was just like, if I hate, you
4:43
know, could go back in time, I would count or I would play
4:46
drinking game and have to drink every time they talk about
4:50
regulation. Did they mention at least facts
4:52
one time? I hope.
4:53
Yeah, Yes, let's go. Dude, so many times.
4:58
So anyway, let's get into the conversation with MO and Mara.
5:02
And before we do, I want to give a huge shout out to Quantum
5:07
Black and thank them for sponsoring this episode.
5:11
Those that do not know, Quantum Black is the AI arm of McKinsey.
5:16
McKenzie is absolutely killing it when it comes to AI, and
5:19
Quantum Black is the forefront. They're the cutting edge.
5:22
They get to sit down and advise all kinds of enterprises on
5:27
their AI strategies and you can see from Talking Tomorrow and
5:31
MO, they know what they're talking about.
5:33
They have the experience to back it up.
5:35
All right, so let's get into it. Don't forget to like and
5:38
subscribe and leave us a review because we're almost breaking
5:43
the record for reviews in Spotify.
5:45
That's it. Here's the podcast.
5:53
We started this off with quite a long sound check, and now we
5:58
might as well just go straight to the jugular and talk about
6:01
the new LLM paradigm that have come out.
6:04
And I saw recently that Mackenzie just posted their
6:10
annual report on AI, which I found fascinating.
6:13
I always love reading that because I aspire to be as as
6:18
good of a report creator as the McKenzie reports sometimes.
6:22
Actually, I used it for inspiration when we put together
6:24
our LMS in production report and you just dropped your newest 1
6:29
and I loved it And it basically said like Jen AI is taking over
6:34
and that's like the TLDR of it. And you've got 100 or I think
6:38
there was like 1600 people that over 1600 people that
6:42
participated, which is incredible to get all that data
6:45
and then see how the analysts were able to wrestle with that
6:50
data and come to the conclusions.
6:53
So let's start with what's going on in your worlds.
6:56
I mean, there's been a huge shift and MO, I'll, I'll throw
6:59
this one at you. And like how are companies using
7:02
LLMS? It's, it's a, it's a hype that's
7:04
been going on and it's growing and growing and growing.
7:08
I don't see it calming down. They practically log into to
7:12
hacking news every single day. There's a new library, there's a
7:15
new article, there's a new research.
7:18
You can't sleep for a week without missing something
7:21
crucial in where the the industry is going.
7:26
There's a lot like I think there's a lot of companies are
7:29
trying to figure out how to use it.
7:31
A lot of them are trying to figure out, you know, where can
7:34
we do, we drop everything we've been doing with analytics just,
7:37
you know, but all our data scientists subscribe to our
7:41
service and now we just build an application and it's crazy.
7:46
It's crazy because it's not clear for such a new technology
7:52
where it's best used, what it can achieve today.
7:55
What is the hypothetical reasons for it?
7:58
One of the ways we came up with to help companies and the
8:02
clients and their organisations we serve.
8:04
Think about it is the three ways that LLMS are being used today.
8:09
We've got part what we call takers, ones that we call the
8:14
shapers and then ones we call the makers.
8:17
And the kind of distance between them is if you think of the
8:20
takers or anybody who is able to subscribe to a service like Open
8:24
AI or Cohere or bot and simply plug in their UI and some prompt
8:30
and be able to just use it off the hook.
8:33
They don't need their own data. They simply need a a prompt of
8:36
some type and some context to feed into it.
8:39
And you're up and running. And this is where you see a lot
8:43
of the startups are trying to occupy the space where you know
8:47
with some user input I should be able to derive something,
8:50
whether it's information extraction or something based.
8:54
For example, this would be like when I'm using Notion and I hit
8:57
the AI function and I say like write this poorly written
9:02
paragraph, make it look beautiful, and write it much
9:05
better than I could do. Exactly.
9:07
Yeah, regardless of how much other information it had about
9:10
you, the output would look very similar because it's purely
9:14
using it, let's say skill in writing nicely worded confident
9:21
sentences to to produce an output for it, or whatever mood
9:25
you decided to wear. You said confident.
9:28
I like that. But we can get back to that
9:30
whole idea of the confidence that it had.
9:33
And so these are the takers, right?
9:35
This is the takers, OK? And this is one of the, this is
9:38
the most popular. And then you've got the shapers
9:40
and the shapers apparently saying that, yeah, you will need
9:43
these services or some of the open source models and so on.
9:46
But you want to add extra content, extra components to to
9:50
actually deliver a much more complex solution.
9:54
Whether it is you have a large knowledge that you'd want to
9:58
create embeddings for to be able to retrieve from in order to
10:02
pass further context that the user doesn't have when they
10:06
prompt you. Or whether you actually have a
10:08
model that you want to refine a little bit simply, you know,
10:14
it's instructional, but it doesn't know how to do cooking
10:16
instructions. It knows how to extract
10:19
information, but it doesn't know how to extract information out
10:22
of a plane ticket. And, and, and, and so that is
10:26
priorly saying we can build on the shoulders of the existing
10:30
services and giants, but we'll be able to add extra knowledge
10:34
or extra preprocessing on top of that to assist in creating a
10:39
more sophisticated use case. So that's anything using like a
10:43
vector database. And a classic example of this is
10:46
like talk to your data. If you have some knowledge
10:49
augmentation, or if you want to figure out what's going on in
10:52
JIRA and you've set up some kind of way to navigate that with a
10:58
chatbot, then that would be an example of a shaper.
11:02
Exactly. It's like I have this huge
11:05
corpus in my confluence in all of my docs.
11:08
It takes me hours to find. Can I just process all of my
11:13
knowledge base so that my users now can simply query it like
11:17
they do with ChatGPT? Yeah.
11:20
OK. And in here you're bringing your
11:22
own data, you're bringing your proprietary stuff.
11:24
And then finally you've got the makers, which is is the least
11:27
popular. But it's good to define and, and
11:30
it's important to define because sometimes organisations jump to
11:34
do that. They want to do that straight
11:36
away, which is I want to build my own one.
11:38
I'm not satisfied with the status quo.
11:41
I would like to build something from scratch.
11:44
Whether it is because they want to create a new competing thing
11:49
or whether they believe what's available in the market is not
11:52
good enough or the licencing offering is a huge complex
11:56
haywire that we can go into of you know, what's mine, what's
12:00
not, what's my IP in the long term on, but the objective there
12:04
is building your own. Good open in the future because
12:09
I feel like everyone is trying to find you in their own model
12:11
now. Like you know, everyone at first
12:12
tried to build their own big model and as they realised it is
12:16
very expensive and it takes months and months.
12:20
Whereas now like everyone was trying to find you in their own
12:22
models, right? I think it's an interesting
12:26
problem in that there's a lot of, at the moment we're looking
12:30
at LLM as a huge single BLOB of just a model.
12:34
And I believe at at some point where we're seeing a lot of the
12:37
big companies now use multiple models and we're seeing a lot of
12:41
architecture where you say you can use different models.
12:44
So I do believe there will be a place for the general models
12:47
that you can just use off the hook, whether they're going to
12:49
sanitise your data, extract sentiment or not.
12:52
But there might be an opportunity for creating these
12:55
almost expert LLMS and going back to old school expert AI
13:01
models where they're really good at one thing.
13:03
You might need the expert LLMS where they're really good at,
13:07
you know, doing a single task out of your problem state.
13:11
Yeah, it's like the Twilight Zone.
13:12
Like, I'm pretty sure I've seen this before.
13:15
I don't know why it looks so familiar.
13:17
Am I having Deja vu right now? Exactly.
13:21
So, so if you look at it through this paradigm or Takers,
13:25
Shapers, Makers, then the easiest is obviously the Takers
13:29
because you know you could do a hackathon and end up with three
13:33
apps as an organisation by Sunday evening.
13:35
And so it's very easy. The Shapers is probably when you
13:39
satisfied that initial urge and figured out what what can you
13:43
get without your data and what can you get within a single
13:47
context before you have to do a lot of magic.
13:51
But the makers become the kind of the top end of Do you really
13:56
need to build your own custom thing or can you satisfy the
14:01
situation with the shape? And have you seen some companies
14:03
being successful actually, like combining expert models and
14:07
direct models, or do you think everyone's trying out everything
14:10
in the moment? I've seen a few things that it's
14:14
probably funny to talk about seeing companies try to build it
14:17
on their own or organisations try to build one on their own
14:21
because they believe that the existing ones are either from a
14:24
competing company slash country and thereby they want their own
14:29
proprietors national model that is still developing.
14:33
So we'll probably see how successful they are and becoming
14:36
sustainable because he also noticed, like in Hugging Face,
14:39
there's a lot of models that just get published and never get
14:41
refreshed. And so feel like JavaScript
14:46
frameworks where there's just a new flavour every week.
14:50
With the combining, I think Marlow actually talked about at
14:55
some point when we do later on, we've been combining ourselves
14:59
and we have seen others combined where you want to do specialist
15:03
things either to increase confidence or capture things
15:09
that models necessary don't capture whether it's, you know,
15:12
bad behaviour, ensuring that the intent is appropriate.
15:17
So whoever is asking that specific LM model, for example,
15:21
if they're asking it within the realm that the provider has
15:24
deemed it to, I wanted to, you know, question just my error
15:28
logs and not try to gaolbreak and do other advance.
15:33
So you may want to do expert models ahead of it and to limit
15:38
and validate the intent. So this is awesome because it
15:43
feels like the takers are in one way are those hackathon winners.
15:49
And it's almost like you hear about these projects as being
15:56
super creative, but you have to be creative because there is no
16:00
Moat. There's nothing that people
16:03
can't do because it's just like an API call, right?
16:06
So you have to find that area where nobody's touched yet, and
16:11
then you can go and attack it for X amount of days.
16:15
And then five other competitors pop up if they see that real
16:19
revenue is being made. I think to that what's happening
16:23
is that the mote is moving from the model to what application
16:28
are you offering The first the sophistication of the the
16:32
application and the interface. Remember like other
16:36
organisations have built a kind of LLM before Open AI, Open AI
16:41
had LLMS before ChatGPT. Well, that interface is what
16:46
started the whole fire of now people woke up and there is an
16:49
interface that I can just question and it knows how to
16:52
answer. So when you do move the
16:54
challenge of we do have a model, my greatness, what is the user
16:59
experience, what the application and how would it look like?
17:02
And and I always, I always laugh with strings when I tell them,
17:05
you know, we already had enough chat applications before
17:09
Snapchat showed up, but it still, you know, took over the
17:12
market for a certain segment and a certain population and it's
17:16
doing really well. Yeah.
17:19
Are you a Snapchat user? Is that what you're trying to
17:20
say? I'm not, unfortunately.
17:23
I hear you have to be under 30 to do that.
17:25
And do you think maybe we'll have like a new job in the
17:27
future And then like, you know, like we have like developer
17:30
experience, then we have UX designer would be like some
17:34
we'll have a new job that is going to be in between maybe the
17:37
repair experience in UX to make sure that you know, you have
17:40
something is. Mara's expertise, you're hitting
17:43
your like all of the soft spots of Once we speak about the
17:46
technology, what exactly do we need help.
17:50
There we go. Yes.
17:51
Exactly, Sweet spot on that. I think we have already those
17:58
kind of experts. I, I, I feel that I kind of
18:02
belonged to that grey area between understanding how to
18:08
design the technology. But because the first we think
18:13
about what's the outcome in terms of business that we want
18:17
to get and especially what's the problem of the people that are
18:22
going to use these application, this technology and what are the
18:28
expectations? How do we ensure that the LLMS
18:35
outcome aligns with people's expectations and business
18:40
expectations? Especially right because it's
18:45
very easy honestly getting rushing into developing a shiny
18:50
object like exactly just calling an API.
18:55
But eventually it's very difficult for organisations and
18:59
especially from what I observe through my word understanding,
19:05
OK, how do we adopt this into our business?
19:10
Where are the room for opportunities where we can plug
19:15
in the use of LLMS and especially how eventually we're
19:20
gonna either integrate or completely revolutionising the
19:25
user experience of the people that are really working with a
19:29
totally different process today that likely will be completely
19:37
changed by the use of LLM. So these are very urgent
19:41
questions that organisations are trying to find an answer to and
19:46
everything. It's really driven by the
19:48
understanding of who are the people whose problems are we are
19:53
trying to solve with LLMS and what are the business outcomes
19:58
that eventually we want to generate for this organisation.
20:03
And once you have that concept, the very well defined, then it's
20:08
easier to really work on the design of the.
20:14
The model itself and exactly as per the point before how?
20:19
What types of approach do we use right?
20:22
Personally, I'm aiming at pushing more and more
20:27
organisations from the ticket approach to the shipper approach
20:31
because I want them to see the multiple opportunities that they
20:36
have to play with these models instead of just exactly using it
20:42
as it comes. But really, I always stress the
20:46
fact that to reach really a great outcome eventually, we
20:51
still need to sign, we still need to think about problems and
20:55
how we solve problems. Also with traditional analytics
20:58
combined with LLMS, a lot of the use cases that I developed for
21:03
clients right now have these combined approach for Mohammed
21:07
Point before. So I think it's fascinating how
21:11
many different solutions we can get by by playing with LLMS and
21:16
also traditional analytics. So, so it feels to me a little
21:21
bit like in this framework, and I love these ideas.
21:25
It's easy to remember the different takers, shapers,
21:27
makers, of course, leave it to some consultants to give us a
21:31
framework on the podcast. I would expect no less from you.
21:35
All right, so you're bringing the from the fire that I would
21:38
want you to bring. But when I think about moving
21:43
from taker to shaper, it feels like the unlock there and the
21:49
lift that you need to do, like the overhead is not that high
21:53
compared to the unlock that you get, right?
21:56
It's not as hard to go from a taker to a shaper.
21:59
And then it unlocks so many new possibilities and so many new
22:03
use cases and so many new ways to incorporate AI into your work
22:08
flows. But then from a shaper to a
22:11
maker, it feels like it's prohibitively just, it's just
22:15
like disgustingly difficult. And I'm not sure if many
22:20
companies really need to be doing that, should even be
22:23
thinking about that. And the other thing is like, how
22:27
much does it actually unlock? But you all are the experts.
22:32
So can I get a Fact Check on that one?
22:36
It feels like the way you think. I think between the taker and
22:39
the shaper, it's a promotion while between the shape of to A
22:43
to a maker, you're almost getting a change in career,
22:47
right. It's a completely different
22:48
business model. Remember you would be joining
22:52
the other service providers that are carnels provided to the rest
22:56
of the you'd have to own and process and manage everything
23:00
from scratch. And so there is a lot more
23:04
involved in it I think. But even from the taker to the
23:08
shaper, there's a lot of challenges and they become less
23:13
if you treat LLM projects and products as software and you've
23:19
taken on all of the good practises that come from the
23:21
DevOps, from the envelopes. Remember that when we described
23:26
to the taker, we've said that the model is already solved.
23:28
So you're thinking of prompts, but you still have to think of
23:31
all of your application and your application development and
23:35
design. Doing that correctly and
23:39
accounting for what it means to use an external service will
23:44
mean that when you move to the shaper later, you're well aware.
23:49
OK. And and yeah, maybe to to go a
23:51
bit deeper on that one, like you mentioned treating the software
23:55
and also using ML OPS, like what what do you mean with that?
24:00
I'll start with probably the silliest thing.
24:03
Are people versioning their prompt?
24:06
Are they simply copying and pasting from a chat into a code
24:10
and don't understand what has happened from one prompt to the
24:13
other? OK.
24:14
And so that's like the first step, like, OK, you like hope at
24:17
least everyone does that. And.
24:20
Exactly right. So the object, the first thing
24:22
is probably like can I build? So it's part of your software
24:26
development. If you think kind of the well
24:28
factor principles is kind of I have ACI pipeline, right?
24:32
Yeah, I am writing a prompt, Am I writing tests to these
24:36
prompts? Am I recording against which
24:38
version of the model when I'm using these services?
24:41
We know some services update their model weekly, some updated
24:45
4 weeks, 8 weeks, 3 months and so on.
24:48
So me being able to track what prompt against what tests and
24:53
using which version of the model and how it's performing means
24:58
because I'm consuming these models from a service provider,
25:01
I could still have the same application and it's performance
25:04
could either get better or worse without me changing a single
25:07
line of code. And if I don't have any sort of
25:12
tracking or validation against sort of metrics, I would be
25:17
leaving it to in somebody else's hands for the success of my
25:23
solution. OK.
25:24
And and also you mentioned testing prompts.
25:28
How would you do that? For example, how do you do it
25:30
with clients maybe or because you know, sometimes the answers
25:34
might be different. So it's also like, I guess you
25:36
can't really, you know, test for the exact same answer.
25:40
So how do you make sure that your prompts and the answer that
25:43
you get from your model is to what you expect was the correct
25:46
one? From a user perspective, like I
25:50
just ended like a project on that.
25:52
And for example, what we did was to to validate exactly the
25:57
answers with different versions of this model.
26:01
You had seen what we did with a client was really creating by
26:06
ourself data sample to to test that exactly the type of answers
26:13
given by the LLM with different versions exactly running behind
26:17
the scene. And we created this sample
26:20
because it was easier to trace back in the actual documents
26:24
that the client gave us the information.
26:28
And so we could immediately spot not just if the answer was wrong
26:35
or correct, but also if it contained the right element for
26:41
that we were looking for inside the answer of the model.
26:46
So it wasn't just about a wrong or good answer, correct answer,
26:51
but also qualitative information retrieval.
26:54
And then after doing this test by yourself, we got back to the
26:58
real users and check with them if these types of answers that
27:04
resonated with them. It was exactly the type of
27:07
answer that they would have retrieved from those documents
27:12
because that case was like them having to manually look in
27:16
through a lot of PDF in a very short time possibly.
27:21
And so it was the best key scenario where like by
27:25
implementing this new application with an NLM, they
27:29
could have done the same thing that day-to-day do in a very
27:34
painful and time consuming way. And so the testing of that
27:40
scenario was really super clear and easy to do because you had a
27:45
very quantitative testing first with the sample of questions
27:50
that you create at least 100 minimum.
27:54
And then you could get the qualitative testing with the
27:57
users and see how the really the quality and the format and how
28:02
the answer is created by the model resonates with them.
28:07
Yeah, I was going to actually add to that.
28:09
I will add. It depends on who the end user
28:12
of the, of the, of the LLM product is.
28:15
And I'll give you an example. We've done, I think one of the
28:17
cases was you listen to a call with an advisor and you're able
28:22
to transcribe that call and then you're able to summarise that
28:26
call so that the agent is actually helping the individual
28:30
or other than focusing on making notes and so on, right, or at
28:34
least making notes in a particular way.
28:36
Now, once the summary has been made, the advisor is able to
28:40
look at the summary, like it, update it or approve it.
28:44
And once they do that, they give you a feedback on how good the
28:49
suggestion was. And as long as, because the
28:52
effort to feedback is so low, you're getting it with almost
28:57
every single interaction. So when you do introduce a new
29:01
update to the model, you're able to see, am I getting more thumb
29:04
down or I get the more thumbs up?
29:06
And so you're almost using this AB testing mechanism or kind of
29:10
quantitative feedback cycle to figure out am I making end users
29:16
happy or not? Obviously this doesn't lend
29:19
itself to all applications, but for in the enterprise world
29:23
where the applications could be used by a few end users being
29:29
employees, you you get to capture that.
29:33
And one thing that I think about a lot is, let's say all of a
29:36
sudden you're getting a lot more thumbs down, is that then, you
29:41
know, you have to go and play around with the prompts and
29:45
like, what's the next step after you see, oh, things just went
29:48
haywire? Like, what are some
29:51
troubleshooting steps when that does start to go wrong?
29:55
You know what I mean? Yeah.
29:56
This is where I said LLMS was like, let's go back to the
30:00
principle we have the software engineering.
30:02
We'll be making applications for the for more than 30 years as
30:07
engineers, right? You should have the logging and
30:10
the monitoring in place to allow you to reach a diagnostic state
30:14
very quickly. And this is where I said from
30:16
the taker to the shaper, it's a very smooth transition.
30:20
If you treated it the software and you had all of the
30:23
infrastructure, the DevOps that envelops recommended and you are
30:26
tracking these things because you don't have to jump into the
30:29
logs to figure out what's changed.
30:32
Hey, Laszlo here. If you're serious about
30:35
envelopes, you hit subscribe right now.
30:42
You should actually be aware there was a model update.
30:47
You should be able to benchmark against your set that you've
30:51
created like Morrow is saying. And you know the performance
30:54
there was was not captured. And even when all of those fail,
30:58
you have the necessary infrastructure in place to be
31:02
able to replay the particular behaviour and figure out what is
31:07
different between today and yesterday.
31:10
When you get a huge amount of thumbs down, the first thing you
31:13
ask is what's going on? Then we have a problem, right?
31:16
You should be able to know at least what has changed between
31:20
the state of one and another. Yeah.
31:24
I mean, in traditional software, you roll back, but in model
31:28
updates, especially if you're using somebody else's model, you
31:31
can't roll back. So what do you do there?
31:36
That, that, that's one of the exciting things, right?
31:38
This is where it's, it's been a journey educating people, our
31:42
education organisations that when you're relying at the
31:45
service, you're bound by the services terms and conditions.
31:49
At a certain point, you're, you're going to be firefighting.
31:52
It's like when you have a downtime because of your payment
31:55
provider, did you buy 2 payments providers and thereby 1 is on
31:59
standby for disaster recovery And you've tested against it.
32:04
You know, do you have a, an enterprise agreement with that
32:06
service provider that they shouldn't switch off a version
32:09
before giving you three months notice and thereby you failed in
32:13
your own internal processes to, to test a new version?
32:16
A lot of this is answered in typical software agreements,
32:20
right? It's it's traditional software,
32:23
It's just there's data and then in analytics involved.
32:28
Yeah. And then it's very good on the
32:30
logic actually with the payment providers.
32:32
But it's like, you know, at least with different providers
32:35
depending on the country. I mean for us, for Volt, we're
32:38
25 countries. We have, I don't know how many
32:40
payment providers we use, but like hundreds.
32:43
But then you know, the quality might differ from 1:00 to the
32:46
other. So it's going to be the same,
32:48
you know, with LLMS like maybe your backup LLM will that you
32:52
may never use, like you don't really have tests for it or the
32:56
quality might not be as good. Therefore then you know your
32:59
user is not going to be happy. Like you know, if our customers
33:02
say if you talk to a customer support, for example, and you
33:05
decided that LLM was going to be the future and you fired
33:08
everyone you know, and then what?
33:11
Like as a backup, you use the LLM and then suddenly the
33:15
quality is not as good. How would you, how would you do
33:18
that? How do you try to solve that
33:20
maybe? There's a lot of analogies that
33:24
can throw at you. If you're constantly backing up
33:27
your databases but you've never restored them, then you know
33:31
you're never going to figure out if that backup work.
33:33
So it's like if you do have a backup provider, but you're not
33:36
testing it during your releases, then it's not necessarily a
33:39
backup provider, right? It, it, I've signed up to it.
33:43
Hopefully we never use it and that could be bad.
33:47
I think there are scenarios where we can definitely learn a
33:51
lot from traditional software, cloud, DevOps and ML OPS.
33:54
There are scenarios where it's new territory just from the
33:58
notion that, you know, it's the, the model is in somebody else's
34:02
hand. And I have to agree to the given
34:06
terms and condition. Now in the realm where we can to
34:10
do things, I'll give you an example.
34:12
If you're subscribed to multiple models or you're even running
34:15
your local models as well, we already know like from the
34:18
DevOps and the envelopes, you can run your user request across
34:24
other models and keep tracking and see how they perform, right?
34:28
You may not do it to 100% of the traffic.
34:30
You may only do it some sort of kind of IT Canary 1%, two
34:34
percent or shadow traffic, what we'd call.
34:36
And that will just constantly give you a benchmark of what is
34:40
your second tier and 3rd tier model should you need to use it.
34:44
And you could even say, you know, my provider has provided
34:47
me a model and a new model gets released every month.
34:50
Thereby I'm actually going to test the new models and keep
34:53
tweaking my prompts and with the live traffic to see how they
34:58
perform. So that's a traditional DevOps
35:02
ML OPS practise to, you know, leak some traffic to a new
35:06
model, evaluate and so on. We're talking about using AP is
35:11
right now and we're talking about using external services.
35:14
What about when you decide to do this in house and you are using
35:19
your own model or you're using a an open source model that you've
35:22
grabbed off the shelf? How have you seen things differ
35:26
there and what in your eyes have you seen as far as trade-offs
35:29
go? I think open source is
35:31
definitely the future and it's an important dimension to always
35:34
keep where where a lot of companies and where a lot of
35:39
teams trying this out got caught is that for these managed
35:43
services, you got a credit card and an API endpoint and a
35:46
prompt. That's it.
35:47
You suddenly had to forget that somebody was managing this
35:51
service, running this service 24 hours, setting up all of this
35:56
amazing infrastructure and skeleton for you just to be able
36:00
to send that request any time of the day and get a response.
36:04
And that skewed the effort to consternation you of why can't
36:11
we use our own model? It's just free and it's
36:14
downloaded. And so when companies go in with
36:18
that mentality and you quickly fail and you fail very fast and
36:21
you say this is not for us and you just go back to the taker
36:24
approach, but there's a lot of challenges.
36:28
I think before going into that, when you are going to host your
36:31
own open source model, you have to think of all of this, which
36:34
is how do I orchestrate the infrastructure?
36:37
How do I ensure this model is running on an optimal cost
36:41
model, whether you know, there's one request, 0 request or 1000
36:45
requests? Have I done enough benchmarking?
36:48
Maybe I don't want to run it on a Google scale, but I do want to
36:51
run it to my, you know, predicted organisation scale.
36:56
And once I have done that, what other solutions do I need to
37:00
accompany my model, whether it's caching the same query, whether
37:04
it's, you know, providing optimal hardware and optimal
37:08
setup. Once you've done that, yet a lot
37:12
easier to replacing the models, replacing APIs, experimenting
37:16
with different models, because what you've achieved is you've
37:19
treated this as a real application, the real software
37:23
engineering application. And you said how do we introduce
37:27
new dependency, how do we sunset dependency and how do we track
37:32
and build talent in our team in order to ensure we know how to
37:37
use it? You know, no, that's, I mean,
37:39
it's very interesting. And also I'm a full believer of
37:41
open source. I mean, a whole other platform
37:44
is built on open source. So completely agree on that one.
37:47
But maybe as a follow up, because you know, then use like
37:49
a lot of companies have been publishing their models, they've
37:52
been like, yeah, it's fully open source.
37:55
But if you take the licence, you're like, yeah, it's fully
37:57
open source if you're it's on your computer, like Meta, for
38:01
example, released a really big bottle, whether they were like,
38:04
if you have more than a 700 million users it you can't use
38:07
it commercially or Hugging Face, you know, like they had a bit of
38:11
trouble last week, I think, or two weeks ago about like a new
38:14
licence. What do you think of those
38:16
things? Because, you know, like
38:18
technically it's open source, but then in my opinion, if you
38:22
can't really use it commercially, I wouldn't call it
38:25
open source. I don't know.
38:26
Also, if Mara, you have an opinion on that Mohammed?
38:30
Yeah, I think that especially metre example, I mean Llama,
38:35
it's open source and you can use it commercially and that the
38:38
game changer. Honestly, I'm like you a very
38:42
big supporter of open source. And I think that what they have
38:46
done, I mean some of them, you're right, they have if you
38:50
check the licence, they have a lot of caveat about the use that
38:54
you can actually do with them. But like Meta's example shows a
38:59
completely different scenario because now you can use it.
39:04
Of course, the performance is not like GPT 4, but it's like
39:09
GBT 3.5 and maybe it's enough for many organisation, for many
39:14
use cases and it's open source and it's really bringing
39:18
something different to the competition and to the way that
39:22
the community can also keep developing and improving this
39:26
model without having to pay a third party providers.
39:31
I can be more confident about the use of that I do of my data
39:35
and know the privacy and security issues.
39:38
So I think that what we're seeing, it's very interesting.
39:42
And again, for for Muhammad's point before it can be traced
39:46
back to the older software engineering school and the older
39:51
history of open source from the very beginning, right.
39:55
I think we're going to see a similar trend like what happened
40:00
in the past. And I think the competition now
40:03
that is urging from the open source models, it's gonna bring
40:08
us the different business choices.
40:10
I mean bringing toward party providers like Open EI, making
40:15
different business choices in the terms of how users can do
40:20
and and pay for it for their for using their models.
40:24
So I think it's an interesting time to see a lot of challenges
40:29
that I had. So you all get to experience
40:36
lots of different companies out there.
40:39
I have this lingering question in my head, Do they really care
40:45
about privacy or do they care about convenience?
40:48
And like on that scale, where does it sit?
40:51
Because I know in theory privacy sounds really good, but then MO,
40:56
you were talking about all these things that you need to do that
41:00
are much harder than just hitting an APII.
41:04
Think to to simplify it, right? There's there's companies that
41:10
are really serious about privacy.
41:11
There's companies that are really serious about how they
41:14
get how they capture data, how they use data, how do they track
41:18
data and how do they respect the customer's consent, Because it's
41:22
not just about the privacy. It's also about once I pulled,
41:27
you know, I no longer accepted my your terms and conditions.
41:31
Are you fulfilling your obligation to clean me out of
41:33
the system? And there's, it's a wide
41:37
spectrum from there are companies that love and stand by
41:42
it and that's just based on their values and heritage.
41:45
And then you go all the way to the spectrum of you've got, you
41:48
know, you have to because of fear of news and so on, and then
41:51
slowly ramping up the regulation.
41:54
And so is it a serious concern for businesses doing to to
42:01
continue doing their business? Yes.
42:03
Is it all for the same reason? Not essentially, but as we
42:07
slowly realising like for example people analytics like
42:12
the especially with the ease of this L&M and how companies are
42:15
suddenly able to adopt it in such a short period of time, it
42:19
means I can integrate it into some process very quickly.
42:24
How much does it breach on the regulation need in terms of, you
42:30
know, providing a fair service to people of various categories,
42:34
whether it's ethnicity, religion, age and so on, is how
42:38
we can see a huge push because of regulation as a wake up.
42:43
It's not necessarily been pushed back as a business, but I think
42:47
what regulations are providing is a mechanism of what should we
42:52
look at and how should we react when we deal with our customer
42:56
data. So you can almost think about it
42:58
as traditionally a lot of companies have come up with
43:01
their own way of evaluating this.
43:03
Slowly, with the maturing of various regulatory bodies and
43:08
what they're asking for, we're reaching a consensus of how do
43:11
we achieve this as a society and how do we evaluate it.
43:17
And I would add that finally we're getting to to this point,
43:21
right? Because like, if we think just
43:23
the two years ago, Trustworthy I regulations were just nice to
43:29
have for organisations, I mean, especially Trustworthy I.
43:33
And now finally with text to the LLMS actually and what these
43:38
models brought to the table. Finally, there's been a big push
43:43
on AI regulations in different countries and before.
43:47
Instead, if we think just the 2021 when the EUAI Act first
43:54
started, no one really cared about it, no one really talk
43:58
about it. Now it's all on the 4th.
44:01
It is of all the media outlets that that gives an indication of
44:05
how important it is for giving a framework to organisation and
44:11
how really use and adopt safely these models especially for the
44:17
privacy concerns right and how they handle customers data on
44:22
that. I think we are learning a lot
44:24
and there will be needed them in many, many iterations to get to
44:29
the right framework, to the right way to do it and ensure
44:34
privacy and I mean safety eventually on the use of these
44:39
models. Yeah, and the other funny part
44:42
is that maybe the company bans it, but you can have the rogue
44:49
employees who copy paste into ChatGPT and because it helps
44:54
with their workflow. And so it's almost like water's
44:58
going to find the easiest or the least resistance, the least path
45:02
of resistance. But what I'm really interested
45:04
in hearing about it if you all had to score it and maybe both
45:08
of you can give answers at the same time or something so you
45:13
don't bias each other. But what would you say the
45:16
companies that super are in one camp versus the other?
45:22
Like they're a little bit more relaxed.
45:24
They're OK with being a taker and using the off the shelf AP
45:28
is. And then looking at it as far as
45:30
like percentage wise versus the ones who understand that they
45:35
can't do anything with the AP is because at this moment in time,
45:39
they're not able to. I would love to hear if you all
45:42
have just an idea off the top of your head, like who in
45:48
enterprises that you're looking at, where do they fall?
45:51
What's the percentage split like?
45:54
I would say the majority and I will caveat that in a second.
45:57
So that means more than 70% are concerned doing something about
46:04
it. Oh.
46:06
I second that. Really.
46:09
OK, so somebody told me. OK, interesting.
46:12
Name is the caveat, right? The caveat is education, which
46:16
is once leadership understands the possible use cases and how
46:23
it can be either used as part of accelerating the day-to-day work
46:28
or what kind of security implication it causes by leaking
46:32
IP and so on and so on, They know it's very difficult to
46:36
contain, as you've said, right? Which means now can you put
46:41
enough and fast enough guardrails to stop people using
46:45
it? Or do you just have to make sure
46:49
you provide an adoption path in your organisation that is safe,
46:54
that is monitored, that is accessible to ensure that your
46:57
people aren't using, you know, a, for a lack of a better word,
47:02
the black market version or under the table LLM to do their
47:06
work. And we see that, right?
47:08
Which we see it a lot, whether you see a lot in the news of
47:11
people leaking, you know, IP through asking the LLM or people
47:17
are posting silly things that you know that shouldn't be asked
47:22
to to models. And so any organisation I have
47:27
worked with that understood the implication with the benefit
47:31
that LLMS can give to an individual.
47:34
Remember that if used correctly, this is a driving factor for AI
47:40
adoption in an organisation. Traditionally you had to
47:44
persuade people to use your anti new analytical product.
47:48
If you have a workforce that is eager and hungry to use
47:52
analytical products, this is the right time to provide a safe
47:57
mechanism for them to use it because it will ensure as you
48:00
roll out more products, whether it's accelerates them, whether
48:04
you know it, it helps them do their job or not.
48:06
Secure and and more accurate. They're willing and they're more
48:10
acceptable of it and the the second, which was always going
48:15
to be the risk, right? The risk and the security.
48:18
You can't stop people using it on their phone.
48:20
You can't stop people using it on their second laptop.
48:23
People work from home a lot that like you've practically got an
48:27
entire breach of security perimeter.
48:29
We're going to have to do infosec all over again.
48:31
And guess what? People just type texts to this
48:34
interface and get an answer. It's not even code.
48:37
So how can you philtre it from their e-mail?
48:39
How can you like it's? It's a huge topic to talk over.
48:44
Fundamentally, anybody that understands it's potential at a
48:47
risk from an organisation would do the best they can in order to
48:52
provide their org with it in a safe medium to ensure that
48:58
they're not putting the organisation at risk and that
49:00
people are able to benefit so they're not left behind as well.
49:04
I feel like Mara has other thoughts that you want to jump
49:07
in. What else do you want to say
49:09
there? I see you.
49:12
That 100% with Mohammed, especially on the topic of
49:16
education in the sense that today, I mean trust drives
49:21
adoption. You can't really infuse that any
49:25
of these models and you have the new technology in an
49:28
organisation without really making employees understand how
49:33
to use them safely. And you can do it by really
49:37
start ensuring that you have AI governance programmes again, a
49:42
nice to have just until last year.
49:46
Now something that you must start building to get exactly a
49:50
safe path there in well defined frameworks for people
49:56
understanding not just how to use these, but also what to
50:00
build with that what the impact eventually on your customers.
50:03
So what are the risks the the possible effects, the bad
50:07
consequences that they might create and cause and work
50:12
backwards on mitigation strategies to really preserve
50:18
your company is from these risks and all the penalties associated
50:24
and the brand reputation risk. So all of that you really, we
50:29
need governance word today and AI governance board that really
50:33
has the a word on what's being developed in your organisation
50:38
and understands the risks also to, to better manage those
50:43
things eventually. So it's really about a new
50:46
parenting shift on how we approach AI and educational
50:51
support employees about data EI literacy eventually because some
50:56
of them might not know anything about this, maybe the last, the
51:01
first time that they heard about it, it was when exactly Charging
51:04
PT was releasing back in November.
51:07
Maura, you mentioned a few ways that you can mess things up when
51:11
it comes to using AI and why responsible AI frameworks are so
51:14
important. I would love if you just humour
51:17
us and tell us other ways that you can fuck things up when
51:20
you're using AI and how you can make things go all pear shaped.
51:26
Yeah, yeah, many, many cases. I'm laughing because I remember
51:32
one of my colleagues actually where we were working on a
51:35
project about with a high personalization.
51:38
We were doing the testing with the model and everything.
51:41
And to really test the model and understand, I mean, we were
51:46
trying to evaluate some metrics like hallucination, toxicity,
51:51
harmfulness and see how the model would respond, which is
51:56
sound like toxic user inputs, right?
52:01
And some very, very bad things that a user might have to tell
52:07
to the model. And then remember that there
52:11
were these very sorry for my friend, just shitty
52:15
conversations where we really engage with the model to really
52:20
push it to the limit to see how bad the things.
52:23
I mean, the answer could get up to really then build gathering
52:29
systems around it, right. And and so I was laughing before
52:34
because I remember like my colleagues saying stupid things.
52:39
Those are terrible things, really pushing the model to the
52:42
limit. But if you think, I think that
52:45
people can really create now very easily boats that
52:50
disseminate toxic content full of hate, I mean that they can
52:57
even skew someone's political, I mean political idea on a
53:02
candidate over elections like we are all talking about right now.
53:09
Things can really, really get bad.
53:12
The things that scared me the most right now, in fact, is the
53:17
this information, honestly, most of everything.
53:20
This is where I think, yeah, I can really get very bad because
53:27
we would need more, more guard race, more ways to check the
53:32
trustworthiness. I was expecting you to say when
53:37
you were talking about the bad things that you were or the
53:41
silly things you were talking to the model about and evaluating
53:46
it. And I was expecting you to say
53:47
something like. In the end, we got the model to
53:51
say that you could put pineapple on pizza.
53:56
That that no, for sure, no. I will create gantries and
54:01
prevent the model to say such a freak.
54:04
I mean every Italian. So no pineapple on pizza.
54:07
Never ever. I had a feeling you would think
54:12
that was sacrilegious. I just had a feeling in Inkling.
54:16
But MO, you look like you wanted to jump in and say something
54:18
else too. None other other pizza they the
54:22
question of the whether whether something whether they're on
54:28
Prem or the self hosted versus the service.
54:31
This has been a really interesting conversation and
54:33
it's even more going back to Stephen when you asked earlier
54:36
for the licencing. We've noticed that the
54:39
definition of open source has been tested to the limit with
54:42
LLMS. It generally isn't, but with LLMS we were just getting
54:46
all sorts of and I really opinionated about this because
54:49
there's only one definition generally of open source and so
54:52
you can't try and use it for marketing purposes.
54:56
But what I noticed is that if, and you've probably looked into
55:00
it, if you spent long enough time seeing how the ripple that
55:04
the first Llama model released by Meta was grabbed by
55:09
researchers, then downloaded, then distributed through various
55:12
channels like torrent and so on. And how the weights were leaked.
55:16
And then people started rewriting it in more performant
55:19
languages. And now some like some, some,
55:22
some engineer in spare time, Harry wrote in C++ from the
55:26
Ukraine, I think. And then we have LLM compiling
55:31
on your local laptops without GPU, with just ACPU.
55:35
Then it you you're able to push it on your phone.
55:38
Now nobody had anticipated you could run such a large language
55:42
model on assault. And when you think of the open
55:48
possibility of making sure you having a permissive licence and
55:52
a open source community testing all the possibilities, you're no
55:56
longer just testing 1 hypothesis by 1 product team of how their
56:02
product should be used it by. I think somebody was mentioning
56:06
to me that this was one of the reasons that Meta ended up
56:10
changing their licence from the initial and and to to adopting
56:14
Apache with a special clause of 700 million, which if you read
56:19
into it all that means and they're likely saying we don't
56:23
want other cloud providers to offer the service.
56:26
Now I understand it's not the full definition of open source,
56:29
but at least I understand who's the intended target.
56:33
And from that, I believe with the the rise of open source and
56:40
with at least any source of knowledge leaking into the open
56:45
community, we're going to get faster cycles of change that a
56:49
single company can't afford to do because of budget and human
56:53
resources. And that's why it's super
56:56
important. And I don't believe like in 10
56:59
years time, every company has to develop its own LLM from
57:04
scratch. But I do believe as long as we
57:07
continue to push the remit of what is open source and making
57:10
sure that we know what data it was trained on, how it was
57:14
developed and the code and so on.
57:16
That you're able to pick up a solution of the market or of the
57:20
community that you can use as a starting point or a combination
57:25
to fight or to tackle whatever challenge you had without
57:31
arguing on the compromises we're talking about now.
57:33
Whether it's a change of version or a change of terms of
57:37
conditions of how you use a certain service.
57:39
And maybe to like preempt it with like a small thing, the
57:45
LLMS and what they use today, there are small patterns of
57:49
interactions with LLMS. The real magic is still within our data.
57:53
It's what we provide the LLMS. It's our application.
57:57
When you see like to like a really small one, content
58:02
extraction, content summarization, we need an LLM to
58:06
be super good at it, but it's our data that's bringing in the
58:09
value. The problem is solved not by the
58:11
LLMS into a kind of knowledge, it's by the LLMS knowledge of
58:15
achieving what we wanted to do. And thereby, once we do get
58:19
general models that do those tasks really well, and we do get
58:24
large enough contexts to solve our problems, we should be able
58:27
to chain these purpose built services, open source or
58:32
whatever they could be, and we'd be in a better place.
58:38
Yeah, no, but true. But I completely agree with with
58:41
what you said, but and also to come back to what Mara said a
58:44
bit earlier, like I feel like and tell me if, if I'm
58:47
completely wrong on that one. But you know, it's like with
58:49
like open sourcing your models. It also helps a lot with like
58:53
safety wise, you know, because everyone can actually have a
58:55
look at it and I can, you know, like evaluate it and you can
58:59
really make sure that yeah, from I don't know if it's from a
59:03
society point of view, but at least, you know, you have like
59:06
other people can have like a different look at the model and
59:09
be like, OK, is it safe or are we are we doing something wrong
59:12
with it? Yeah, I, I actually, something
59:16
comes to mind. This is really crazy.
59:18
And I, I sent it to a friend of mine where a, a tech, I think a
59:23
tech influencer, she's a, she's a leader in some industry.
59:27
I can't remember had used a generative AI model or a app to
59:32
enhance her picture that was bad.
59:34
And she was of East Asian origin and it made her kind of
59:38
Caucasian. And so like this kind of thing
59:43
is only there because the models have only been tested on a
59:48
certain thing. If we don't have these models
59:50
available to everybody so that we can test what is right in the
59:54
US, is the same in Europe, is the same in Asia, is the same in
59:58
Africa, we're going to end up with a very self centred model
1:00:03
that only believes their own opinions are right and whatever
1:00:07
data they've trained on is right and everybody else is right.
1:00:10
So I mean, then we can go down the rabbit holes of how the
1:00:14
cultural influences you have and like you said, the data that you
1:00:19
gather from all these different places will add to the model and
1:00:24
how that changes things. I think that's a fascinating
1:00:26
conversation. I think it's probably one for
1:00:30
our next chat though, because we've started to hit up on time
1:00:34
and I want to end it there. I want to end it with a joke
1:00:39
though, because you said it, MO, you led me into this and it was
1:00:44
around how you said, you know, there used to be only one type
1:00:49
of open source. And I thought, yeah, that was
1:00:53
the kind where you would open it up and then Amazon could come
1:01:00
and steal it and run it as a managed service.
1:01:03
And now there's all kinds of different ways that Amazon can't
1:01:08
do that. You can thank mongo DB for that.
1:01:11
Yeah, exactly. So the good folks at Mongo DB,
1:01:16
we are appreciative. And it's the other last piece
1:01:20
that I'll say is the interesting piece about open sourcing in the
1:01:25
AI world is that the data also has its different open sourcing
1:01:29
licences. And so you have, I know we open
1:01:33
source the ML OPS community data, the Slack data, and we
1:01:37
made it so that anybody could use it, but nobody could build
1:01:41
like a commercial product with it.
1:01:43
And you have to like think about all these things when you have
1:01:48
important data and if you want people to use it, you want
1:01:51
people to play around with it or build things with it, like how,
1:01:56
how are you going to allow that? Or at least I, I had to think
1:02:00
through it. Like, do I want people to get
1:02:03
rich off of all this data that's been created?
1:02:06
Well, not if they're not cutting me in on it.
1:02:08
Let's be honest. They got to talk to me.
1:02:10
You want to build a business, talk to me about that.
1:02:13
But anyway, folks, this has been awesome.
1:02:16
I think we'll end it there. OPS, you have to immerse
1:02:21
yourself in the ML OPS content. The best way to do it is to
1:02:27
subscribe to the e-mail OPS Community podcast.
1:02:31
So good luck and keep learning.