Latest / Agentic Conversations (formally mlops.community) / A Blueprint for Scalable & Reliable Enterprise AI/ML Systems // Panel // AIQCON

Transcript

A Blueprint for Scalable & Reliable Enterprise AI/ML Systems // Panel // AIQCON · Agentic Conversations (formally mlops.community)

0:09
All right. We'll go ahead and get started
0:11
here. So once again, our next topic is
0:14
a blueprint for scalability or scalable and reliable enterprise
0:19
AIML systems. Please help me.
0:20
Welcome to this stage, our next 4 speakers, Hira Dongle with
0:24
Bank of America, Rama Akaraju with NVIDIA, Nitin Agarwal with
0:29
Google and Steven A Luke with IBM.
0:33
Come on up. Good afternoon everyone.
0:58
I hope each one of you are having incredible time out here
1:02
at San Francisco based this AI quality conference that's
1:08
happening. So super excited to be part of
1:11
today's this phenomenal discussion.
1:14
My name is Hira Dangal, I work for Bank of America and I lead
1:19
AIML and automation platform development.
1:22
So for the next 30 minutes, we will be spending on this
1:28
specific topic called the blueprint for AIML systems,
1:31
right? It's kind of like the the
1:33
interesting timing for us and how you can get some of the
1:38
insight tool interest and information from my incredible
1:42
panel out here from the enterprise landscape.
1:47
So we'll try to see if we can get some time for Q&A towards
1:52
the end. So I'm I'm joined by my
1:55
incredible panellist fellow from across this kind of the industry
2:00
spectrum, Rama, who is from the NVIDIA and then needing in fact,
2:07
I think the side needs to have a chance that he just moved to the
2:10
Microsoft. So it's more like, you know,
2:11
Google to Microsoft kind of the transition phase and then we
2:16
have Steve from IBM. So so without further ado, I
2:20
think I we can get started with some quick intro to to to begin
2:24
with, right to stay the stand. And then we'll talk about, you
2:28
know, the biggest kind of the the the blueprint, the focus of
2:32
our discussion to think about the how vision and its the
2:35
strategy execution to like from the GNA perspective.
2:39
Maybe we'll cover a little bit about the more on the, you know,
2:42
value proposition of aspect of the, you know, this blueprint.
2:45
And then AI risk management is the incredible end of the most
2:49
topic today. We, we, we need to focus on and
2:52
then how this leads up to the adoption phase, right.
2:55
So with that, I think I'll let Rama to quickly introduce
2:59
herself and what's the most fascinating things happening in
3:02
the NVIDIA side and I will take on the other side too.
3:05
Yeah. Thank you, Hira.
3:06
Thanks for having me here. Can you all hear me OK?
3:09
Yeah, great. My name is Ramaki Raju.
3:13
I lead the AI and automation team at NVIDIA, specifically
3:17
reporting to the CIO. My mission is to transform
3:21
Nvidia's own enterprise and leverage as much of Nvidia's
3:25
technology as possible along the way.
3:27
So I look at not only IT transformation, which is where I
3:31
sit in the, the CIA organisation, but HR, sales,
3:35
marketing, supply chain, finance, all of those business
3:39
processes and business functions are up for transformation with
3:42
AI and generative AI. So I work with all of those
3:46
business functions to apply AI generative AII.
3:49
Look forward to sharing some of those, you know, lessons learnt
3:53
and experiences. I've been with NVIDIA for two
3:56
years. Prior to that I was an IBM
3:58
fellow. I worked at IBM for much of my
4:00
career. I was the CTO for AI OPS and
4:04
also in the initial phases of Watson development led the
4:07
Watson platform and natural language processing, speech
4:09
recognition and all of those foundational AI services which
4:14
we now take for granted. So that's about me.
4:17
Nitin. Can you hear me?
4:19
Hey everyone, I'm Nitin the title that you see.
4:22
That's my old life. So I currently lead the
4:25
generative AI team for marketing at Microsoft.
4:27
We are building various kind of assistance in a simpler terms,
4:30
you can call it as a Co pilot for marketing focusing on
4:32
content creation. We're talking about ideation,
4:34
talking about analytics platform, creative assistance
4:37
and so on and so forth. So that's a part of a team
4:39
including product managers and engineers.
4:42
This particular topic is very close to my heart.
4:44
It's about like the scalable and enterprise ML systems.
4:48
And the way he asked that question, what's most
4:50
fascinating, it's the consumer for these kind of assistance.
4:53
In the past, those were very boring systems typically being
4:57
used in the form of an AP is integrated with some fancy UI,
5:01
but in with these kind of a generative AI and LLS, the AI
5:04
systems, the core AI systems are being used by executives and
5:08
business audience. So how those kind of a system
5:11
has been evolved and they're evolving over a period of a
5:13
time. That's what fascinates me a lot.
5:15
So great to see you all. A pleasure all.
5:20
Steven E Luke from IBM. I've spent the better part of
5:24
seven years building out the corporate services for IBM in
5:27
terms of data, data standards, data governance, regulate or
5:32
governance around AI and ML OPS and AI OPS and AI platforms,
5:38
data privacy. Doing all of that internally
5:40
for, you know, our chief financial officer all the way to
5:43
our legal officer and to our privacy officers.
5:46
Building the services that make up the backbone for IBM all the
5:49
way through from corporate to software to consulting.
5:53
So supercharging about 300,000, you know, employees throughout
5:56
the company working across 150 plus countries, working across,
6:00
you know, 10s of thousands of AI and data assets, trying to aid
6:05
in that consistent approach where you have A and platform
6:08
for all users, you have a consistent governance structure
6:11
for all users and all applications and really trying
6:14
to add as much automation to these places as possible.
6:17
So a lot of instrumentation, a lot of understanding as a
6:20
practitioner, what goes into building out AI use cases and
6:24
trying to take that know how to supercharge those that have not
6:27
had, you know, the experience working in these domains.
6:30
Now before that, go back to things like Imagenet and
6:33
sponsoring, you know, interesting peoples at
6:35
interesting universities and had the my pleasure to work at
6:38
Samsung Electronics and built supercomputers and worked on
6:43
deep learning and distributed deep learning and model parallel
6:46
distributed deep learning and just love that we're all here
6:49
talking about this ever evolving space.
6:52
All right. Thank you, Steve.
6:55
So today's topic as we will talk about, you know, blueprint for
6:58
the scalable, reliable and the secure deployment of the AI
7:03
system. We're talking about that, right.
7:04
I think to begin the conversation on here, I think
7:08
how how would you make sure that the AIML initiatives or the the
7:15
work that you drive aligns with the business objectives?
7:19
I think you might be working mostly with the leaders out
7:21
there, right. So what are the kind of the the
7:24
basic requirements are needed to to to have that kind of
7:26
alignment and the buy in from the, you know, business
7:29
objectives standpoint? Yeah.
7:32
Yeah, of course, that's where it all has to start, right.
7:35
AI or any technology for that matter is a means to an end.
7:39
That end is that the business, whatever business metric that
7:42
you're trying to improve, you have to have a plan for how this
7:45
technology that you're going to use is going to help you improve
7:48
the the state from where you are to the next state.
7:51
So I can give you some specific examples.
7:53
We are working on applying AI, generative AI for IT
7:57
transformation and that includes IT operational improvements
8:01
would be, you know, improving SRE productivity.
8:04
So how do we measure? So before we start these
8:06
projects, we, we say, well, we what, what are our current set
8:10
of metrics that we are measuring?
8:11
What is the baseline, the meantime to detect incidents,
8:14
the meantime to resolve them, meantime to diagnose them and
8:18
how much is the business impact that's happening?
8:20
How many incidents are happening within a month?
8:22
So these are all the metrics that we take as baseline and
8:25
say, OK, if we do incident summarization, if we did better
8:28
detection of incidents and if we implemented, you know,
8:32
additional Gen AI capabilities for SRE productivity in terms of
8:35
chat bots or, or better integrations and better alert
8:38
noise reduction capabilities, better anomaly detection
8:42
capabilities, let's say, how are all these metrics going to
8:44
improve? And from time to time, with each
8:47
iteration of the AI model deployments, we have to measure
8:51
and see if we're, you know, making progress in the right
8:53
direction to improve this business metrics or not.
8:56
That's one example. And you know, I can talk about
8:58
other examples in employee productivity domain and supply
9:03
chain domain. You know, if planners are taking
9:05
five hours, 3 hours to do what if scenario analysis each time,
9:09
let's say we now have capability to do GPU optimization and apply
9:14
the GPU optimization for redoing the planning.
9:16
But with what if scenarios now you can do it in 3-5 minutes.
9:20
You know, what is the amount of time that we are saving for the
9:22
planner. So that's another business
9:24
metric, productivity metrics. So we for for every project
9:28
there is a baseline, a set of metrics, and then as we are
9:31
making iterative improvements, we continue to measure those and
9:34
see if you are making improvements or not.
9:36
You know, it's easy to get carried away with technology and
9:39
get excited. And sometimes, you know, you
9:40
say, you think you know, conversationalizing everything,
9:43
building chat bots is the right way to do it.
9:46
Maybe it is, maybe it isn't. If something can be done within
9:49
one click, you don't want to make it like a five different
9:52
turn conversation. In the end, the experience wears
9:56
off after the initial month or two and people will say, why do
9:59
I want to come here? You know, I just want one click
10:01
and things will should get done automatically or even maybe
10:04
automated. No click is the right solution
10:07
for it. So you really have to understand
10:10
what is it that you're trying to optimise and what is the metric
10:12
that you're going after and design and use technology
10:15
accordingly for that purpose. It's a means to an end.
10:19
Yeah, that's definitely help out, right, In terms of defining
10:22
the use cases. And I think from the audience
10:24
standpoint, our discussion will, most of our discussion will
10:28
align to when we say aim or system means there is 2 two kind
10:31
of distinct phases, right? There is a generative AI
10:34
component and then we have traditional data centric model.
10:37
That's how the most of the enterprise are kind of currently
10:40
deploying those kind of use cases out in the production.
10:44
I think Nitin, maybe you might be able to shed some light based
10:49
on your previous roles working a lot of field work, right?
10:52
In terms of taking, if there is some sort of like the best
10:56
practises taking any use cases from prototype to production,
11:02
what involves in this kind of the phases, right?
11:04
That would be something you can give some examples to, right?
11:08
So for this would be helpful to us to understand.
11:10
That's an interesting question. So when you're talking about any
11:13
production level systems, two key things that business
11:17
audience are looking out for either to gain some operating
11:22
efficiencies or to get some new net revenue.
11:25
These are the two key metrics that has been seen across any
11:28
AIML domain. We started with data analytics
11:30
back in 20, 2012, 2013 moved into statistic analysis.
11:34
We moved into machine learning, then deep learning, now journey
11:37
DVI. These two metrics remains the
11:39
same. When we're talking about
11:41
production level systems. Again, these two kind of a
11:43
metrics start coming to the picture.
11:45
Who is your end user? Is it a business user or is it a
11:48
technical users? How you are integrating it?
11:50
What kind of an accelerators, what kind of a platform that
11:52
you're using it, what kind of a metric that you are currently
11:56
monitoring metrics, it can be business metric, it can be
11:59
technical metric. In the old ways we used to say
12:02
precision, recall, F1 score, accuracy, those things.
12:05
These days we started replacing them with blue score, Rouge,
12:08
everything but that kind of a metric, but few metrics at the
12:11
system level. For an example, latency
12:14
throughput, that remains the same in trustedly, in my
12:17
experience, people are getting a little bit impatient.
12:20
They used to wait for a few minutes, few seconds to make a
12:23
call to their AI models. But with Gen AI, they need
12:26
everything in real, real time. Hey, I'm typing it.
12:29
I don't even type. I don't want to type.
12:31
You should be able to understand what I'm saying and make a call
12:34
and generate a response. So those system level metrics
12:37
are being changing. We used to say that, OK, five
12:39
nines, 7 nines of system availability, that might be
12:42
going to 10 nines. So even like A1 failure in one
12:46
year, that will be considered it as a deal breaker for those kind
12:49
of systems and accelerators. So talking about some of the
12:52
best practises, definitely going back to the first principles to
12:55
whom you're building that particular system, what kind of
12:57
a metrics that you're monitoring.
12:59
And the second thing is that OK, how robust and how futuristic it
13:02
can be, Just having a fancy layer of technology may not be
13:05
able to help you out. Yeah, that's that's not true,
13:09
right. I think everyone agree that data
13:13
today is more like oil to economy, right?
13:15
So without data, there is no AI solution, without data, there is
13:18
no model or data centric models can be built.
13:22
I think Steve, maybe you can help us to kind of like uncover
13:26
some of the insights around the how the data silos can be
13:33
managed to ensure that there is a efficient and seamless
13:38
integration that can happen across the enterprises, right.
13:41
So your data is very much fragmented across the
13:44
infrastructure. What kind of like you know, the,
13:46
the, the concept and the governance piece could be
13:49
involved there. So just.
13:51
I think it's, it's the consistency of the practises,
13:54
right? So everyone has a justification
13:56
of why their data should be siloed, you know, and sometimes
13:59
it's it's rightfully so. Sometimes it's customer data
14:01
where you can't have that data, you know, commingled with other
14:04
pieces of data. But it's really about the
14:06
understanding of the data that needs to be consistent.
14:09
And data standards really play a key role in that because if you
14:12
have the same level of standards and you can search across that
14:15
data in very consistent way. So let's say you're looking for
14:18
certain aspects of privacy related data.
14:20
If you have the same standards for privacy related data across
14:23
the business, you're you're all measured against that same
14:26
standard, right? Let's say you're trying to bring
14:28
together similar types of data for a specific use case.
14:32
You do searches across the standard, you bring that data
14:34
together, right? And we all know the data game,
14:37
right? So the more data that you have,
14:38
the better the models get. So that siloed approach is
14:40
really problematic. You just basically want to be
14:43
able to understand when there's restrictions around the data
14:46
that might prevent you from, you know, taking it out into product
14:49
or, or releasing to a customer, things like that.
14:52
But I think the other really important thing to realise is
14:55
that like it's never one and done.
14:56
This stuff is changing all the time.
14:58
There's new regulation that's coming, there's new use cases
15:01
that are coming. This stuff is always evolving.
15:03
So think of the sustainability aspect.
15:06
So don't make a decision today that will be really hard to
15:08
implement tomorrow. You know, so like think of
15:10
regional constraints of data or sovereignty aspects that are
15:13
coming from certain countries in the world.
15:15
These are real constraints that will hit you.
15:17
And it's not just about your data.
15:19
There's country restrictions, there's, you know, geographical
15:22
restrictions about how data can move throughout your own
15:25
systems, right. So always understanding, you
15:27
know, the implications of the data that you're working with
15:30
and then the benefits of like potentially consortiums of
15:33
groups that come together as well.
15:35
You know, there's different ways that we can share data, you
15:37
know, sharing weights. You don't actually have to share
15:39
the raw data. You know, there's different ways
15:41
that you can handle the same problem.
15:42
It's about the awareness about, you know, how it plays in
15:46
practise and latency constraints too.
15:49
Is it a latency aware problem? Do you have to bring that data
15:52
close? Do you have to cache it?
15:53
Like all of the basis is go back to understanding what the
15:57
problem is in the use case that you're trying to reflect, right?
16:00
But again, inconsistent approaches at very large
16:03
companies just doesn't scale. So at a small company, you might
16:06
be doing your own things, but when you want to really scale
16:08
things, consistent approaches across the company are really
16:11
the ones that really matter. I can tell you probably, you
16:14
know, hundreds of use cases internally that have unfolded
16:17
just because these consistent practises, you can think of
16:20
financial reporting as you can think of normalisation of IT,
16:24
metrics that are coming back. If it's inconsistent, how do you
16:27
commingle that data? How do you create a viable, you
16:30
know, AI use case, right? So consistency standards, this
16:33
will help you with regulation. This will help you with your ROI
16:36
and sustainability. You know, Hira, I want to add
16:40
something to what you said. You know, all of those things
16:43
hold, and we've noticed that in the world of Gen AI, there are a
16:47
whole new set of interesting things that Gen AI exposes us to
16:51
on the data dimension that we have to take care of.
16:54
You know, these include sensitivity.
16:58
This is derivative risks with generative AI.
17:00
So what do I mean by that? In enterprise, you know, there
17:04
are a lot of documents that are out there that are not properly
17:07
protected. Now with Gen AI you have
17:11
powerful tools with LLMS and vector databases to improve your
17:14
search and to summarise all of that information.
17:16
So now suddenly what starts to happen is that all of this data
17:20
is now exposed and is searchable and findable.
17:24
And so if you have any technical debt in terms of documents that
17:29
are out there that are improperly protected, now
17:31
everybody out there can find it and that it starts to expose a
17:34
lot of your, you know, risk, increase the the risk of
17:38
exposing sensitive data. And and so there is new sets of
17:42
things that come up every time when we advance the technology
17:45
that we are noticing that we have to take care of this.
17:47
So then we go actually apply generative AI technology itself
17:53
to now classify sensitive documents and to automatically
17:57
change the permissions on all of those enterprise documents that
18:00
are improperly protected first before you can, you know, ingest
18:04
and make all of this data searchable and findable, right.
18:07
So that's, there is another new kind of risk that's coming up.
18:10
And also in order to establish trust associated with, you know,
18:13
the responses and the, the, the, the, the suggestions that
18:16
generative AI provides, you now have to provide citations,
18:19
references and all of those things better and pointers back
18:23
to the data with right kind of snippets and chunks.
18:26
So I, I feel that, you know, we, there is a lot of stuff that's
18:30
happening in data. All of those things are
18:32
absolutely right on, but there is a whole new set of risks and
18:37
interesting problems that we have to solve in the world of
18:39
generative AI as it pertains to data.
18:42
And we are only scratching the surface in finding those things
18:44
out. And all these data platforms
18:46
that are out in the industry will now start to also build
18:49
some of these capabilities around finding sensitive data,
18:51
classifying sensitive data, and, you know, having the the right
18:55
kind of controls, guard rails for controlling the derivative
18:59
risks associated with LLMS and generative AI.
19:02
That's so true, right? I think knowing data, where the
19:05
data is coming from, who is generating data and where data
19:08
sits, that's that's the crucial steps before you go into ML OPS
19:13
or LLM OPS with respect to Gen AII.
19:16
Think maybe that's where I think we need to spend more time
19:19
talking about, you know, if you are talking about LLM OPS, how
19:24
do you know where to rag versus, you know, maybe fine tune or
19:27
bring in some kind of like in house, bring your own model kind
19:31
of the view, right? And then you can do all kinds of
19:34
evaluation to get ready for the production deployment.
19:38
And also traditional ML OPS have similar kind of the viewfind,
19:41
which is more like training of the model, right?
19:44
Where do you see the gap and the challenge that today we need to
19:47
better understand on those fronts?
19:51
That's an interesting concept when talking about LLM OPS.
19:54
It's very different than ML OPS and definitely these are my
19:57
personal views. If you're cohort, I'd love to
19:59
get your feedback. The concept of CICD has changed.
20:04
There is nothing called a CI. There's nothing called a CD in
20:06
case of LLM OPS. Even if you're talking about
20:09
model retraining, there is no concept of model retraining.
20:11
Either we are going for a rack based implementation or we are
20:14
going for a fine tuning the whole concept that OK, hey you
20:17
will be getting that particular data training that model on then
20:20
you will be getting those metrics comparing it model
20:22
management and then talking about model versioning.
20:25
That particular concept has been changed.
20:27
So the way that we are envisioning the LLM OPS, the
20:29
whole system has been changed by itself.
20:32
So if we're talking about the kind of a system that are
20:34
running in production in today's world that's dependent on data.
20:39
I just want to quote one thing. One recent study published by
20:43
Stanford Hai team and they did some analysis on the legal
20:46
reports and they say that, OK, hey, some of the results that
20:49
they were getting from the LLM systems, they hallucinity more
20:53
on the legal documents and the legal data as compared to the
20:57
retail data or the marketing data.
20:59
The question is obvious because these kind of a system, that
21:01
kind of a data is not available in open source domain, you will
21:05
start getting that particular feedback.
21:07
The question is who will be providing that feedback in that
21:10
case, how the CD will look like and how the CI will be
21:12
integrated back to the system. So if we're talking about those
21:16
basic concepts, 3 basic concepts like CICD and CD, those are
21:20
getting changed with LLM coming into that particular picture.
21:24
And I really go hard with that point that I think Steven
21:28
mentioned is about the data. Definitely data is a mode, but
21:31
at what point of a time until what section it will be
21:34
providing you that kind of a competitive advantage.
21:36
That's not very clear the way that models are learning.
21:39
Previously some systems used to be calling us that, OK, this is
21:43
a differentiating factor now. They're not a differentiating
21:45
factor now. You need more data now.
21:47
You need more kind of a different nuance into that
21:50
particular data so that things are changing and the whole
21:52
system designed for LLM OPS are changing.
21:55
I think at some point, let's say you are ready to deploy in a
21:59
model or integrate the model output to some applications,
22:02
right. There's a process involved.
22:04
I think the from technology standpoint, CICD integration
22:07
deployment, I know Lane out there, but I think in most of
22:11
the enterprises whether you are operating within the regulatory
22:15
heavy spectrum versus non. I think maybe Steve, what's your
22:21
experience of deploying the model and the productions,
22:25
right? What are the processes involved
22:26
in that to to operate within the risk frameworks?
22:29
Maybe you might be following the model risk management functions
22:32
or if it is kind of bringing the vendor solution in house, there
22:35
is a toll gets involved. Do you, do you, do you, do you
22:38
have experience of like you know we're following those as Rick's
22:41
framework and what is out there? You know, I'll come back the
22:44
same question to Rama as well to understand a little bit.
22:46
I think like we take it back to the use case specific situation,
22:50
like what are the implications of that use case when things go
22:53
wrong. We always take that approach to
22:55
understand like there's definitely a workflow that we
22:57
followed to build the use case and to understand its
23:00
implications and the ROI there. But then what are the
23:03
implications of of when it gets it wrong, basically?
23:06
So in those types of cases, false positives, false
23:09
negatives, all these, all these items propagate, right?
23:13
So looking at those implications internally is really, really
23:15
important. Now, normalizations of practises
23:18
and continuously monitoring these use cases as they go into
23:21
production is critically important because what we often
23:24
find is like a great release only to find in a week or two or
23:27
a month, you know, it's no longer reforming as the team
23:30
expected it and they're already on to the next process.
23:32
So building automation and tooling around validating that
23:36
it's doing what you think it's supposed to be doing.
23:38
And usually what we found too is that having the group who
23:41
created the model create the criteria for the evaluation of
23:44
the model is a bad idea because often the criteria is like kind
23:48
of self absorbed. They're trying to showcase
23:50
something that it does really well.
23:51
You need an outside group to audit and invalidate and justify
23:55
kind of the metrics that you're tracking.
23:56
So continuously monitoring these releases to validate they are
24:00
doing what you're doing, what you're expecting those releases
24:02
to be doing. And then I think the other one
24:04
would be like that I mentioned in the opening regards creating
24:07
consistent platforms of releases, right?
24:09
So if you do a consistent platform, like the same process
24:13
and the same platform for your releases, then if you do need to
24:16
augment something in that platform, then everything
24:19
benefits from it. If you do it in a silo and you
24:21
do it in an individual way, that one project benefits from that
24:25
know how, why don't you try to bring that across the
24:27
organisation? So the whole organisation, you
24:30
know, benefits from that. So I think consistency in the in
24:32
the platform approach that you're releasing on really
24:35
understanding the the implications of of the use case
24:38
in practise and then potentially outside views, an audit
24:42
committee, red teaming, things like that to validate and to
24:45
justify how the model is performing.
24:48
So, so let me ask this question a little bit slightly to, you
24:52
know, am I right? So from the timeline
24:53
perspective, how long it does take to take the models, maybe
24:59
like a traditional predictive models or generative AI kind of
25:02
the models to production In your case, what's the kind of
25:05
duration? Does it take like 3 months, six
25:08
months or maybe weeks? What's the kind of the duration
25:10
view from like a non production environment to deploy in
25:14
production? So it depends, it depends on
25:18
what what you're actually building and what kind of a
25:21
model that is. Now, if it's a but for every
25:24
model, there are a set of processes that we have to
25:26
follow. Testing is the one that takes
25:28
the most amount of time. It's easy to build, especially
25:31
with Gen AI, easy to build proofs of concepts and
25:33
prototypes and get super excited about deployments.
25:37
But the the road from Pocs to pilots to actual production is a
25:42
long one, especially in case of Gen AI.
25:45
That's because you know, in the case of traditional models, at
25:49
least you can have, you know what is false positive, what's
25:54
you know, true positive, false negatives.
25:56
You can, you can have ground truth data sets and you can
25:58
measure the, the accuracy of the models.
26:02
And so you kind of know if you have a threshold, say I want the
26:05
accuracy to be at least 8590% for this use case and you have a
26:09
good representative data set, you can test it against that.
26:13
And if you pass the bar, you can put the model in production.
26:16
In the case of Gen AI, let's take chatbot for example, that
26:20
you're building. The responses are natural
26:23
language based, right? Sometimes they're very
26:25
subjective. And so creating ground truth and
26:29
evaluating that ground truth is, is, is not an automated process.
26:34
I mean, yes, you can use LLM itself as a judge or LLM as jury
26:38
and measure, you know, how good the response is.
26:41
But a lot of times humans have to really evaluate the answer
26:44
and that because it's a natural language answer if it's creating
26:47
a summary and that makes it kind of human in the loop, kind of a
26:50
testing process, which which increases the time it takes to
26:54
take it to production. But overall, the principles are
26:57
the same whether it's classical AI versus generative AI.
27:00
You know, you have to start with a good representative test data
27:03
set with good ground truth and you are always measuring against
27:07
that and to see how far you got and have some threshold to see
27:12
if you have reached that threshold and if the user users
27:15
who are actually going to use it have tested it and they're
27:18
comfortable with it. Yeah.
27:20
I mean, in our case it really varied.
27:22
In the case of some of the chat bots that we deployed, it took
27:27
maybe 6/6 to 8 weeks to build the first version.
27:32
But you know, the road to deployment was long because we,
27:39
our bot was answering questions that are sensitive, not because
27:42
of anything that's wrong that it's doing, but because of some
27:45
of our sensitive documents had improper access controls.
27:49
And so we had to stop that bot from actually getting deployed,
27:54
put it still in early access testing and 1st go fix the
27:56
access control problems and documents.
27:58
And imagine that is a long process, you know, in a company
28:02
that may have, you know, hundreds and thousands of
28:05
documents, fixing the access control permissions could be a
28:08
long road, right? And you have to use different
28:10
techniques to do that. So some of those bots haven't
28:13
even made to production, but some of the more bots that are
28:16
operating on open data that are more public, you know, we have
28:21
them in production from six weeks, it went to production in
28:23
three months. And so it really kind of depends
28:26
on the domain, the use case and such, but the principles are the
28:29
same. Test, test, test.
28:30
Evaluate accuracy thresholds. And that's so fascinating,
28:33
right? I think in enterprise landscape
28:35
that's so true across, you know, regulatory heavy function
28:39
because of the rigour and the testing and the risk framework
28:42
that you need to operate on any kind of model function that you
28:46
need to put in the production because of the implications,
28:49
right? And there is responsible AI and
28:52
AI government basically come into play as well to support
28:55
your kind of deployment schema. So Nitin, I'll come back to you
29:00
on in terms of like now reset the production deployment,
29:03
right? I think there is a beyond steps
29:05
involved in that, which is kind of like monitoring, which is
29:09
what we talk about. And do we monitor like
29:11
performance level of the models or we look at from like
29:15
governance angle, right? Which is kind of like in GNA
29:18
space, we heavily talk about the responsible AI component, which
29:23
is like, you know, there is a risk AI risk management
29:27
functions and things like the security, privacy and the
29:29
compliance. And then there is whole piece of
29:32
the, you know, standards of ethics, right, the bias check,
29:35
biasness check. So where do you see that from
29:37
the monitoring standpoint we need to do at least the minimum
29:41
of these components view as you deploy the models in production
29:45
and from the day one you need to start supporting and monitoring
29:48
those kind of the insights and the metrics around there.
29:52
Thank you so much for asking that question.
29:54
So I'm totally agreeing with what Rama mentioned about data.
29:58
So we are working with some human labelers and they were
30:01
saying, is that OK, hey, you need to provide the feedback to
30:04
a model. And we asked them a question
30:06
that, OK, hey, you give thumbs down to this thing and you give
30:09
thumbs up to this particular response.
30:11
What's the reason they say I like it?
30:14
OK, be more specific. What do you like it?
30:17
So this is more relatable. The question is how will you
30:20
make that subjective decision and quantify it so that the
30:24
model will be able to understand the relativity and as well as
30:27
the likeness for that particular output.
30:30
And that's the biggest confusion, that's the biggest
30:32
question for the responsible AI. Some of the questions, and
30:35
you're absolutely right, on the governance side, we are tagging
30:39
some flags, like for an example, toxicity, we're talking about
30:42
unfriendliness, we're talking about some sort of an adult
30:45
content. There are some quantification of
30:47
the factors currently available, but it's very hard to measure
30:51
that thing. And that's really making it very
30:54
hard to govern these kind of a systems.
30:56
So if we talk about any of the models, we're talking about
30:59
Google Gemini, talking about ChatGPT, talking about 5,
31:02
talking about Sydney, any of those kind of a systems, none of
31:06
these systems, we can say that, OK, this is foolproof.
31:09
Each and every of these systems failed to a certain extent.
31:14
The question is which one got the more marketing and more
31:16
visibility. That's another point.
31:19
But are we saying is that OK, these kind of an organisation
31:21
have not built a robust framework or they have not
31:24
tested their system very well before launching?
31:26
No, it must have gone through all those kind of a processes.
31:30
But the challenge is the base technology is so subjective,
31:34
it's so hard to measure and governing those things.
31:37
So I'm very hopeful that in future, Stronger Robbers and
31:42
quantified responsible AI metrics will be coming to the
31:45
picture. We will be able to govern these
31:47
kind of a models with better feedback and we will be able to
31:50
generate those responses. But I feel it's not just the
31:53
onus on the model creator, it's on the user as well if you want
31:57
to make this system or if you want to make this AIML model a
32:01
success or not. In my experience, again in my
32:04
limited experience, I might be wrong.
32:06
If I give a model to an individual and say that if I
32:09
test it out, that person will test that model for all the use
32:13
cases that model was not intended to except everything
32:17
inside the world. So the focus is more about, hey,
32:20
I got that model. How can I make sure that this
32:23
model is failing on those things and testing the limits and
32:27
dimensions of those models other than making it work for certain
32:30
use cases. So at this moment, I don't say
32:33
that, OK, there is a one approach, There is one framework
32:36
that's already available. There are multiple things are
32:39
available. Nexus flow is there.
32:41
We're talking about like land graph is coming up with some
32:44
sort of a metrics around it. Colena is like building up some
32:47
sort of a system around it. The multiple system exist, but I
32:51
can't see a system where I can say that, OK, this is the Holy
32:54
Grail or the golden truth that you can use anywhere.
32:58
And if it's passing this particular test, you are good to
33:00
go. So I'm very hopeful in future.
33:02
Just a really quick follow up and I think we're over here is
33:05
that even if we could do everything and create that Holy
33:07
Grail of, of a model, etcetera today, tomorrow it's going to be
33:11
different. So the criteria, the eval, the
33:13
rag, the information going into, it's going to change tomorrow.
33:16
So just expect that. So if you're building a model,
33:18
look at the frequency of the updates to the models, look at
33:20
that, you know, feed that into your ROI to really understand
33:23
the true cost that creates a sustainability outlook for the
33:26
model and the AI use case in practise.
33:28
But Sir, I think you're trying to come on stage.
33:31
I was just trying to say this is a great conversation.
33:33
I would love to continue. If we can wrap up with just
33:36
about 30 more seconds, we have one more speaker.
33:37
Yeah. I'll ask last questions.
33:40
I think this would be very interesting to learn from our
33:42
panellists is like what's the next big trends happening in the
33:46
AI, right? Are we going from like classic
33:49
RPA to, you know, Co pilot to agenting workflow or something
33:53
else? Just maybe a couple of, you
33:56
know, thoughts Rama, starting from you and then we'll wrap up.
33:59
Yeah, I'll say, you know, when Jen AI came out, everybody
34:02
thought, oh, let's go build things with LLMS. Then it turned
34:06
out it's not just LLMS. If you're building chat bots and
34:09
such, retrieval is really critical.
34:12
Getting the retrieval accuracy is, is the most important thing.
34:15
Now it's not just retrieval and LLMS, it's agentic workflow.
34:18
So it's really the good old software engineering where you
34:22
have good agents, good orchestration that's bringing
34:25
everything together with good retrieval and good LLMS. So the
34:29
whole thing has to come together for it to work.
34:31
That's my experience shifting from LLMS to retrieval to
34:36
agentic workflows. I see LLMS for January as an
34:41
orchestrator, as a tool for automation, whether it's agents,
34:45
chat bots, anything in such a way that anybody can use AI.
34:49
It's not just touted for some data scientists or AI engineers,
34:53
just making things simple. Just really quickly, yeah, I
34:56
think the agentic view is, is really an important one.
34:59
Those agents are able to do real tasks within the companies.
35:02
And then I think just the general access and and
35:05
understanding of general data, I think we are able to access data
35:08
like we've never been able to access data before both
35:11
internally and externally. And this is just going to get
35:13
better over the next couple of years.
35:15
And ultimately as a society, we're going to get better.
35:17
I just hope we don't lose our original thought.
35:20
OK. So that's a good.
35:22
Thing, Thank you. I think we're on the top of the
35:24
hour. Thank you all the panels for
35:26
your incredible, actionable insights to our audience and
35:28
everyone for joining us on this session.
35:31
Thank you very much. Yeah.
35:32
Thank you.