Latest / Agentic Conversations (formally mlops.community) / The Future of AI Operations: Insights from PwC AI Managed Services

Transcript

The Future of AI Operations: Insights from PwC AI Managed Services · Agentic Conversations (formally mlops.community)

0:00
We have a very special episode. I have the opportunity to hang
0:04
out here in the PwC offices in San Francisco for my trip, and
0:09
I'm here with Ronnie. We're going to talk all about
0:11
what she's been up to and how you're looking at ML OPS, AI
0:16
OPS, agents, everything of the sort.
0:28
We should probably start with a little bit of who you are.
0:31
Sure. Thank you Dimitrios, and excited
0:33
to be talking to you. I've watched a number of
0:36
episodes of your podcasts and very excited to be in.
0:40
By your own will, I imagine you were forced.
0:43
Absolutely. Not, I decided to torture.
0:45
You it's 100% of my own free will, but and so lovely to be
0:50
here talking to you. So a little bit about myself.
0:52
I have about 3 decades of experience in the consulting
0:57
industry and over the last 20 years I've been with PwC.
1:01
Today I'm a partner with PwC and I lead our data analytics and AI
1:06
managed services space. So with PWCI, spent the majority
1:11
of my career in healthcare in payers and providers.
1:15
And then over the past five years, I decided to make a
1:18
shift, you know, back into technology.
1:21
And this was a very, it was a very deliberate shift where I
1:28
wanted to move it to the managed services space.
1:30
And the way PwC approaches managed services is we want to
1:35
lead with automation and AI. So that excited me because
1:39
managed services or keep the lights on is a space that
1:42
consumes over a large majority of a CIOs time and resources and
1:48
all of that and having the opportunity to shape that to be
1:51
more efficient excited me. For the uninitiated, what are
1:56
managed services? Managed services is the space
2:00
where we take over support of anything that moves into
2:03
production. So it's, some people call it
2:06
keep the lights on. It is as soon as somebody works
2:10
on a project, it's live with consumers.
2:12
All of the support related to that software asset, whether
2:17
that is the hardware, the software, the experience, right,
2:21
all of that falls into the bannered services space.
2:25
And the area that my team focuses on is data analytics and
2:30
AI. And I think AI managed services
2:33
in itself is one of the coolest things that is happening today.
2:37
Yeah. So you have your work cut out
2:39
for you, Yes, You think things in production and making sure
2:43
that they're going off without a hitch.
2:46
I imagine there's a lot there. Definitely, definitely it's it's
2:50
an exciting time to be in the space A because I think AI is an
2:55
area that has been growing exponentially and at least in my
2:59
lifetime, it is the area that's I've seen grow the fastest.
3:04
And so not only our new technologies and tools being
3:08
introduced, also the pace at which innovation is happening,
3:11
which is the applicability of AI, that is also something that
3:15
I've never before seen, right? The, the pace at which things
3:18
are growing. I would say 12 months ago, I was
3:24
in conversations with clients where they were talking about
3:27
investing in AI and experimenting, piloting in AI.
3:31
Today, Fast forward even in the last six months, I would say I'm
3:35
seeing clients who have implemented AI and are now
3:38
looking to thinking about ROI, how to scale it, how to sustain
3:43
it, which brings brings in right, the managed services
3:46
component of scaling, sustaining and and so on and so forth.
3:50
But yeah, it's so funny that you mentioned that because I think
3:53
we went through the exploration phase.
3:56
Now it is in the phase of OK, So what can this actually do?
4:02
And how can this, how can I prove that it is better for my
4:09
company if I'm using AI? Yeah, I'm sure you've, you've
4:12
seen the recent paper that came out that was saying people using
4:17
AI think that they're doing things faster, but in reality
4:21
they're doing it like 20% slower.
4:23
But when they're asked, hey, did AI help you?
4:27
It's like, Oh yeah, it was way better.
4:29
And we all instinctively are probably a little bit like that
4:35
where if we're using AI in some way, shape or form, we feel
4:39
like, oh, yeah, this is better. But then if you were to compare
4:45
that with us not using it, I often wonder and I ask myself
4:48
sometimes like, yeah, is it, is it better or am I just fooling
4:51
myself? And a friend of mine told me,
4:53
yeah, that dude, look, the experience of chatting back and
4:58
forth and getting being able to say like, OK, now go off and do
5:01
that and, you know, write me that thing that we've been
5:04
talking about is so much nicer than having to really, like,
5:08
squeeze every one of those words out of your mind and all that.
5:12
But sorry, I digress. I went on a bit of a tangent.
5:14
It's all good. Yeah, I think there is a the
5:17
it's probably a two-part conversation, right?
5:19
There's a personal growth, a personal productivity that which
5:23
which is what you're talking about.
5:24
And then there is the enterprise growth.
5:26
Enterprise productivity, right, on the other hand too.
5:29
And you've got to like take both of these together.
5:33
And one of the recent studies that I saw said that, you know,
5:37
if you're not spending, say 4 to 8 hours a week or like X percent
5:42
of your time learning and studying and upskilling, then
5:46
you're not going to be AI enabled, right?
5:49
And I think that is true for personal growth, but also for
5:52
organizations who are looking to upskill their employees.
5:56
There is a huge investment in terms of training and, you know,
6:01
giving people access to the tools and technology so that
6:04
they know how to be productive, right, with all of these tools.
6:07
And I think there is a whole change management piece that
6:11
goes along with AI enablement, which if you undermine that
6:15
effort, then it's never going to be successful.
6:17
Yeah, it's a cultural thing. It's a cultural thing, 100%,
6:21
yeah. So ground us in the current
6:23
state of what you've been seeing out there with IT operations,
6:27
how you see Mlops. Maybe let me start with what is
6:31
the current state of organizations, right?
6:34
There's tons of data out there. You know, most organizations I
6:38
know have alerting systems. So there's lots and lots and
6:41
lots of alerts that if a human being has to sift through all of
6:45
it, a lot of it is noise. And I'll give you maybe a couple
6:48
of examples. And so incident detection today
6:53
is, you know, it is. So if you get 100 alerts, which
6:58
one is truly an incident, which one is not, is something that if
7:01
you have to go through 100, then you're losing time on it.
7:05
That is if you train a model to look for these, you know, this
7:07
criteria, it's going to pop up, hey, here are the 10% of alerts
7:11
which we think you need to look at.
7:12
And then triaging becomes a lot more easier.
7:15
You're bringing down the meantime to resolution just by
7:19
doing that. Another example is, you know,
7:22
when it comes to infrastructure, right, like the say CPU
7:27
utilization, resource utilization sometimes tends to
7:30
be cyclical. And if you are actually
7:33
monitoring for certain events, you know that the usage is going
7:36
to spike six hours from now or 20 minutes from now.
7:40
And if you had a system that was looking for these things, then
7:44
you would know, hey, I need to add more memory to the server,
7:47
otherwise you know this is going to crash in like 2 hours.
7:50
So this is what I would call, you know, proactive monitoring
7:54
where you're preventing an incident from happening because
7:58
of how you're effectively using your data.
8:01
So those are examples of how I think ML OPS and AI OPS can be
8:05
combined to create a very compelling case for IT
8:08
operations. Yeah, the predictive monitoring
8:12
is incredible. I really like that.
8:14
And then also alert fatigue is real.
8:17
It is real. What about agents and where are
8:20
you seeing them? What have you been doing with
8:22
agents? Because that's like the buzzword
8:24
of this the last year, year and a half.
8:27
Yeah, I think the the concept of agents is very real and we are
8:32
starting to see, I think some organizations adopting agents
8:36
first and foremost. There's a lot of process
8:38
standardization work that needs to be done.
8:41
If everyone is following a different process, then it's
8:44
very hard to put in an agent to replace or to do a process,
8:48
right? So this process standardization,
8:50
the second piece is data quality, right?
8:53
I do not know of a single organization that says that I
8:57
have perfect data. The quality is great.
8:59
So, you know, there's, and, and so for these two things, I, I do
9:04
see a lot of data modernization, app modernization, type of
9:08
initiators that organizations are taking on.
9:11
And so once you have this foundation set, then it becomes
9:14
a lot easier to go and say, well, these are my standardized
9:18
processes and these processes get used 75% of the time.
9:23
So I would get the most bang for my buck if I were to identify
9:28
these processes, right? And and then and, and I am
9:33
oversimplifying it as I'm kind of walking through this, right,
9:36
But the other piece of it is I often get asked, should we start
9:41
with one agent, one process and then look at how that goes,
9:46
experiment with it, build a pilot and then scale it or do we
9:50
do multiple right? And so I've, we've done a lot of
9:55
reading on this and, and thoughts on this, right?
9:57
And what we've seen work is I think centralizing the
10:01
management of agents is important because you do not
10:04
want different people just using different technology within the
10:07
same organization to build their own agents.
10:09
So there's some kind of centralization that's required
10:13
that also then advocates for responsible AI and all those
10:17
ethical considerations. But then it is OK to start
10:21
parallel development of agents if there are there are processes
10:26
that are being used 75% of the time in four different
10:29
departments start writing up like 4 agents.
10:33
And today with all the IDs and the low code no code solutions,
10:37
it is so easy to spin up an agent.
10:39
So I think the most the majority of the time needs to be spent
10:43
designing the agent, then a little bit of time writing the
10:48
agent and then the rest of the time and evaluating.
10:51
So the evals and the, you know, and, and measuring the impact of
10:56
the agent, looking for hallucination, looking for
10:58
biases. That's where then the rest of
11:00
the time gets spent, right? So this is how like we see a lot
11:04
of clients like embracing Asian take close.
11:07
And I think this is still pretty nascent, right?
11:10
There's not a lot of, I think organizations who have agents,
11:14
too many agents in production, right?
11:17
I think we are still in that, like I'm building an agent, I'm
11:21
evaluating it. Yeah, Yeah.
11:23
It reminds me a little bit of the MLM stays early MLM stays
11:29
when I started the community in 2020 where a lot of folks were
11:33
trying to get the models into production still.
11:36
And it's just really hard to make something a that you can
11:42
clearly attribute ROI to and then B, something that actually
11:48
like works. That's.
11:49
Right. Yeah.
11:50
And my buddy Zach was just telling me this morning, one
11:54
thing they see is if folks can get an agent on these
11:58
standardized processes into production quickly, then they
12:02
start to see where the long tail of their data or their knowledge
12:09
base is missing. And so because you run into
12:13
problems constantly and then you have to amplify or you have to
12:18
really enrich your data. And it's almost like a a way for
12:24
you to recognize where you're faulty.
12:28
Yeah, I think it is interesting how we build upon these things
12:33
and you know, yes, you get your agent into production and that's
12:37
where I think you know, so that's what that's what makes
12:39
managed services so exciting for me because these things are now
12:43
in production and then we are tuning, optimizing, training it.
12:48
So managed services for AI takes on a different meaning than your
12:53
traditional managed services, right.
12:55
Traditional managed services is all about break, fix, incident
12:58
management, make sure the system is up and running.
13:01
Here we are not only doing that, but we're also making sure that
13:06
we are delivering on the right outcomes.
13:08
Yeah, that that it was intended to deliver to begin with.
13:13
And that means that we are, you know, constantly measuring.
13:17
And so even the skill set of the people that we have in AI
13:21
managed services is very different from your traditional
13:24
managed services folks. We are hiring ML OPS engineers,
13:27
we are hiring data scientists, we are hiring, you know, more
13:31
advanced analytic folks and who can tune and optimize the models
13:36
to kind of reduce the bias, reduce the hallucinations and so
13:40
on and so forth. And so on that note, right, We
13:43
also have a framework called Agent OS that PwC has developed
13:48
for our clients. So we can go and it's all based
13:52
on, you know, Python based frameworks and things like that.
13:55
So it's very interoperable, works with all the hyperscalers,
13:58
has APIs to connect into any ecosystem.
14:02
And it is an agent framework. So you can build 1, you can.
14:06
It's got an agent orchestration layer inside it.
14:09
And I know there's a lot of them in the market.
14:11
But when we go to clients, we do go with the framework.
14:13
And you know, I think that's one of the things we tell the
14:16
clients that you don't have to build from scratch.
14:19
There's so much that's already out there you should leverage
14:23
and then build on top of it. So then the focus becomes the
14:27
client specific business outcomes versus trying to figure
14:30
out the technology that you need to make this happen.
14:34
Yeah, that makes sense because as you know from last night,
14:38
there's a new startup that comes every day.
14:41
Every minute, I think. In this space, it's it's a hot
14:44
space and again, there's a lot of unsolved problems.
14:48
So that makes it ripe for these startups to come up and try and
14:53
find the best way in the most optimal solutions.
14:56
And you said something interesting there where it's
15:00
different from the traditional managed services because there's
15:03
not that break fix and it's, it feels like it is very like
15:11
random, but not random in a way. I guess that's not the right
15:14
word I'm looking for. I don't know if if that makes
15:16
sense. Like it's you have to kind of
15:19
just let it let it do its thing or break fix is very clear
15:24
because it's doing this thing and it's not working.
15:27
There's so many other things that could be going wrong with
15:29
an agent that it could be working in that it actually
15:34
executes, but it's not working because there's all this other
15:38
stuff that it's doing wrong. Yeah, You need a human in the
15:41
loop until it is proven that this model can work without a
15:45
human in the loop. And there are probably very few
15:48
use cases today where someone would be comfortable making
15:52
that, you know that judgement, judgement right then.
15:56
And it's interesting, I think we've when you talk about humans
16:00
and agents, there are so many, there's jargons that have come
16:06
out, for lack of a better word, like there's human in the loop,
16:09
that's human on the loop, and there's human out of the loop.
16:13
Oh yeah. It is, you know, how many more
16:16
ways can we use those words together, right?
16:18
The, the human in the loop is where you have to approve a
16:22
workflow, right? Approve the outputs that the
16:24
agents are coming up with. Human on the loop is where
16:27
you're approving only the exceptions.
16:29
So you're comfortable that the regular workflow is something
16:33
that the agent is able to take care of.
16:34
And outside the loop is outside the loop, right?
16:36
So and I find that very interesting because in highly
16:39
regulated industries, I think they kind of move or lean
16:43
towards the human in the loop model where they are approving
16:46
everything, not just the exceptions and the norms.
16:50
Yeah, I find that fascinating that especially because the like
16:57
going back to the idea of we create these systems so that
17:02
they can hopefully give us incredible lift, right.
17:07
But at the end of the day, if we're still in the loop, are is
17:10
it? Is it giving you the lift?
17:12
Is it giving you the lift? And that's probably the, the big
17:14
question that I imagine a lot of these folks are grappling with
17:17
like how much better are we because we're using this, you
17:22
know, it's there like, and, and so you can't say I don't want to
17:26
do it because, you know, there are companies that are finding
17:29
success with it. And and I draw this parallel
17:32
back to like the traditional predictive ML and how you have
17:38
gigantic companies that are optimizing their fraud detection
17:44
by point O 1% and saving millions of dollars.
17:51
And so other companies see that and they think, well, yeah, we
17:55
have to do machine learning because look at that.
17:59
But then there's all these nuances of like, yeah, are you
18:02
at that scale though? Are you?
18:04
Is that the right use case for you?
18:06
And these questions come into play and maybe you can start
18:11
with something a little bit more simple just to see and like, put
18:16
your foot in the water, test it out, see if it works and and
18:20
actually see, start having the conversations of what we need to
18:23
do to get that into production. Yeah.
18:25
Yeah. And yeah, 100%.
18:27
And, and you, I, I also think that most organizations have
18:32
started in their back office, you know, playing around with
18:35
agents and AI and all of that. And that has given them a lot of
18:39
comfort because it's a more forgiving place to really start,
18:43
right? And then we are now starting to
18:45
see them get into their own core products and services.
18:50
And I think that's the next wave.
18:51
And this is, this is going to get very interesting because now
18:55
you have to deliver on business outcomes.
18:57
And that means there's going to be business people who are
19:01
actually going to take advantage of that natural language
19:03
processing. And I think This is why vibe
19:06
coding is like picking up so much, right?
19:08
Yeah. And that's now the latest that
19:11
we hear about every, everywhere because they've made it.
19:14
They, as in the technology companies have made it so easy
19:18
for just about anyone to write a piece of code because you can
19:21
say it in English and then you can get code out of it, right?
19:24
That's that's really the whole concept behind it.
19:27
And have you seen good ways or ways that you can get alignment
19:35
from the two sides of the houses?
19:38
Like, and I know it's normally not so black and white where
19:42
it's like the tech side and the business side, but maybe where
19:46
folks are like you're saying, business folks are coming into
19:48
this and they're starting to have their say.
19:51
And they also have needs on how it's being built.
19:55
And they, they play with ChatGPT.
19:57
So they think like, hey, AI should do this or it should be
20:00
like that. Yeah, I, I think it's a, it's,
20:04
it's a really good question and I have seen pockets of this,
20:08
right. So as I, I think mentioned, my
20:11
background is in healthcare and I'll take a healthcare a couple
20:14
of healthcare examples, right. So 1 is a function, you know,
20:20
when you go to a doctor you get approved for the insurance.
20:23
And so there is something called a pre authorization that's done
20:26
for services. The pre auth process, right, is
20:29
pretty manually intensive, like it involves calling the
20:33
insurance company to make sure you're covered and so on and so
20:36
forth. That is a process that I think
20:38
has been identified or AI enabled and it you know, and and
20:44
so that works, but they still need to make sure that if
20:49
something gets denied, you know, by AI, they have a human in the
20:53
loop to make sure that you know, this denial that's appropriate
20:57
and and so on and so forth. So that is an area where you
21:00
know, this cannot be done by an engineer alone.
21:03
This has to you know, you have to bring in people from and I
21:07
will call this the business right to come in and say, yes,
21:10
you know, these are the right policies.
21:12
This is the right reason for a denial and so on and so forth.
21:16
And when you build it keeping the business person's
21:21
requirements in mind and in the forefront, I think you get it
21:24
right. But if an, if you try to do it
21:27
from the back end with an engineer trying to come up with
21:29
a solution based on because you see, as an engineer, you see the
21:33
data flows, you see there is a call from this system, there's A
21:38
to this system. And this comes back with, you
21:40
know, this sort of an answer. This is the data flow diagram.
21:43
That's how I think an engineer's mind works.
21:45
Whereas a business person looks at it saying, well, a person
21:49
comes in with a fever and you're making, you know, asking them if
21:52
you can get surgery for it. Like you're going to deny this
21:55
is exaggerating? Again, this goes back to like
21:57
the latent failures or the silent failures where it's it's
22:00
failing but in ways that the engineer isn't going to see.
22:03
It exactly so I think that is where you know you've got to
22:06
take the business input to create these agents and that's
22:09
where it becomes more successful.
22:11
And so the the the extension to that is, you know, Epic Systems
22:15
is one of the biggest like electronic medical record
22:18
systems, right That's used by a lot of with the big healthcare
22:23
providers. They have recently, the past
22:25
couple of months, they've introduced AI capabilities into
22:28
their software and I think organizations are still adopting
22:32
and they're check, you know, they're still in that
22:34
experimenting piloting mode where they are trying to figure
22:38
out if this is something that they want to introduce.
22:40
But an example is drafting mails to patients, right?
22:44
That is a functionality that they have introduced in it.
22:47
And today people do spend a lot of time drafting those emails.
22:53
And what the system is doing is giving them a draft.
22:56
And then yes, I think you have to change most of them for tone
23:04
for things like that. But it's a start that, you know,
23:08
actually saves a lot of time. I, I think honestly, that's a
23:11
really good example of, you know, usage of AI.
23:15
Yeah. And well, especially if you're
23:18
typing in, and this goes to a little bit of like a data
23:21
engineering type of thing because it, I imagine the
23:24
drafting of the e-mail isn't high.
23:26
Your appointment is this day. That's right.
23:28
Yeah, it's a lot of random variables that need to be piped
23:32
in. And so it's not as simple as
23:35
just having a template. That's right.
23:37
Yeah. Yeah.
23:38
And there's contextualization. Yeah, right.
23:40
There's tonality. There's so many things that have
23:43
to be looked into. Yeah, yeah, totally.
23:45
So when you're working with companies and and you get
23:48
alignment and then something's actually in production, one
23:51
thing that is always fun and as a friend of mine would always
23:57
joke like you can never go wrong If at the end of any talk that
24:02
you see, like at a meet up, you ask the question and say, yeah,
24:05
but how does it scale? And so whenever I think about
24:09
scale, I think about that exact phrase.
24:12
And so I'm going to ask you now, like, how do you scale?
24:15
Yeah, you know, I'm going to use that for your next meet ups,
24:18
but. I'm sure the presenter is going
24:20
to be happy because she was telling me and like, oh, there's
24:24
always one person that asked that.
24:26
How does it scale? And technically it's a good
24:30
question because yeah, you want to know, but also.
24:34
Yeah, right. See, I, I think of scaling in
24:38
two different ways. So one is, you know, today's in,
24:43
in today's world, the way you go through SDLC, for lack of a
24:48
better word, right? It's not waterfall, it's not
24:51
agile. But the way agents are being
24:53
developed is you spend some time thinking about what is the use
24:57
case or what is the process you want to identify and then or you
25:02
know, you want to AI enable, right?
25:04
And then you think about, you spend a lot of time in design
25:07
and then once you have the design ready, the coding really
25:10
takes very little time. It's the least amount of time.
25:13
And then the evals and the measurement piece, right?
25:15
That takes like the next big chunk of time.
25:18
But once you move something into production, right, scaling is
25:22
not a problem. If process A is what was
25:26
automated or you know was, was changed, then at the same
25:31
process being used in multiple places in the same company,
25:34
that's an easy way to do it. So coding right, when you think
25:38
about scaling the coding piece is not a challenge because of
25:42
all the tools that is there. But the second piece is how do
25:45
you scale? If you got one process right,
25:48
how do you get the next 10 processes right?
25:51
That is what I think takes more time.
25:53
So this is where it goes back to having a good handle on your on
25:57
the processes within your company and standardizing a lot
26:00
of the processes. If the sorry, if the processes
26:04
are standardized, then I think scaling becomes a lot more
26:07
easier, right? And going back to like my
26:11
example of pre auth, a pre auth in a, in the case of a cold or a
26:18
cold or a fever, it's probably going to look very different
26:21
from a pre auth from an oncology department, right?
26:24
So the process is the same, but the business context is really
26:28
important. So I think those are
26:30
considerations right to bake in. But I think if you have a good
26:34
foundation of processes and if you have business weighing in,
26:38
the coding is not the challenge, it is the adoption.
26:41
And we've been back to culture, right?
26:43
The change management piece of it, I think that is where the
26:46
focus needs to be to scale it. And then the third piece I would
26:50
add is the measurement piece, right?
26:53
The evals are so important when it comes to scaling because you
26:56
don't want to scale something. That is, they're giving you
26:59
different outputs than what you want.
27:02
So as long as you're able to measure and correct and measure
27:05
and correct, and you have a fairly good, you know, model in
27:09
place, then it becomes a lot easier to replicate.
27:13
I look at evals and I had a friend say, man, we tried a lot
27:19
of different tools, but we realized that the tooling isn't
27:23
the bottleneck or the, the, what the tools do isn't where the
27:28
bottleneck is. What the bottleneck is is a lot
27:31
of this data labeling on, as you were saying earlier, when the
27:35
business comes and they look at the interactions and they say,
27:40
wait a minute, why are you giving them a hamburger when
27:42
they ask for pizza? That's not what they're asking
27:44
for. And so you have those data
27:48
labeling issues that take a lot of time because if you're
27:52
labeling it with humans and you're labeling it with humans
27:55
at the company and you have a lot of data, that can just be a
27:59
lot of work. And at the end of the day,
28:03
that's going to give you the best lift.
28:05
So he, he told me, you know, you can do these evals and you can
28:09
get this LLM as a judge or an LLM as a jury, all this fancy
28:13
stuff. But what we found works the best
28:16
is just getting people into an office, having a pizza party and
28:22
labeling data together. Well, that and also having a
28:25
good baseline to start with, right?
28:27
And that's was the standardization piece I was
28:30
talking about, right? If you know what this yielded
28:33
before it was identified, then you know what the increase in
28:39
productivity is, if it's working, all that good stuff,
28:41
right? Yes, that is.
28:43
And this is also where we talked a little bit about measuring the
28:47
ROI, right? And to me, measuring the ROI is
28:51
not just a straight, hey, I replaced 4 humans with four
28:55
agents. So now I'm going to take out the
28:57
cost of the humans. It's not that, right?
29:00
Because there is a cost to putting these agents in.
29:02
There is a human in the loop who is reviewing it for evals or
29:06
whatever, right? And so there's a cost associated
29:09
with that. There is a cost of retrieval,
29:12
storage and all of that in the back end, right, which you're
29:14
not accounting for, which was not there when it was just a
29:17
human doing the work. And you've got to like compute
29:20
all of this together to get the true ROI.
29:22
Yeah. Yeah, it reminds me of the build
29:25
versus buy question, what everybody thinks.
29:27
Yeah, I'm not going to buy it because.
29:29
It's too expensive and then. They're like, I'm just going to
29:32
go hire 4 engineers, yeah, to help us maintain it.
29:35
And you're like. Yeah, exactly.
29:37
I hope those engineers aren't based in San Francisco.
29:41
Let's talk for a minute about some of the sustainability
29:45
considerations that you've thought about.
29:47
Yeah. So I, I would bucket
29:49
sustainability into three different buckets and I think
29:51
some of it we already talked about a little bit.
29:54
The first one is environmental, right?
29:56
So that is the energy used, the carbon emissions, water
30:01
consumption, right. Are you picking the right sites
30:04
where recycled water is being used, you know that sort of
30:08
thing, hardware materials like e-waste, all that stuff, right.
30:12
So that's kind of the first chunk of environmental.
30:15
Then there is the economic cost that we talked about a little,
30:18
right? So that is the cost of building
30:21
the agent, cost of the human reviewing the agent, cost of the
30:25
storage, the retrieval. And when you don't need a large
30:31
model, are you running a smaller model, right?
30:34
Are you making those selections very deliberately, right.
30:36
And that's what I would put into the economic bucket.
30:40
And then there is a social bucket, right, which is hugely
30:44
important. And we talk about responsible AI
30:47
quite a bit. We talk about the ethics
30:48
associated with it. So that's good.
30:50
But there's also accessibility, localization, transparency,
30:56
explain ability of the models, right?
30:58
There's all those kind of things that I would add into the social
31:01
bucket. So to make something
31:03
sustainable. And I think this is what I was
31:06
saying at the beginning, I think we are just getting into that
31:09
phase where we are starting to think about what is
31:12
sustainability? How do you measure the ROI?
31:15
So far it's all been about I'm so excited about AI and the
31:18
possibilities of AI. Where do I start now?
31:21
People have started, they've deployed a few things and it's
31:24
like now how do I scale? But hey, wait a second, how do I
31:27
sustain this? So I think these are all going
31:30
to come in, you know, in into the picture a lot more, I think
31:36
over the next three to six months or yeah, I'm saying three
31:39
to six months. I would have said three to six
31:40
years if you had talked to me a decade back.
31:43
Yeah, yeah. Yeah.
31:45
So who knows? I do like how you are facing
31:49
this. You're kind of abstracting it
31:51
away from the weeds of the tech and really trying to think a bit
31:57
more big picture and say, look, if we want to sustain these
32:04
efforts, there's certain things that we're going to need to
32:08
invest in. Right.
32:10
And none of those things are like this new hot tech that or
32:15
framework that just came out. That's not the big picture type
32:19
of thing we need to look at. The big picture here is how are
32:24
we going to make sure that we can justify this?
32:28
Yeah, that's that's 100% it because we are going to get
32:31
there and you know, think about all the waves that we've been
32:34
through over the past couple of decades, right?
32:36
Whether it's the Internet, whether it is the cloud, right,
32:40
moving to the cloud, it all came back to like even you think
32:43
about the cloud, the CIO's came back to wait, I'm going to spend
32:48
a lot of money if I just keep storing all my data in the
32:52
cloud. Is there a more optimized way of
32:54
storing the data in the cloud, right?
32:56
It kind of and I think that's the next wave with AI as well.
33:00
Yeah, I've already heard folks talking about how they're
33:05
starting to look at the cost and they're starting to think if we
33:11
do deploy this to production, as soon as you start thinking
33:14
enterprise scale for that feature, now you're hurting
33:20
every time the user says please. Yeah, I think efficiency and
33:25
perfection has a price. And I think everybody or every
33:29
company has to decide what is that right trade off, right,
33:34
where you want to spend so much money for just so much
33:37
perfection. And you know, everything doesn't
33:40
have to be 100%. I'm sure you're seeing it on the
33:42
different use cases where there's more appetite for risk
33:47
or there's less appetite for risk or there's more appetite to
33:50
get this into Roduction just because we really need something
33:56
here. Yeah.
33:57
But on this side, maybe the process is they're not fully
34:00
dialed in and so we can take a little bit longer.
34:03
That's right, yeah. And it's, I think the same call
34:06
or a similar call that a lot of organizations make even in the
34:09
Dr. space, right. How much do I need and how much
34:12
is good enough, right? You can keep building on that.
34:15
And those are massive investments.
34:18
It's, and I think everywhere it's like the same principles
34:21
that you've got to apply. I think the skills, skill sets
34:25
and the skill set, if you need both to build and maintain,
34:28
right from a, from a software engineering lens, I think that's
34:33
evolved so much, right? The as you're saying, right, the
34:37
low code, no code Ides, it almost is like you don't need a
34:41
full stack developer as much as you needed them.
34:44
I'm not saying you don't need them at all, but you don't need
34:47
them as much as you need, Yeah. But you need them for other
34:49
stuff. If you're looking at the whole
34:52
product as a circle, you need them for this part of the
34:57
circle. And then you can try and get the
35:00
the other stakeholders involved in this part.
35:03
And this was always kind of the case.
35:05
I don't know if you remember it with the ML, you needed
35:07
stakeholders involved, but they couldn't really act.
35:11
They had to be in the meetings and talk about things and say
35:14
like, yeah, but we can't do that because of this or that.
35:16
But you didn't have this idea of.
35:19
Well, let's just empower you with a no code solution.
35:22
That's right, yeah. And that's where I think it's
35:24
becoming very interesting. Right.
35:26
So the skills that you need on the build side is, is less
35:31
coding and more designing. And once you design the right
35:37
solution, it's very little time to to code.
35:41
And then you spend a lot of time in evals and measurement and all
35:45
that. So prompt engineering, you know,
35:48
data scientists who you can measure, compare, run models,
35:51
simulations, that comes into the picture.
35:55
And then on the managed services side, it is again data
35:58
scientists who can tune the models.
36:00
The skill set is completely shifting, right?
36:03
It's not about coding anymore, you know, when you're deploying
36:08
agents, which is very interesting, right?
36:10
Because that impacts, I think the skill set of who, who you
36:16
hire, you know, as, as you're like beginner kind of workers to
36:21
how they grow, what path do they take to grow right within the
36:25
organization. And I think the business
36:27
knowledge is becoming more and more relevant.
36:30
You know, now you want, you know, yesterday, I think when I
36:33
was at your event, I heard about how product owners can start
36:38
coding because you don't right, need to know code.
36:41
So, you know, the product owners who understand the
36:43
functionality, who have been designing the product are
36:46
probably the luckiest right in in this whole mix because now
36:50
they don't need to rely on an engineer to come out with a
36:53
code. They can actually code, right?
36:56
And it's oversimplifying it, obviously, but I think I do see
37:00
that shift. Exactly.
37:02
I I've been hearing the term product engineer more and more
37:06
and then I also saw the trend of the CPOCTO where they're the
37:13
same person. And so it's like, yeah, this is
37:16
a it's a product person, but they're technical and that's the
37:18
new cool thing to do. And but I, I do think that the
37:23
the hard part, yeah, isn't maybe it's not in the code.
37:27
The hard part is in the making sure that if you have an agent,
37:33
it has the right context. And so there's still is a lot of
37:36
data engineering that's going on in the background because the
37:39
getting that context that's all moving data around and who's
37:44
really good at that, the data engineers.
37:45
That's right. That's right.
37:47
Yeah. So it's interesting how it's all
37:49
shifting. And it's also, I think, you
37:51
know, if you if you raise it up to the C-Suite, right, that is
37:55
the birth of the chief AI officer, right?
37:59
We had the CIO, we had the analytics and digital, and then
38:03
now we have AI. And I think with AI, right, it's
38:07
shifting more towards the business.
38:09
It's more of someone who understands the business who can
38:14
then advise the organization on what are the right AI
38:18
investments that's going to then impact, you know, the company's
38:22
productivity and efficiencies and all that good stuff.
38:25
We'll wrap up here. There's a few really cool ideas
38:28
that I'm taking away from this conversation.
38:30
One being the idea of thinking about your system from this
38:36
sustainability first. I always thought about
38:39
sustainability in the other bucket that you put it in, which
38:42
was like, yeah, the hardware try and reduce, reuse, recycle.
38:45
Sustainability. Yeah, this is like, we want this
38:51
AI product to have a long shelf life, So how can we design with
38:58
that goal in mind? Right.
39:00
And that's probably like the biggest thing that I'm going to
39:03
be taking away. It's, it is very oversimplified
39:06
because yeah, of course you would think like that.
39:08
Why wouldn't you? But I, Yeah.
39:12
And it's also easier said than done, Right, Right.
39:14
Right. As anyone knows, you're working
39:18
it on Rag today and then you're working on Agents tomorrow.
39:23
And now next time it's prompt engineering or it's context
39:27
engineering. And so.
39:28
Yeah. Yeah, Yeah.
39:31
You know, if I had to talk about future ready IT operations,
39:37
right, it is really you look at it with a business mindset,
39:42
right? There's I, I think as we look
39:45
into the future, like making sure you're stabilizing your
39:49
current state is so important with data modernization, app
39:53
modernization, process standardization, all those good
39:56
things, right. They are all I think very
39:57
important for a large organization to start with And
40:02
and I also think that you know if.
40:05
If a company has never dabbled in AI before, start with your
40:09
back office. Like I said, it's more forgiving
40:11
and and then kind of move slowly into your front office as you
40:15
build that muscle right within the organization.
40:19
Invest in good training and upskilling right for your people
40:23
sort of thing. Because it's not, you're not
40:27
going to replace people like overnight.
40:29
You're going to upskill them, you're going to elevate them.
40:32
You're going to still have a human in the loop and educate
40:35
them so that it doesn't become a culture of fear, but it's a
40:38
culture of learning. And then, you know, there's, I
40:43
would say, you know, start pilots and different
40:46
departments. So everyone starts getting
40:48
excited, but with that kind of centralized governance, if you
40:52
will, right? So that there's some guardrails
40:55
around everything and take advantage of all the cool
40:59
technologies that everyone's putting out there.
41:01
Like the number of AI startups that's out there is just mind
41:04
boggling. And the names, right?
41:06
So interesting. Also fun, yeah.
41:08
Also fun and you know like the like Asian toys that PwC has is
41:13
a lot of frameworks out there. Take advantage of those
41:15
frameworks because a lot of thought has gone into building
41:18
those frameworks just to make this journey easy.
41:21
And then last but not the least, I would say measure, measure,
41:24
measure, right. That feedback loop is just so
41:27
important.