Latest / Agentic Conversations (formally mlops.community) / AI Hype vs. Real Value

Transcript

AI Hype vs. Real Value · Agentic Conversations (formally mlops.community)

0:00
What's going on, everyone? Today we are talking AI hype
0:03
versus real business value with Manish Dasur.
0:06
He's working at PwC. He's been in the data and AI
0:09
game for over 20 years, helping clients over 100 different
0:15
clients navigate the AI disruption field and also
0:21
evaluate and deliver business value through the strategic
0:26
adoption and application of data AI and now agentic AI efforts.
0:34
Huge shout out to PwC for supporting this episode.
0:37
Let's get into it. I'll tell you what I was
0:45
building real fast because I think this is in line with what
0:49
we're going to talk about right now as far as value and looking
0:52
at costs. I built a little widget that can
0:56
track the spend, my API spend across different providers and I
1:02
gamified it a little bit. So every X amount of money you
1:06
get to choose if it's $20 increments or $100 or $1000, it
1:11
will go cha ching. And so that, you know, you just
1:16
spent some cash and you want to make sure that you have eyes on
1:21
that. A lot of times we don't even
1:23
realize how much we're spending, especially if it's going through
1:26
the API. And next thing you know, maybe
1:30
it's your boss or your boss's boss that's trying to drill down
1:34
and say, hey, what's going on here?
1:37
What are you getting from this? Yeah, Yeah, I couldn't.
1:40
I couldn't agree more. I mean, look, I'm glad you're
1:42
tracking it. I think it's something that's
1:43
also surprising a few organizations, right?
1:46
Like all of the AI programs they launched, they were all
1:49
predicated on a ROIROY equation that they, they got themselves
1:54
aligned to and approved internally.
1:56
And that ROI equation starts to really get some pressure and
1:59
stress if you're not careful about the compute cost and the
2:02
tokenization costs and all those things that can add up.
2:04
So I, I've got a lot of clients right now thinking about how do
2:07
I apply fin OPS to that discipline?
2:10
And to your point, they haven't gotten quite to the Vegas to
2:12
Ching sound, but certainly are thinking about in the same way.
2:16
How do you really mitigate that cost?
2:18
Yeah, I've heard the term tokenomics become more and more
2:22
popular and it's really just looking at your spend and
2:28
recognizing where's the value from it.
2:31
I was talking actually at length last week with a bunch of fin
2:35
OPS folks on how they're classifying it and how they're
2:39
deciding what is valuable spend versus what is a little bit
2:44
supervaluous. Yeah.
2:45
I think that you have some cool thoughts on where and how to
2:50
justify your different initiatives with AI.
2:54
And so maybe we can even just like start with what is getting
3:00
value from AI look like in your opinion?
3:03
Yeah. Well, let's first, let's first
3:05
set a target, right? When I think about clients who
3:07
are doing this, well, I think they're reporting about like a
3:11
30% improvement on value, right? So if they were, for example, if
3:15
they were measuring code production or software
3:18
development, life cycle production for their teams, what
3:21
they're reporting is, hey, previous to giving my team a
3:24
series of AI tools to help them through SDLC, prior to that, we
3:29
were seeing about 100 Sprint points come out of that Sprint,
3:33
right? So I gave my team a task.
3:34
They're going to go develop a bunch of code, and in the past
3:37
they're kind of averaging 100 Sprint points of output per
3:41
Sprint. Now I've enabled them with a
3:43
series of tools, arming them to accelerate to the software
3:47
development life cycle. These tools are often writing
3:50
user stories, They're often writing test scenarios in cases.
3:53
Obviously they're doing code assistant code generation and
3:56
debugging and things like that. And what they're kind of
3:59
reporting back is now that team is able to get through 130
4:03
points per Sprint, right? So they're kind of measuring it
4:06
that way. They're measuring the output
4:07
that way. If in the past it was 100
4:10
points, now it's 130 points. So that's one example of what
4:13
good looks like, right? It's kind of a 30% efficiency
4:16
gain. If I think about some other
4:18
clients, they're applying AI to do some, some more aggressive
4:21
workflows in finance or HR and supply chain, also reporting
4:25
about a 30% automation efficiency in those cases.
4:28
So when we think about how do organizations get value from AI,
4:32
let's first start with what do we mean, what is value?
4:34
And I think 30% roughly is a good target for organizations to
4:37
think about when they think about value from AI, from
4:40
automation and what it can drive.
4:42
Now, how to get to that value is always the challenge, right?
4:45
And, and there's a few points I would share, Dmitri 1, focus on
4:50
the workflow, not the use case right?
4:53
In, in, in client examples that I get to work with clients that
4:57
have clients that describe their AI initiative.
5:00
As you know, we're targeting this pilot case here and this
5:03
proof of concept here and we're working on this use case here.
5:07
Often times I feel like they don't see the end to end the
5:10
EBITDA value that they would want to see.
5:13
Whereas when clients are saying, hey, look, I'm going to focus on
5:16
a workflow, I'm going to embed AI into that workflow.
5:20
And if the workflow previously took me 20 steps and 100 humans,
5:26
now it'll take me 5 steps, 5 AI agents and 15 humans.
5:31
And that delta is the value in EBITDA value that they can take
5:35
home. So if you think about it like
5:37
I've got an accounts payable workflow or I've got an accounts
5:41
receivable workflow, and I'm going to embed AI into that
5:44
workflow, right? So I know in step one, a human's
5:47
going to do that a Step 2, and AI agent's going to do that Step
5:50
3, the invoice is going to get created.
5:51
So when I start applying AI to the actual workflow and I can
5:55
look at the workflow end to end and compare what my old one was
5:59
to what my new one was, that's how I'm seeing clients really
6:01
get EBITDA value, not disconnected use cases,
6:05
connected workflow redesign with AI generating the value.
6:10
So that would be kind of .1. So let's not get into isolated
6:13
tools. Let's think about a connected
6:15
workflow and how AI really changed that workflow all.
6:19
Right. I want to double click on the
6:21
idea. They're real fast about how you
6:23
were saying that 30% increase. And one thing that I've heard
6:28
folks talk about is how they will look back a year ago and
6:33
recognize certain big pushes that they've had to do and what
6:38
productivity looked like around there, how much they code they
6:41
were shipping, whatever these metrics were that they were
6:44
tracking. And they can then compare that
6:48
to now. And it's, it's not necessarily
6:52
apples to apples, but it's a much better way of deciding,
6:58
hey, how much are we spending on AI versus how much more are we
7:03
producing? Yeah.
7:05
And then you can see that in a way that isn't just asking your
7:10
employees to tell me, how much more value do you think AI is
7:17
providing to you? How much more productive are
7:19
you? That self reporting metric I
7:21
think is a little bit biased. And I'm always weary when I see
7:25
those. But I just want, I wanted to
7:27
talk about that. And there's so many other things
7:29
on this workflow piece that I want to dive into, but I know
7:31
you were going to say something else.
7:33
Well, I mean, look, I think your point's a good one.
7:35
In fact, every transformation initiative that I've seen really
7:39
be successful is always challenged to prove value
7:43
ongoing basis, right. I want to every transformation,
7:46
if within the first quarter you want to be able to start to
7:48
demonstrate value, you want to be able to update or increase
7:51
that value quarter over quarter. To your point, it's got to be
7:55
very, very quantifiable, right? So the other maybe point I'll
7:58
add to how do organizations make sure they get value?
8:01
What I've seen a lot is organizations will seek to prove
8:04
the ROI and they'll set up a champion challenger kind of
8:08
test, right? So as they roll out AI
8:11
automation, they'll have a control group working in the old
8:15
process and they'll have a test group working in the new AI
8:19
automated process. And at least for a couple of
8:21
cycles, they'll compare the outputs, they'll compare the
8:24
results. Depending on the use case, the
8:25
comparison might be different, but they'll compare the results
8:28
so they can get themselves to a quantifiable value statement and
8:32
then find a way to tweak it, improve it over time, right?
8:34
So I think that's super important to, to have that
8:38
measure of whether you're measuring cost or speed or
8:41
revenue uplift, whatever it is that you're measuring, but have
8:43
that champion challenger kind of mindset set up that way.
8:47
To your point, you're not guessing it, you're not talking
8:49
about it. You're actually demonstrating
8:51
with some numbers. And that, I think is a is a
8:53
winning formula for organizations.
8:55
Yeah. You know, I saw that and I think
8:59
these days there is no software engineer that I know that
9:03
doesn't want to use some kind of a coding agent.
9:06
So it's not necessarily what are we going to do AI versus no AI
9:13
with especially software engineering and even product
9:15
development, that type of thing, those internal type tools when
9:19
we're dealing with software. I think the really difficult
9:23
part here is you've got some software engineers that are
9:27
burning a ton and getting a ton done and then you've got other
9:31
software engineers, AKA me, who are burning a ton and not
9:36
getting a lot done. And so how do you distinguish
9:39
between that efficient burning of tokens versus the non
9:45
efficient ones? Yeah.
9:47
No, it's a great point. I think we're going to talk
9:48
about a little bit of the the fin OPS angle as well.
9:51
And look, I don't think a lot of organizations have found that
9:54
answer quite yet, right? I do think a lot of
9:57
organizations have been pushing AI everywhere, AI for everyone,
10:02
right? And, and while that's fantastic
10:03
and the right thing to do, the, the cost of that can be quite
10:06
high, right? But I do think now organizations
10:09
are starting to shift a little bit.
10:10
They're thinking about, do I use the right model for the right
10:13
use case? I don't have to use the same
10:14
model for all use cases. So I can kind of right size the
10:17
model to the use case I'm driving that can help me with
10:20
some cost. I can do workflow engineering or
10:23
prompt engineering in a way that can help me manage my cost.
10:26
Right. So those, and then of course,
10:27
I've got the fin OPS angle talking about, to your point,
10:30
the cha ching thing that you were describing kind of every
10:32
time I use, every time I use that compute, use that token,
10:35
I've got a measurement that way. And like I said, I, I think all
10:38
of that has to be baked into the ROI equation of the program of
10:42
the transformation that we're seeking to drive.
10:45
But I will tell you overall, I do think organizations are
10:48
reporting good benefits. 30% benefits are hard to ignore when
10:52
you think about the size of these organizations.
10:54
If you can drive a significant reduction in operating cost or a
10:59
significant growth in revenue generation, then it does at the
11:03
large AI benefits will exceed the value associated to it.
11:06
Yeah, so you gave us the point #1 which was these workflows and
11:11
trying to figure out where you're plugging AI into.
11:14
I wanted to mention that I have a theory that the reason you saw
11:21
so many successful use cases with support chat bots and
11:26
support agents. Yeah, was because that is a
11:31
field where there's very clear SO.
11:35
PS And because of that, it's all mapped out.
11:39
It's easier to create these workflows like you're talking
11:42
about and go end to end. And if you really need to, you
11:46
can escalate up to a human when there's something that will
11:48
stump the agent or it's not in the books.
11:51
But as a friend of mine told me, especially in call centers, you
11:56
routinely get folks who are very new at their jobs and all
12:00
they're doing is looking through the SO.
12:02
PS Yeah. And they're doing a poor job at
12:05
that, and an agent can do it much better.
12:07
So that idea of like having the workflows defined and then being
12:11
able to plug in the agents or the LMS where you can makes a
12:15
ton of sense to me. I look and customer care is an
12:19
excellent example. So let's double click on that
12:21
for a second because I got to tell you a fun story as as as
12:25
important it is as it is to architect it into the workflow.
12:28
And I agree with you, customer care is a fantastic example for
12:31
that. It is still freaking hard.
12:33
And let me tell you, let me give you a couple of things that in
12:36
my opinion, Stump, I had a, I had an opportunity to work with
12:39
a large telecommunications company and they were, they were
12:42
seeking to do exactly what you described, 80 percent, 75% of
12:46
the calls coming into my call center.
12:48
I want an AI agent to be able to handle them and only take it
12:51
into a human when I need to for that remaining 2025%.
12:54
So that's the, that's the, that's the attempt, that's the
12:57
workflow automation. So I, I would say the two
13:00
hardest challenges we found in that work was 1.
13:04
As you know, when you start with a model, it's relatively
13:07
generic. You have to give it an context
13:09
and training to make it specific and useful in the workflow that
13:13
we're talking about, right? So for example, I've got a
13:15
generic customer call centre type of model.
13:18
It needs to be trained in my company's policies, inventory
13:21
plans, things like that, so that when a customer calls, I've got
13:24
relevant information to give it. So when you think about training
13:28
a model like that, a customer could be calling about anything.
13:32
They could be calling about my invoice, my trip, my phone,
13:35
what's wrong with it, etcetera. So the context, the amount of
13:38
training you have to give that model is quite unique.
13:41
And some of that training comes from previous call logs that the
13:45
customer care reps have written down and written down.
13:48
So of course we're going to feed that into the model, right?
13:50
But the model also needs to know about when Dimitri calls in.
13:54
Let's let's use an example. Dimitri calls in and says, hey,
13:57
I'm travelling through Asia next month.
14:00
Mr. cell phone provider, would you plead at please add roaming
14:04
to my plan, right? So I can go around Asia and use
14:07
my phone. Now for an AI agent to be able
14:09
to handle that call, first it needs to know who Dimitri is.
14:12
So it's going to need a way to connect into this.
14:15
I'll call it ACDPA customer data platform, customer 360, whatever
14:20
terminology you use that so the, so the model knows who Dimitri
14:23
is. How many lines does he have?
14:25
Has he paid his bill on time? What phone does he have?
14:29
Does he, how often does he travel?
14:31
What plan does he have today? Right.
14:33
So it needs to know who Dimitri is.
14:35
Then once it knows who Dimitri is, it's going to ask you some
14:37
clarifying questions. What countries are you traveling
14:40
to? What dates are you going to be
14:41
traveling in? Based on the answers you give
14:43
it, it needs to update the algorithm and go into the
14:47
inventory of plans I could offer you and select the one that we
14:51
think is the right one for you, Right?
14:53
Then I've got to make that offer to Dimitri on the phone.
14:57
Dimitri is going to say, yeah, OK, I like that plan, give me
14:59
that one. He's going to confirm the cost
15:01
is, I don't know, an extra $10 a month, and he's going to say
15:04
good at it for that month. Then the model needs to have the
15:07
authority to go back into your invoice and confirm that you are
15:11
now going to pay 10 more, charge your credit card and update your
15:14
plan. That's quite a complicated
15:16
transaction. It needs to connect to your
15:19
billing system is likely written in one like 1 platform.
15:23
Your, your, your planning system is likely in another platform.
15:27
Your customer system is likely in another platform.
15:29
So there's a bunch of API connectivity, a bunch of
15:32
integration that you've got to solve for bunch of context and
15:35
training. You've got a software.
15:36
So it, you know, for those kinds of things, it can be quite
15:39
complicated. But that's why I think having
15:41
the right governance approach, right structure approach and and
15:44
a plan to attack that workflow end to end is super critical.
15:48
Don't forget you if I'm calling you on the phone, you have to be
15:52
doing this with very low latency.
15:56
That's right. That's right.
15:58
And you remember like the models of the past, they were taking
16:00
voice, going to text, generating a response in text and bringing
16:05
it back to voice, which created a lot of latency, right?
16:08
But now in the new world, we can go voice to voice.
16:10
We don't have to go to voice to text to text to back to voice.
16:13
So we can go to voice to voice with the latency is certainly
16:15
starting to get better. But I still think training
16:18
models, contextualizing models and then applying all the
16:20
integration you need with all the enterprise corporate systems
16:24
that models need to pull from and interact with it in order to
16:28
satisfy the customer query is quite a tall order, right?
16:31
And that's why I think organizations need a a practical
16:34
approach to solving that problem.
16:37
Yeah. So there we go.
16:39
That's the workflows on point number one.
16:42
Yeah, you had more points and I remember I cut you off.
16:45
So it keep it rocking on the different pieces that you want
16:48
to hit on. Well, the last thing I look,
16:50
we've talked about focus on workflows.
16:53
We've talked about, you know, use the champion challenger
16:56
model to really to really prove out the value.
16:59
The Third Point I would just tell you is I do think it's very
17:01
important for leaders to build the culture of innovation right
17:05
at the, the only places where I've really seen AI be adopted
17:09
is when leaders are rewarding innovation and pushing the
17:13
organization to reimagine how they do it.
17:16
It's a hard thing to do, right? People have been running these
17:18
workflows for two decades in their jobs, in their roles, and
17:22
now we're saying, hey, that workflow is going to change.
17:24
So if you can in a, in a leadership way, in an
17:27
organizational way, find a way to reward a culture of
17:31
innovation, reward a culture of let's try something new.
17:35
Let's adopt something new. I think that's super critical to
17:38
your success as well. And I'll just tell you, man,
17:40
like I think our, we did some surveys on this stuff.
17:43
I think we saw like 88% of companies say they're doing AI,
17:47
but only 33% of companies say they're scaling AI, right?
17:52
So I still think there's quite a barrier here.
17:54
And as organizations go from I have 10 agents to soon I'll have
17:59
100 to soon I'll have 1000, right?
18:02
These are the kinds of things that are going to help them get
18:03
there. Have you seen this idea of
18:08
rewarding AI and creating that culture of innovation, being
18:14
successful across organizations, especially the larger
18:18
organizations that I can imagine?
18:20
You have folks that are set in their ways and they've been
18:24
doing something for a while, and it's very hard to change.
18:27
Yeah. It's a combination of things, I
18:30
think, right? But I do think I see
18:32
organizations seeking to reward, promote those who are adopting
18:36
innovation. I've even started to see
18:38
organizations who are measuring how often you use AI, right?
18:42
If you're a regular consumer, you're doing your job.
18:44
Organizations are starting to measure how often do you prompt
18:48
your, your local LLM or your local authentic solution?
18:51
How effectively are you actually using AI?
18:54
And that's becoming one of the criterias for for your
18:57
evaluation, for your performance.
18:59
And I think adopting a culture of innovation and and a culture
19:02
that pushes the boundary I think is a good thing to do.
19:06
Yeah, I've seen that at certain companies that can remain
19:11
unnamed. But it's also almost like the
19:14
double edged sword is that it becomes toxic and then you have
19:18
token maxing or just prompting to prompt.
19:22
Yeah. And if you only are going off of
19:24
that, then it can be dangerous too.
19:26
So it it's like everything. It's quite nuanced in that
19:29
regard. Yeah.
19:30
Absolutely, absolutely. But I think, you know, if you've
19:33
got I, I think the best thing you can do as a leader is
19:35
embrace to change yourself, right?
19:37
Be the example that you're seeking for your team to be use
19:40
AI in your everyday use it for the right use cases.
19:43
Be mindful of how much you're spending on it, but also, you
19:46
know, don't hesitate to drive top line growth or bottom line
19:49
growth with AI and be the best example of that.
19:51
And if you can do that as a leader, then hopefully the rest
19:54
of the organization can and use that and follow.
19:57
Yeah, bring them with you. And yeah, where have you seen it
20:00
Feels like you. This is a nice segue into where
20:04
you can fail and where you've seen these efforts fall flat.
20:09
Those who get it right versus those who stumble right is
20:12
always a good question to kind of think about it.
20:14
And it helps clients think about this too, because then they
20:17
avoid those same mistakes that others have failed at.
20:19
The first one I would tell you is the concept of AI everywhere.
20:24
I don't think is the right concept for most clients to
20:26
think about it. When you think about AI
20:28
everywhere, you end up incurring a lot of cost for not a lot of
20:31
value. So I the first kind of point of
20:33
guidance I would give clients there is you don't want to think
20:36
about it like AI everywhere. Of course we want AI to be
20:38
pervasive. That's not what of course that
20:40
we, we want that right. But we also want AI to be very
20:44
focused on those high value use cases that you think are going
20:47
to be game changing for your organization.
20:51
You also might want to get AI focus and start with some low
20:53
risk use cases that won't be super sensitive to the market or
20:57
to your brand. So I think you the first, the
20:59
first point of guidance I would share is don't think about
21:02
applying AI everywhere. Instead think about it as I've
21:05
got this fantastic tool. Of course, eventually it will be
21:08
pervasive in my organization. But if I'm starting out or if
21:12
I'm in those stages, going from 10 implementations to 100
21:15
implementations, I want to be strategic about where I use it.
21:19
I want to be prescriptive about where I use it.
21:21
I want to measure the value that it creates, right?
21:24
So the first thing I would tell you is it's not AI everywhere.
21:27
It's AI applied and utilized in those workflows, in those use
21:31
cases that are going to generate the biggest bang from my buck,
21:34
right? So that's the first point, maybe
21:36
I would share. Second point I would share is,
21:38
look, you have to watch out for the compute costs, right?
21:41
I think, I think consumption driven cost models sometimes
21:45
great challenges. I think it's very, very
21:47
important that organizations have the appropriate governance
21:49
set up. So they're, so they're too,
21:52
they've got some oversight and it's not a surprise.
21:54
The surprise is what I worry about, right?
21:56
A lot of clients have been enamoured with what AI can do.
22:00
They haven't really focused on what do the cost look like as it
22:03
starts to scale. And often they can get
22:05
surprised, right. So I think there you've got a,
22:08
first of all, you've got a architect for efficiency one,
22:12
it's not one model that fits all use cases, right?
22:16
So it's very important that you think about what am I trying to
22:18
do? And I select the right model for
22:20
that. If I'm doing image creation, I
22:22
might pick a certain model. If I'm doing deep thinking, I
22:26
might pick a different model. If I'm doing PowerPoint
22:28
creation, then I might do a different model, right?
22:31
So using the right model for the right use case will help clients
22:34
mitigate cost. 2 the prompt engineering discipline is, is
22:39
one that I, you know, I have to tell you is going to become
22:42
increasingly important. The type of prompt you put into
22:45
a model can generate different types of cost as well.
22:47
So prompt engineering and making sure we've got the model
22:50
architected, the workflow design architected to minimize token
22:54
usage, I think is, is the second thing that I would kind of think
22:56
about there. And then lastly, like you, you
23:00
started the conversation with fin OPS, right?
23:01
But having a financial way of tracking cost per workflow and
23:06
enforcing that budget and forcing that governance, I would
23:10
certainly say is a is a key thing for clients to do.
23:14
So yeah, what I hit on, I hit on, I hit on not AI everywhere.
23:18
The cost model is super important.
23:21
And maybe the last point I'll still share here is as AI become
23:26
smarter, as the reasoning capability of these models
23:30
becomes better, we can apply them to more complex use cases.
23:34
And as the autonomy gets better, as they're able to execute
23:37
actions on their own behalf better, now I can have them make
23:40
complex decisions and execute actions against those complex
23:43
decisions. But what I also see happening as
23:46
those two factors are improving is clients are seeking to use AI
23:51
in more sensitive use cases. I want an AI, we talked about
23:54
customer care. I want an AI agent handling my
23:57
customer care call. Well, that's very important
24:00
because your brand, your policies all need to be is how
24:03
your AI agent needs to behave in a way that's consistent with
24:06
your brand, in a way that's consistent with your policy, so
24:08
your customer has a consistent experience.
24:11
I want an AI agent approving purchase orders.
24:14
And now in this case, I've got an AI agent making a financial
24:18
decision on your behalf and approving purchase orders.
24:20
And money is actually going out the door based on a decision
24:23
that an AI agent has made. So as I think organizations seek
24:27
to use AI in that way, seek to put AI in more sensitive use
24:31
cases, really having a robust framework and methodology for
24:37
testing accuracy, for testing hallucination, for testing
24:41
responsible behavior, right, for essentially auditing agentic
24:45
behavior is going to become super important for our clients.
24:48
That's something at PwC we're pretty excited about because
24:50
obviously we do a lot of risk and auditing and, and you know,
24:53
we'd like to extend that kind of thinking to your AI world as
24:56
well, right? And I think in the workforce of
24:58
tomorrow, if I imagine it to be humans and AI working together,
25:03
we've got to be able to really review, validate, audit that
25:08
work product to make sure it's consistent with what we're
25:10
doing. And as we gain more trust there,
25:14
that'll be the unlock of allowing AI to participate and
25:17
really make decisions in sensitive use cases.
25:21
And just like, just like the cost point, I think the testing
25:24
point, the responsibility point is still an area that a lot of
25:27
our clients haven't explored just because it's relatively
25:30
new. And I think that will be a key
25:32
unlock for the for the industry as well.
25:34
Have you thought through how that could potentially look we?
25:39
Absolutely. We're starting to build
25:40
capabilities now that can test, audit, validate results that are
25:45
coming out of an LLM model. As you know, that's not easy to
25:48
do. In the past, systems we designed
25:51
was one input has a series of finite outputs, Let's say it has
25:55
five. I can test for those outputs
25:57
relatively simple. I can put the input in, I can
26:00
validate that I got the one of the five outputs, and if I did,
26:03
then the test succeeds. Much, much harder to do in the
26:06
agentic world. In the LLM world, one input
26:08
could have million different outputs.
26:10
So we are now building frameworks and testing
26:12
methodologies that essentially use AI to test AI.
26:16
But to be able to do that at scale, that's really the only
26:18
way to do it, right. So yes, we are absolutely
26:20
starting to starting to build frameworks and capabilities that
26:23
we can take the clients and offer them some help in being
26:26
able to test and validate AI results.
26:29
I do agree with you. However, like today, when you go
26:32
buy a car, there's AJD power associate certification.
26:35
So you know you're buying a car that you can trust.
26:37
I do agree with you the, the I don't know what the answer will
26:41
be yet. I don't think the marketer
26:42
industry has has aligned on what the answer will be yet.
26:44
But I do agree, I think clients and organizations are going to
26:47
want a way to essentially certify or some, some sort of
26:52
validate that this agent is good to go.
26:54
I can use it in a sensitive use case.
26:56
I can put it in front of my customers.
26:58
I've got a way to mitigate the risk to my brand and to my
27:01
policy. And I certainly think that once
27:04
once the industry has aligned on the right solution, that'll
27:06
allow us to take a leap forward. In the meantime, I think some of
27:10
the frameworks in the agentic work that we're doing is exactly
27:12
how class should be thinking about that answer.
27:16
Yeah, I really wonder how it would look and what it would
27:19
shape out to be. I've joked around about how you
27:22
potentially would have verified MCP servers, but it's not
27:28
necessarily. What's so weird about this is
27:31
that you can't quite say like, well, maybe you can.
27:37
Companies are Sock 2 certified, right?
27:40
And that means that they've done a bunch of stuff and they've
27:45
gotten that checklist of things and they've passed their Sock 2
27:49
certifications. And there's more stringent
27:51
versions of that and less stringent potentially every
27:55
agent that's out there. You can ask what you can just
27:59
like have a sock 2 type certification for the agents
28:04
that are out in the wild, but it feels much harder to police.
28:08
And it also feels like that might not be the right
28:11
abstraction. So I'm a little bit at a loss
28:15
for what and how you would audit it.
28:18
What I do know is that the idea of just evaling the output, like
28:23
you were saying, that's not going to get you anywhere.
28:26
Yeah. And I, you know, I, I, I
28:29
completely agree. I think we, I think the answer
28:31
to exactly how it's going to be done is still being formulated.
28:34
And it, and it is very nuanced, though it's still very use case
28:36
dependent, right? But I do think responsible AI,
28:39
trusting AI is still, it's still quite a challenge, right?
28:43
When I speak with organizations and we're talking about a
28:45
complex use case, I usually get a mix of opinion of executives
28:49
on the table. There are some executives that
28:50
are like, yeah, good to go, love it.
28:52
Why wouldn't we do it? Let's go right.
28:54
There are other executives who are focused on the financial
28:57
picture and they can't ignore the benefits that AI can drive.
29:00
But then there's always a, you know, executive or two that has
29:02
a higher trust quotient. That feels like we've got to be
29:05
able to demonstrate the the clarity, the accuracy, the
29:09
compliance in order to move forward.
29:10
And I think that's going to be an important lock for the
29:13
industry. Absolutely.
29:14
Another piece that I want to hit on that you mentioned and I tend
29:20
to ponder about quite a bit is in these workflow type
29:25
scenarios, how some of the optimization efforts can be just
29:32
figuring out where to have that human sit in the loop, if at
29:37
all. Because maybe the ultimate
29:40
optimization is that you can let the agent just go wild and
29:45
depending on the use case, like you were saying, if it is this
29:47
low stakes kind of high value use case, well, let it run wild
29:53
and see if it works. But then if an, if a human's
29:56
getting pinged consistently about something that an agent is
30:03
doing, is the value really there?
30:08
I don't know. I guess if, if I don't have to
30:10
do it, I'm going to be happy to have a computer do it.
30:12
I'm not doing math by hand, right, even if it so I do
30:17
understand that. But the whole idea, if you look
30:21
at a workflow and you have these graphs and you have these nodes
30:25
and you're thinking, well, an agent can take over this process
30:28
and then we're going to have a human in here when it comes to
30:32
approving invoices or anything that has to do with money.
30:36
What happens with me, I don't know if you've had this happen,
30:40
is that when I'm asked to verify stuff that the agent will throw
30:45
at me, I just like automatically say, yeah, except let's see what
30:50
happens type thing. And so I'm worried that that
30:54
type of a habit will get instilled in us as we start
30:59
trusting the AI more and more. And next thing you know, we
31:02
don't even look at it and we're there as a box that you have to
31:07
check. Well, I mean, I will tell you
31:09
it's an evolving scale as, as the way I see it, most clients
31:13
who are implementing use cases, implementing workflows with AI
31:17
today certainly are having a human in the loop, right?
31:20
Especially for, for the ones that I would classify anything
31:23
as medium or high sensitivity, they're certainly keeping a
31:26
human in the loop, but it's also with an evolving scale, right?
31:29
The intention is these models get smarter, they learn over
31:31
time, they get more effective over time.
31:33
So the intention is that, hey, when you first start with that
31:36
automation, maybe you have the human look at 50% of what the
31:39
models performing, right? And as you start to gain more
31:42
trust, as you start to gain more confidence that the matter model
31:47
is behaving accurately, and as the model becomes smarter and
31:50
more efficient at what it's trying to do, maybe you can get
31:52
that 50% down to 20%, down to 10%, right?
31:56
And that's how you kind of move up the chain eventually.
31:58
I do think just like when you punch in a number to your
32:00
calculator today, you trust the answer.
32:02
You're not double checking that work today, right?
32:04
But eventually, I do think we get there.
32:05
But I would advise clients and organizations to think about
32:09
that as an evolving scale for the if you've got a sensitive
32:12
use case, for a medium sensitive use case, you want to keep a
32:15
human in the loop. You want to validate that you're
32:17
getting the right answer. And once you've really start to
32:19
prove yourself that over a few quarters, then you can start to
32:22
reduce the amount of human in the loop that you need.
32:25
OK. So I wanted to also talk to you
32:28
about this idea of now you have AI working within your company,
32:36
you're having success. You can do more with less is
32:39
what we're hearing a lot of folks say.
32:41
And I was literally just talking to a friend this morning about
32:45
how I don't buy into the idea of, oh, we're all going to be
32:52
able to do more with less. That means that we're going to
32:56
not need as many people to work because if we have all of this
33:03
free cash flow that's being spit off now, yeah, we're assuming
33:07
that companies are not going to reinvest that into their
33:10
companies and either hire more people or potentially lower
33:15
prices. Yeah, right.
33:17
I agree with you. I think our research indicates
33:20
the same, right? We don't we, we certainly see, I
33:23
think more than 75% of jobs will require reskilling or
33:28
upskilling, but that doesn't mean those jobs are going away
33:31
or that hiring will stop. It just means they get reskilled
33:33
and upskilled, right? So we certainly see the trend
33:36
heading that way. Look, I think about it as the
33:39
future of the workforce is humans plus agents working
33:42
together, right? The benefits are too great to
33:44
ignore. I think we will see that.
33:46
But largely I, I, my hypothesis is more closely aligned to
33:49
yours. I think the job reduction, the
33:53
job displacement is relatively modest at this point.
33:56
I think the job reskilling, upskilling is much more
33:59
relevant. But I always tell folks that,
34:01
look, this might be a controversial thing to say, but
34:05
I will, I will say that if you're in a job function,
34:08
regardless of kind of what your function is, if you're an
34:10
accountant or if you're a salesperson or if you're a
34:13
supply chain person. And if you, if, if you're, if
34:17
you're kind of at the bottom 25% of other accountants, near the
34:21
bottom 25% of other sales leaders, then you probably will
34:25
be challenged with AI. Because if you're, if you're, if
34:27
you're a low performer in that way, you're likely doing a
34:30
series of mundane tasks that AI can replace.
34:33
But if you're in the top 50% of those roles, if you're the top
34:36
50% of accountants, if you're in the top 50% of sales leaders,
34:39
then AI is going to assist you in being able to do a lot more.
34:44
So I kind of think about it. As for those who are good at
34:46
their jobs, they're going to think about AI as a massive tool
34:50
set that they've just gone and now they can go from being good
34:53
to great at their jobs and and that will allow them to create
34:56
more value. Organizations will hire more for
34:58
those skills. We might see a little bit of
35:00
rotation from back office to front office, just like you're
35:03
talking about. As the back office gets
35:04
automated, I can dedicate more time to to revenue course.
35:07
So yeah, I think we'll see all of those.
35:08
But I think if I'm a, if I'm a person in the labor force, I
35:12
always say this to, I don't think AI is going to take your
35:15
job, but I think another human that's using AI, if you're not
35:19
using AI might take your job, right?
35:21
So I kind of the way I kind of think about it as whatever your
35:24
function is, you need to be thinking about how can I do more
35:27
in my role? How can I do more in my
35:29
function, what with using AI? And that's going to not mean
35:34
more hours. That's not going to mean more
35:36
work. It's actually might mean less
35:38
hours and less work, but it's going to mean more output.
35:40
And that's how I think. That's how I think humans should
35:43
be thinking about that. Yeah, in every job, I have yet
35:47
to come across a job that doesn't have just an absurd
35:52
amount of rote work that I would happily automate away any day of
35:58
the week. And I guess I'm very privileged
36:02
to be talking about like this kind of blue collar work.
36:07
It's a much different story than if we're or blue collar is a
36:11
white collar. I'm getting confused here on
36:15
basically I'm sitting behind a keyboard and doing it.
36:18
So I'm much more privileged to be able to sit behind a keyboard
36:22
and do my work in that way. And I know that there's a lot of
36:25
clicking around that I can automate away.
36:29
So I will say that I am fully on board with a ton of rote work
36:39
that can just get out of what I need to do and allow me to focus
36:45
on what is important. And the real key here is this
36:51
discernment of knowing what's important and knowing where I
36:56
can add that extra value. But I still, I still will like
37:03
say this until I'm blue in the face that what I see with
37:10
companies is that they're not going to just say, all right,
37:14
cool, we now can do the work of 100 people with 50 people.
37:20
We're only going to have 50 people now, right?
37:24
And then just continue to do the work of 100 people.
37:26
No, what I'm seeing is they're going to take 100 people, use
37:33
the 100 people, but just have the output now of double that.
37:38
So now they have maybe the output of 200 people.
37:41
And it's a, it's one of those ones.
37:45
Actually, I was just seeing a friend sent me something on one
37:49
of these big corporations that did exactly that, where they
37:53
realized, oh, because of AI, we are now going to not need these
37:59
positions. But instead of this trope that
38:02
you're hearing thrown around a lot of, all right, we got rid of
38:05
20% of our workforce because we're able to do more with less.
38:09
And AI is the reason for these firings.
38:13
They said no. What we're going to do is we're
38:15
going to purposefully reskill these folks like you're talking
38:19
about. Yeah.
38:21
I mean, I it's spot on, spot on, right.
38:23
Like I, I think about it as AI is going to reset your operating
38:27
model, that's how you should think about as an organization.
38:30
It is going to reset your operating model where as we
38:32
automate the workflows, that means the operating model
38:35
changes. And when the operating model
38:37
changes, that means people's jobs change, not that those jobs
38:41
go away. Those jobs change.
38:43
The operating model has changed. I might need less people doing
38:47
one function, but I need more people doing a different
38:49
function. And that other function creates
38:51
more value. And the people that I have in
38:53
that function can use a bunch of tools for the automated mundane
38:56
tasks, right? So I, I completely agree.
38:58
I think you've got to think about the operating model is
39:02
changing. I'm essentially now an
39:04
organization that has humans and agents working together.
39:08
And when I think about that equation, when I think about
39:10
that peanut butter and Jelly sandwich, the operating model of
39:14
that should be different than the operating model today.
39:16
And that will that will create the job change the job
39:20
reskilling that I think we're going to see in the market.
39:22
Yeah, I was just on a podcast a few weeks ago and the title of
39:26
the podcast was we're all managers now, because we've all,
39:31
we now have these, the agent that we can go and spin off and
39:36
if we manage them correctly, hopefully it makes our lives 10
39:40
times easier and we're still able to get everything done and
39:44
more Exactly, exactly right. So I mean anything that I should
39:52
be asking you that I haven't? No, I think, I think we've hit
39:56
out some. I think we've hit on the right
39:57
topics, right. I I guess I would just part with
40:01
transformation. Over experimentation, right?
40:05
I think the first topic we talked about how to get really
40:07
good value from AI, the best examples of that, the most value
40:12
I'm seeing clients generate is when they're applying it to an
40:14
end to end workflow transformation.
40:17
Not an experiment here, not a use case here, and a POC there.
40:21
So that I would kind of wrap us up, but that would be point
40:23
number one point #2 I would say governance, cost governance,
40:29
risk governance, all those those governance is, is a key lever of
40:32
success, right? So and it's separating leaders
40:35
from laggards, right? So if you've got the right
40:38
governance set up, fin op set up it, then then I think you're
40:41
going to have some success. And then, you know, lastly, I
40:44
think mindset matters, especially early on, a lot of
40:49
clients are seeking to make a shift.
40:51
They feel like it's going to be an organizational shift and
40:53
operating model shift. And it is, I think it's very
40:57
important for leaders, but all the way up and down the
41:00
corporate work pyramid to really adopt this mindset of of course
41:04
we're going to innovate, Of course we're going to relearn.
41:06
You know, I think the, the, this is the age of continuous
41:10
learning, right? If you went to school one year
41:13
ago or if you went to school 20 years ago, all good.
41:16
But this is the age of continuous learning.
41:19
Everyone, whether you're 20 years in the game or one years
41:21
in the game, is going to need to find a way to continue to learn
41:25
new capabilities, new tools, new work automations that are
41:28
coming. Being able to learn those
41:30
things, apply those things to my job and and make my work output
41:34
even stronger and better. That's the secret to success
41:37
now. And the beauty is that you can
41:40
use AI to learn about AI. And how to better.
41:45
Yeah, Yeah. Look, I joke around about this
41:47
up. It has never been easier to
41:49
learn. This is the age of information
41:51
and the past. People had to go travel and and
41:54
walk to the top of a mountain to sit down with a guru and learn
41:57
something. Then people had to pay thousands
41:59
of thousands of dollars to attend a university to go learn
42:01
something. It has never been easier than
42:04
today to learn, right. There's so much information
42:07
available so it's never been easier to learn, but the onus is
42:10
on all of us to always be learning and always be applying
42:15
and if we can do that then then we're going to be successful in
42:17
the workforce. All right, Manish, I appreciate
42:20
you man. So great talking to you, so much
42:24
fun. Thank you for having me.