Latest / The DEX Show: A Show for IT Change Makers / 🪟 Same Work, More Windows: Why AI Isn’t Paying Off Yet (w/ Anthony Firmin)

Transcript

🪟 Same Work, More Windows: Why AI Isn’t Paying Off Yet (w/ Anthony Firmin) · The DEX Show: A Show for IT Change Makers

• 0:01
  Speaker 1: Welcome it. Change makers to the Deck Show with Tim
• 0:05
  Speaker 1: Flower and Tom McGrath. Let's get into it.
• 0:08
  Speaker 2: I've got to change that song. Everybody. Oh, don't necessarily
• 0:12
  Speaker 2: mean that I've got to change the song, but I've
• 0:14
  Speaker 2: got to change that voiceover with Tim Flower and Tom
• 0:17
  Speaker 2: McGrath because, as listeners will buy now know, last week's
• 0:21
  Speaker 2: episode was Tim Flower's final episode as co host of
• 0:26
  Speaker 2: The Deck Show and recording this one actually just after
• 0:31
  Speaker 2: our trip to our trip to Lisbon. I'd never been
• 0:35
  Speaker 2: to Lisbon before, but my, what a beautiful place. We
• 0:38
  Speaker 2: were there for the next thing kickoff, which, by the way,
• 0:42
  Speaker 2: for the Deck Show is often and I think the
• 0:45
  Speaker 2: dex industry at large, although it may not always know,
• 0:47
  Speaker 2: it is where the year truly begins. So twenty twenty
• 0:50
  Speaker 2: six Index is now officially begun, and I had a
• 0:54
  Speaker 2: wonderful trip to Lisbon. Hopefully you all enjoyed that episode
• 0:58
  Speaker 2: that came out last week. If you haven't listened to
• 1:00
  Speaker 2: it yet, do tune in it. It was moving for
• 1:04
  Speaker 2: us involved. I think it was moving for the audience,
• 1:07
  Speaker 2: and it was also it was also a lot of
• 1:09
  Speaker 2: fun and a lot of people talking to me out
• 1:12
  Speaker 2: and Lisbon about what this show is going to do next,
• 1:15
  Speaker 2: and I keep saying we're going to be making some
• 1:16
  Speaker 2: announcements very soon, and we are. Things are really coming
• 1:19
  Speaker 2: into view where Look, we take it Tim leaving as
• 1:24
  Speaker 2: a challenge to this show. You know, we know what
• 1:26
  Speaker 2: an amazing co host Tim's been for the last five
• 1:29
  Speaker 2: years and how valued his voices by the listenership. So
• 1:32
  Speaker 2: we know we've got to up our game and we
• 1:35
  Speaker 2: are intent on doing so. We're very excited about what
• 1:39
  Speaker 2: we've got planned for twenty twenty six. People will be
• 1:42
  Speaker 2: hearing more about that soon, as well as some change
• 1:45
  Speaker 2: to the opening show music. You know, there's much to do.
• 1:49
  Speaker 2: Now we've been able to say goodbye to Tim properly,
• 1:51
  Speaker 2: it's time to move on into the future. And with that,
• 1:55
  Speaker 2: I'm hugely excited to welcome back a returning guest to
• 1:59
  Speaker 2: Today Show. He's Anthony Fermin, an AI and Digital Transformation leader.
• 2:05
  Speaker 2: I'm previously a twelve year veteran as an IT leader
• 2:08
  Speaker 2: at E Why I have that right?
• 2:11
  Speaker 3: Great? Yeah? Absolutely?
• 2:12
  Speaker 2: What have you been up to? How are you?
• 2:15
  Speaker 4: Yeah, I'm doing very well.
• 2:16
  Speaker 1: Good.
• 2:17
  Speaker 5: Good to see you again, Tom, of thanks having me back,
• 2:19
  Speaker 5: and I have to say that I'd be lying if
• 2:21
  Speaker 5: I didn't say that there's a a little bit bit
• 2:24
  Speaker 5: of me that's a little disappointed to have missed Tim's
• 2:28
  Speaker 5: last show by one episode. So I wish I've been
• 2:30
  Speaker 5: all the best.
• 2:32
  Speaker 3: What have I been up to?
• 2:33
  Speaker 5: So, as you know, over the last couple of years,
• 2:35
  Speaker 5: I've been operating really at the intersection of AI and
• 2:38
  Speaker 5: experience management, which is really you know, what makes people
• 2:42
  Speaker 5: trust these tools come back to them and actually change
• 2:45
  Speaker 5: how they work. And a lot of the focus has
• 2:48
  Speaker 5: been taking I guess, human signals, you know, what people
• 2:51
  Speaker 5: find confusing, frustrating or genuinely helpful, and essentially closing the
• 2:57
  Speaker 5: loop so that the technology keeps getting better, because as
• 3:01
  Speaker 5: you'll know, you know, AI only sticks when it makes
• 3:03
  Speaker 5: work feel kind of simpler, not just different. And last
• 3:07
  Speaker 5: year I got to deliver this for a very large
• 3:11
  Speaker 5: professional services firm, using gen jen AI sorry Energetic AI
• 3:16
  Speaker 5: to tackle what was a really very real problem of
• 3:20
  Speaker 5: internal knowledge fragmentation and in turning it into something that
• 3:23
  Speaker 5: people could rely on day to day. And I think what,
• 3:27
  Speaker 5: you know, what became really clear to me is that
• 3:30
  Speaker 5: AI needs a kind of persistent enablement capability around it.
• 3:35
  Speaker 3: I have to say, my experience.
• 3:36
  Speaker 5: Probably over the last year says that you know, adoption
• 3:41
  Speaker 5: implies a kind of one off event, and I've been
• 3:43
  Speaker 5: with AI it's quite different. You need this persistent enable
• 3:48
  Speaker 5: enablement capability around it, right, And I think in practice
• 3:53
  Speaker 5: that boils down to an operate model that kind of
• 3:56
  Speaker 5: keeps the portfolio focused on the journeys that mattam, that
• 4:01
  Speaker 5: proves value safely, that builds repeatable foundations, and then really
• 4:05
  Speaker 5: scales what works and continually improves outcomes. So over the
• 4:10
  Speaker 5: last few months I've been kind of designing and stress
• 4:13
  Speaker 5: testing what that capability needs to look like in the
• 4:15
  Speaker 5: real world. So, in short, it's been it's been a
• 4:18
  Speaker 5: busy but fantastic year.
• 4:20
  Speaker 2: I didn't realize you'd had the hands on experience of
• 4:23
  Speaker 2: actually executing. Actually, I've just I'd be honest, I've just
• 4:26
  Speaker 2: been following your articles on LinkedIn and I thought, I
• 4:29
  Speaker 2: thought these are great take specifically on AI and adoptions,
• 4:33
  Speaker 2: of the fact that you've you've they're best stressed, not
• 4:37
  Speaker 2: just takes, and.
• 4:39
  Speaker 3: There you know, battle scars are there, excellent.
• 4:43
  Speaker 2: Well, give us give us your your informed, scarred overview
• 4:48
  Speaker 2: of the good, the bad, or the ugly as it
• 4:49
  Speaker 2: pertains to AI transformation in the in the industry more broadly,
• 4:53
  Speaker 2: at the moment.
• 4:54
  Speaker 3: Okay, So.
• 4:57
  Speaker 5: Okay, So where are we I think as a nation
• 5:00
  Speaker 5: where we're kind of in a messy middle. It's a
• 5:04
  Speaker 5: difficult one to position because obviously maturity varies so much
• 5:09
  Speaker 5: across organizations. But let's surmise at the messy middle. I
• 5:15
  Speaker 5: think the good is, you know, there are genuine pockets
• 5:19
  Speaker 5: of AI value, right, especially in knowledge work, in drafting
• 5:23
  Speaker 5: and in.
• 5:23
  Speaker 3: Support, where AI is really well integrated.
• 5:27
  Speaker 5: I think the bad there's a lot of AI that's
• 5:30
  Speaker 5: still kind of bolted on, you know, so employees get
• 5:33
  Speaker 5: the same work but more windows effect. And I think
• 5:36
  Speaker 5: the ugly is is really the experienced debt so fragmented tools,
• 5:41
  Speaker 5: unclear guardrails, you know, low trust, driving workarounds, and shadow AI.
• 5:47
  Speaker 5: And I think there's probably one more thing I'd add,
• 5:51
  Speaker 5: because it sits underneath all of that, And that's the
• 5:56
  Speaker 5: you know I think I alluded to earlier, But I
• 5:58
  Speaker 5: think too many organizations AI is kind of one and done, build, deploy,
• 6:03
  Speaker 5: move on, And in reality it behaves like a living system, right,
• 6:07
  Speaker 5: quality drifts, context changes, and trust is fragile. Right, So
• 6:12
  Speaker 5: you sort of you need this continuous improvement loop that
• 6:15
  Speaker 5: sort of keeps tuning quality, workflow, fit, and that sort
• 6:19
  Speaker 5: of day to day experience.
• 6:22
  Speaker 2: And you know how when you think about AI in
• 6:25
  Speaker 2: the digital work place specifically, how do you define what
• 6:30
  Speaker 2: it is and what that distinction means for those leaders
• 6:32
  Speaker 2: responsible for driving adoption and transformation.
• 6:36
  Speaker 3: Such a good question.
• 6:38
  Speaker 5: How do you define it AI? I guess I guess
• 6:42
  Speaker 5: I think of it as as a kind of capability
• 6:45
  Speaker 5: layer across tools and workflows. And you know, because the
• 6:51
  Speaker 5: latest jen AI is probabilistic, I hate having to say
• 6:54
  Speaker 5: that word. You know, you're you're you're essentially managing trust
• 6:57
  Speaker 5: in quality, not just kind of uptime. So you know,
• 7:01
  Speaker 5: leaders aren't just deporting software, they're kind of they're changing
• 7:05
  Speaker 5: how work gets done, and they're setting new norms around judgment,
• 7:10
  Speaker 5: risk and accountability. So you know, practically that means you
• 7:15
  Speaker 5: need clear guardrails, you need quality management. And this, as
• 7:18
  Speaker 5: I say, this persistent enablement capability that gives you measurement,
• 7:22
  Speaker 5: feedback loops and operational ownership, right, so that system keeps
• 7:27
  Speaker 5: improvement other always, I think the risk is you you
• 7:30
  Speaker 5: kind of get this burst of curiosity at the beginning
• 7:33
  Speaker 5: and then sort of drop off when people hit those
• 7:36
  Speaker 5: first few kind of I can't trust this moment.
• 7:40
  Speaker 2: Yeah, we'll come back to trust in a minute. I
• 7:42
  Speaker 2: think you've written very interestingly about that that part of
• 7:45
  Speaker 2: the overall picture. But to just harken back to something else,
• 7:49
  Speaker 2: you've already a phrase you've already used here, and you
• 7:51
  Speaker 2: using some of your writing, which is same work, more windows.
• 7:56
  Speaker 2: First of all, this is a kind of an anecdotal
• 8:00
  Speaker 2: kind of perception to a degree, right, But why do
• 8:02
  Speaker 2: anecdotal reports certain why do anecdotal reports matter so much
• 8:06
  Speaker 2: in this area? And how would you characterize that general
• 8:10
  Speaker 2: experience of AI in the workplace so far?
• 8:13
  Speaker 5: Okay, I think, yeah, anecdotes are important because you know
• 8:18
  Speaker 5: they're there. They're that sort of earlier signal that ais
• 8:21
  Speaker 5: increasing cognitive load, right instead of and those anecdotes kind
• 8:26
  Speaker 5: of they often point to, you know, context switching, duplication,
• 8:30
  Speaker 5: kind of unclear happy paths if you like. And you know,
• 8:34
  Speaker 5: leaders can see licenses and usage, but employees, you know,
• 8:38
  Speaker 5: they can tell you when the reality is you know,
• 8:42
  Speaker 5: I'm doing the same job or it's the same but
• 8:46
  Speaker 5: just with more tabs or more copy and paste. And
• 8:48
  Speaker 5: more checking and validating, right, And that's usually the sign,
• 8:52
  Speaker 5: probably the first sign that you're building experienced debt. And
• 8:56
  Speaker 5: so I guess if I characterize the workplace AI experience
• 9:01
  Speaker 5: so far, I think it's really promising, but it's uneven.
• 9:05
  Speaker 5: In pockets, it's brilliant. I think for a lot of people,
• 9:08
  Speaker 5: it's still sort of assist in a sorry assistive, but adjacent.
• 9:13
  Speaker 5: You know, you ask the AI and then you validate it,
• 9:16
  Speaker 5: and then you rework it and then you sort of
• 9:18
  Speaker 5: copy and paste it into the system you're actually working in.
• 9:22
  Speaker 5: And you know that sort of results in this new
• 9:26
  Speaker 5: time tax if you like, right, that context switching that
• 9:29
  Speaker 5: duplication uncertainty around you know what's safe.
• 9:33
  Speaker 3: And in quality.
• 9:37
  Speaker 5: So I think the response or the move is to
• 9:40
  Speaker 5: is to treat them as hypotheses. Right, So map the task,
• 9:44
  Speaker 5: find the friction points, then design the workflow so that
• 9:48
  Speaker 5: AI is integrated. It's not just another tab or another tool.
• 9:52
  Speaker 5: And when you do that and you sort of keep
• 9:54
  Speaker 5: tuning based on experience and usage signals, then you know,
• 9:58
  Speaker 5: I think the anecdotes change. People stop saying more windows
• 10:02
  Speaker 5: and they start saying, yeah, this genuinely saves me time
• 10:04
  Speaker 5: and effort.
• 10:06
  Speaker 2: We're biased because we have a deck show, right, but
• 10:10
  Speaker 2: we do we do, we do. We do see decks
• 10:13
  Speaker 2: as a particularly timely area of it. You know, it's
• 10:20
  Speaker 2: time is now, shall we say? And AI only increases
• 10:24
  Speaker 2: that relevancy. And I think you've kind of spelled out
• 10:29
  Speaker 2: to a degree why that might be. Do you see
• 10:32
  Speaker 2: dex as becoming more significant in this era with its
• 10:35
  Speaker 2: ability to mention human perception, being almost a human in
• 10:39
  Speaker 2: the loop or identifying them.
• 10:41
  Speaker 3: Yeah, absolutely so.
• 10:43
  Speaker 5: I guess as short as I do, I think your
• 10:46
  Speaker 5: perception becomes this sort of central metric because you know,
• 10:51
  Speaker 5: we're very either deciding factories rarely can it do it right?
• 10:55
  Speaker 3: It can't.
• 10:56
  Speaker 5: The deciding factory is do I trust it? And you know,
• 10:59
  Speaker 5: does it use the effort for me day to day?
• 11:02
  Speaker 5: And you know people's confidence and clarity around what's safe
• 11:06
  Speaker 5: and whether AI feels like it's helping or adding cognitive load?
• 11:10
  Speaker 5: They are you know, they're essentially leading indicators of whether
• 11:13
  Speaker 5: adoption will stick. So if you know perception is negative,
• 11:17
  Speaker 5: you'll often see drop off long before you know you
• 11:20
  Speaker 5: see a problem in the project plan. And that's for
• 11:23
  Speaker 5: me where it intersects with DEX. Right, DEX gives you
• 11:27
  Speaker 5: the operational context around those perceptions, so you know what's
• 11:32
  Speaker 5: happening in the digital environment when when people try to
• 11:36
  Speaker 5: use AI. So if sentiment says, you know, this is
• 11:39
  Speaker 5: slowing me down or I don't trust it, you know,
• 11:41
  Speaker 5: then telemetry signals can really help pinpoint the why.
• 11:44
  Speaker 3: Right is it? Reliability issues?
• 11:47
  Speaker 2: Is it?
• 11:47
  Speaker 3: Latency?
• 11:48
  Speaker 5: Is it tool hopping, context switching is it I don't know,
• 11:50
  Speaker 5: spikes and escalations, self service is broken, you know. In
• 11:56
  Speaker 5: other words, I guess i'd surmise and say perception tells
• 12:00
  Speaker 5: you that there's smoke, and I think dex helps you
• 12:02
  Speaker 5: find where the fire is.
• 12:04
  Speaker 2: That's cool and mentioned it again there. But so let's
• 12:07
  Speaker 2: drill into the question of trust and the degree to
• 12:10
  Speaker 2: which trust or of a lack of it shows up
• 12:12
  Speaker 2: in that user experience of AI, and what makes it
• 12:15
  Speaker 2: such a critical factor in adoption. And I would add
• 12:17
  Speaker 2: to this question, Anthony, what can be done to increase trust,
• 12:21
  Speaker 2: to shure it up, to support people in these areas
• 12:24
  Speaker 2: where there's that detrimental insecurity.
• 12:28
  Speaker 1: Yeah.
• 12:28
  Speaker 5: Absolutely, I don't think there's a more fundamental questioning when
• 12:33
  Speaker 5: it comes to AI adoption.
• 12:34
  Speaker 3: So, you know, if there's.
• 12:36
  Speaker 5: One thing that makes our breaks AI uptake its trust right.
• 12:40
  Speaker 5: People grumble their way through, you know, a slower, awkward system.
• 12:44
  Speaker 5: We've all done it, but they won't hand over judgment
• 12:48
  Speaker 5: to something they don't believe it. And what's tricky, I
• 12:51
  Speaker 5: think is that organizations are really trying to clear several trust.
• 12:55
  Speaker 3: Hurdles at the same time, whether they know it or not.
• 12:59
  Speaker 5: One is, you know, trust in the technology itself, So
• 13:02
  Speaker 5: is it dependable enough, is it giving me accurate answers?
• 13:06
  Speaker 5: Does it fail in ways that are understandable rather than surprising?
• 13:10
  Speaker 5: And another is trust in the employer. You know, is
• 13:13
  Speaker 5: data and usage being used to impro improve support and
• 13:18
  Speaker 5: service right, or is it there to sort of monitor,
• 13:22
  Speaker 5: score and judge people. And then there's trust in the individual, right,
• 13:26
  Speaker 5: do I know when it's safe to us is right?
• 13:28
  Speaker 5: How to check it? How not to get caught out,
• 13:31
  Speaker 5: you know, by putting something in there that should not
• 13:34
  Speaker 5: go in there. And then there's this kind of there's
• 13:39
  Speaker 5: this fourth layer that often sits in the background, right,
• 13:42
  Speaker 5: which I'd kind of label self protection. I think a
• 13:46
  Speaker 5: lot of employees quietly assume, maybe not so quietly in
• 13:49
  Speaker 5: some cases, but they assume that Once AI is proven
• 13:53
  Speaker 5: and embedded, it becomes an argument for reducing roles for
• 13:57
  Speaker 5: you know, moving work to lower cost locations, et cetera.
• 14:01
  Speaker 5: And in that context, you know, sort of hesitating or
• 14:05
  Speaker 5: failing to adopt is an stubbornness, right, So it's a
• 14:08
  Speaker 5: rational response to or a perceived threat.
• 14:11
  Speaker 3: So what do you do about it? I think from a.
• 14:14
  Speaker 5: Leadership perspective, building trust is bigger than publishing guidance.
• 14:19
  Speaker 3: Right.
• 14:19
  Speaker 5: It's you need, yes, you need kind of a clear
• 14:23
  Speaker 5: set of guardrails, you need quality standards, you need an
• 14:27
  Speaker 5: sort of easy way to escalate or verify. But I
• 14:30
  Speaker 5: think leaders also need to be really explicit about their intent.
• 14:34
  Speaker 5: So how will any efficiency be reinvested? Is it in
• 14:38
  Speaker 5: a better service and innovation in learning? Is it redeploying people?
• 14:42
  Speaker 3: Right? And then don't don't just say it.
• 14:45
  Speaker 5: You then have to sort of prove it through consistent
• 14:47
  Speaker 5: decisions and behaviors. And I think that's when you know
• 14:50
  Speaker 5: when confidence grows, an adoption can become self sustaining.
• 14:55
  Speaker 2: You mentioned you're on the ground experience there, and I'd
• 14:58
  Speaker 2: be fascinated to know how you addressed particularly the latter
• 15:02
  Speaker 2: point in that adoption process. You you've you've worked on
• 15:06
  Speaker 2: I R L because you know it's it's easy to
• 15:11
  Speaker 2: say to employers, well, you need to you need to
• 15:13
  Speaker 2: chart out wherever reinvestment is going to happen, wherever career
• 15:16
  Speaker 2: development is going to happen. But are we sure about
• 15:19
  Speaker 2: the answers to us questions collectively as yet? You know,
• 15:23
  Speaker 2: this seems like something we need to be articulating together
• 15:25
  Speaker 2: a bit more than we currently are. What what does
• 15:28
  Speaker 2: the next generation of what do the next generation of
• 15:30
  Speaker 2: technology jobs look like? For instance?
• 15:32
  Speaker 5: Right? Yeah, yeah, yeah, I think it's I'm not sure
• 15:37
  Speaker 5: that anyone is doing that incredibly.
• 15:39
  Speaker 3: Well and fun.
• 15:39
  Speaker 2: Yeah, I'm aware of that at the moment. It's like
• 15:42
  Speaker 2: I've noticed, you know, Yeah, it's the big.
• 15:44
  Speaker 5: Question everyone's trying to work out.
• 15:46
  Speaker 3: I mean, my my hope is that.
• 15:50
  Speaker 5: You know, what A does to us in this in
• 15:52
  Speaker 5: this phase of AI is in the sort of augmentation phase,
• 15:56
  Speaker 5: is that if it enables organizations to move employees up
• 16:00
  Speaker 5: the value curve. Right, So this isn't about investing to
• 16:05
  Speaker 5: reduce headcount, It's about how do we actually make work
• 16:11
  Speaker 5: more human, get people focused on tasks that generate more valuable,
• 16:17
  Speaker 5: that are more rewarding to those individuals and get rid
• 16:19
  Speaker 5: of the dross and know that you have to deal
• 16:21
  Speaker 5: with on a day to day basis. So I don't
• 16:26
  Speaker 5: have a solid answer for you. I think a lot
• 16:28
  Speaker 5: of organizations are working out. It also depends ultimately on
• 16:33
  Speaker 5: what leadership are really seeking out of their AI, truly,
• 16:37
  Speaker 5: and I'm not even sure that organizations as yet are
• 16:40
  Speaker 5: even clear on what they want from my eye. They
• 16:42
  Speaker 5: just know that they need to invest in it because
• 16:44
  Speaker 5: they'll want to fall behind the curve right and lose
• 16:48
  Speaker 5: out to competitors. So there are some really really big
• 16:51
  Speaker 5: questions that are very much unanswered in a lot of organizations.
• 16:57
  Speaker 2: The trust element it brings to mind sort of ambiguous
• 17:01
  Speaker 2: area of adoption, which I felt was highlighted in the
• 17:05
  Speaker 2: amusing stories around v m dash last year. Right, you know,
• 17:09
  Speaker 2: people being able to eliminate the m dash from their
• 17:12
  Speaker 2: work so that they would occlude the part of the
• 17:15
  Speaker 2: participation of AI in it, you know. And it might
• 17:18
  Speaker 2: be as simple as you know, you put a piece
• 17:20
  Speaker 2: of written work in and you say, give it a
• 17:22
  Speaker 2: quick edit the spelling or something you know, you know,
• 17:26
  Speaker 2: grammar and spelling check, you know, and outcomes the dashes.
• 17:30
  Speaker 2: I mean, suddenly you think and I was one of
• 17:31
  Speaker 2: these people only if everyone thinks your work is a
• 17:34
  Speaker 2: completely AI generated But it points to kind of paradox
• 17:38
  Speaker 2: that people who are using AI often feel like compelled
• 17:43
  Speaker 2: to hide that they're using it as if it is
• 17:46
  Speaker 2: if it devalues their own production, as if it devalues
• 17:50
  Speaker 2: their own contribution, which I think in a different way
• 17:53
  Speaker 2: to not using AI points to that same sense of oh,
• 17:56
  Speaker 2: if I if it will if I seem to depend
• 18:00
  Speaker 2: on it, or if it seems too powerful, it will
• 18:03
  Speaker 2: make me that much more expendable in the eyes of
• 18:06
  Speaker 2: my employer or my colleagues.
• 18:08
  Speaker 3: Yeah.
• 18:09
  Speaker 5: Absolutely, And that's this is this sort of touches on
• 18:13
  Speaker 5: why you know, my operating of the intersect of AI
• 18:17
  Speaker 5: and and you know, experience management is so key because.
• 18:20
  Speaker 3: AI is is a is.
• 18:23
  Speaker 5: It's a human challenge, you know, a technology challenge, and
• 18:25
  Speaker 5: you just touch in one tiny aspect on it. It
• 18:28
  Speaker 5: is about us as humans and our unique and strange behaviors.
• 18:33
  Speaker 2: Euroses AI is not as neurotic, because that's for sure.
• 18:39
  Speaker 2: So let's let's look at r o I. It's an
• 18:42
  Speaker 2: interesting area, right, and it's something that we we we
• 18:46
  Speaker 2: we we saw a lot about in Lisbon a spoiler alert,
• 18:50
  Speaker 2: But how should organizations be thinking about measurement and visibility?
• 18:54
  Speaker 2: More broadly, what does meaningful r o I or meaningful
• 18:58
  Speaker 2: demonstrable ro O I from a I look like in
• 19:01
  Speaker 2: real terms in your view?
• 19:03
  Speaker 3: Right? Okay, probably quite a big answer.
• 19:08
  Speaker 2: To this, confessor, and take your time.
• 19:12
  Speaker 5: So I guess quantitatively, I think most organizations are measuring
• 19:16
  Speaker 5: what's the easiest and most available.
• 19:18
  Speaker 3: And this kind of also takes me.
• 19:19
  Speaker 5: Back to experience management around kind of traditional tools if
• 19:23
  Speaker 5: you're like you know, things like licenses assigned and logins
• 19:27
  Speaker 5: and prompt volume and feature clicks basically activity. Some are
• 19:33
  Speaker 5: also tracking kind of model centric metrics like latency or
• 19:36
  Speaker 5: offline quality scores, and that's useful, right, but it doesn't
• 19:39
  Speaker 5: tell you whether performance is improving. So what we're I think,
• 19:45
  Speaker 5: I think what we're not measuring well is the unit
• 19:47
  Speaker 5: that really matters, which is task and journey impact, right,
• 19:51
  Speaker 5: things like end to end, time to complete, first time right, quality,
• 19:55
  Speaker 5: rework escalation see humans right. And also that the hidden
• 20:01
  Speaker 5: AI tax that I mentioned earlier earlier of verification and
• 20:04
  Speaker 5: copying outputs between systems, because if you if you don't
• 20:08
  Speaker 5: kind of capture and measure the rework and validation layer.
• 20:11
  Speaker 5: Then kind of AI can look healthy on usage, but
• 20:15
  Speaker 5: value stays flat.
• 20:16
  Speaker 3: Right, So you're.
• 20:19
  Speaker 5: You're you're measuring the upside but not the downside of
• 20:21
  Speaker 5: AI at least.
• 20:24
  Speaker 3: So when so.
• 20:28
  Speaker 5: I guess when we when we say that the measurable
• 20:30
  Speaker 5: contribution to performance lags the level of investment, I usually
• 20:34
  Speaker 5: see three drivers tom First is that a you know,
• 20:39
  Speaker 5: the basic is no baseline, so people kick off their
• 20:43
  Speaker 5: air projects and programs.
• 20:45
  Speaker 3: I didn't actually measure before AI, like they can't prove after.
• 20:50
  Speaker 5: The Second is you know, for me, the wrong level
• 20:54
  Speaker 5: of measurement, like they sort of measure the tool, but
• 20:57
  Speaker 5: value doesn't live in the tool. It lives across the journey, right.
• 21:00
  Speaker 5: And Thirdly attribution so AI spread across many surfaces, so
• 21:05
  Speaker 5: unless you're instrument you know the workflow in twind impact
• 21:08
  Speaker 5: gets lost And there's a for me, there's this fourth
• 21:12
  Speaker 5: driver that's underappreciated, and that is an even when you
• 21:17
  Speaker 5: do create real productivity gains, like converting them into clean
• 21:20
  Speaker 5: financial numbers is often hard.
• 21:23
  Speaker 3: Right, So on my.
• 21:24
  Speaker 5: Last program, we saved a huge amount of their seme
• 21:29
  Speaker 5: time because the digital assistant deflected huge volumes of queries
• 21:32
  Speaker 5: away from them, but their sme responsibilities at on top
• 21:36
  Speaker 5: of their day job. Right, so the benefit didn't show
• 21:39
  Speaker 5: up as hard financial savings.
• 21:41
  Speaker 3: Right.
• 21:41
  Speaker 5: It reduced a fractional slize of workload at freed capacity,
• 21:45
  Speaker 5: improved responsiveness, all of which is super valuable. But it
• 21:50
  Speaker 5: kind of boils down to capacity leave release. Sorry, So
• 21:53
  Speaker 5: the ROI question becomes, how do we turn that release
• 21:57
  Speaker 5: capacity into measurable financial value, which isn't always easy.
• 22:03
  Speaker 3: I think in.
• 22:05
  Speaker 5: Terms of obviously, leaders or ultimately want the financial story,
• 22:09
  Speaker 5: and I think the way to get a credible one
• 22:11
  Speaker 5: is to start with the operational leavers. The AI can
• 22:16
  Speaker 5: actually move right time, quality, risk, employee effort. I think
• 22:20
  Speaker 5: if if AI reduces cycle time, it cuts rework, it
• 22:24
  Speaker 5: lowers escalations, and it makes work, you know, feel simpler,
• 22:27
  Speaker 5: then kind of the money follows after. So you know
• 22:33
  Speaker 5: where for me, where OROI usually lands is in a
• 22:36
  Speaker 5: handful of places. Right, capacity releases we just discuss I
• 22:41
  Speaker 5: think cost avoid it, so fewer tickets, few escalations, less
• 22:45
  Speaker 5: dependency on scarce experts, risk and loss reduction so a
• 22:51
  Speaker 5: fewer errors and remediation and sometimes service or revenue uplift.
• 22:55
  Speaker 5: I think when you know sort of responsiveness and quality improves,
• 22:59
  Speaker 5: and you know, some organizations do get to hard costs
• 23:02
  Speaker 5: out you know, contract or reductions or headcount, but I
• 23:06
  Speaker 5: think that only really happens when the gains are really
• 23:08
  Speaker 5: concentrated and they redesign roles and processes. Otherwise, I think
• 23:13
  Speaker 5: the sort of first wave of AI is usually focused
• 23:17
  Speaker 5: around capacity and avoidance. So, I guess wrapping it up,
• 23:23
  Speaker 5: it's got a long winded answer, but the key question
• 23:25
  Speaker 5: for me isn't are people using AI? It's is the
• 23:29
  Speaker 5: work measurably better? Have we decided, you know, how will
• 23:33
  Speaker 5: convert that improvement into value? And that's where you know,
• 23:37
  Speaker 5: experienced data really matters because it tells you whether AI
• 23:40
  Speaker 5: is genuinely removing friction or if it's just you know,
• 23:44
  Speaker 5: shifting effort into checking, reworking extra steps.
• 23:49
  Speaker 2: And does that align with another concept in your writing
• 23:52
  Speaker 2: between the shallow and dp AI usage in is that
• 23:56
  Speaker 2: what aligns there with in terms of shallow usage which
• 23:58
  Speaker 2: is just usage, and deep usage, which is valuable usage.
• 24:04
  Speaker 3: I wouldn't.
• 24:04
  Speaker 5: I'm not sure i'd phrase it exactly like that, I
• 24:07
  Speaker 5: guess for me, So let's let's do an example. Think
• 24:11
  Speaker 5: about it support, so an it support. I think of
• 24:15
  Speaker 5: shallow versus deep I AIS assist versus transform. So shallow
• 24:22
  Speaker 5: AI kind of makes existing process faster. It sort of
• 24:25
  Speaker 5: summarizes a ticket raster response, you know, points you to
• 24:30
  Speaker 5: the right knowledge useful, but the workflow is basically the same.
• 24:35
  Speaker 5: Dep AI for me kind of changes or transforms the flow,
• 24:39
  Speaker 5: so it starts to removed steps and handoffs away from humans, right.
• 24:44
  Speaker 5: It classifies and roots incidents, and it pulls the round
• 24:48
  Speaker 5: diagnostics automatically, maybe proposes a resolution path with sources and
• 24:55
  Speaker 5: in you know, really tightly governed cases, it can trigger
• 24:58
  Speaker 5: you know, an improved automation with you know, human checkpoint.
• 25:02
  Speaker 5: I think on the on the employee side, DPA I
• 25:05
  Speaker 5: can genuinely deflect tickets by guiding self service, verifying, fix escalating,
• 25:10
  Speaker 5: you know, only when the journey is truly blocked. And
• 25:15
  Speaker 5: I guess the important thing is is what that means
• 25:17
  Speaker 5: for leadership. But for me, the leadership difference is huge.
• 25:21
  Speaker 5: You know, shallow AI it kind of I'm simplifying, but
• 25:25
  Speaker 5: it mostly needs kind of onboarding and norms. DPAI needs
• 25:29
  Speaker 5: an operating model. It's clear guardrails. It needs ownership and
• 25:33
  Speaker 5: needs auditability. It's quality controls, right, because you're essentially delegating
• 25:39
  Speaker 5: parts of judgment and action, not just speeding up writing.
• 25:43
  Speaker 5: So and I think you know through a dex lens
• 25:47
  Speaker 5: as you go deeper, think you need visibility into where
• 25:52
  Speaker 5: journeys break, right, So retries escalations tool hopping right, so
• 25:57
  Speaker 5: that you you can tune the workflow and rails and
• 26:00
  Speaker 5: not just push adoption cons Very cool.
• 26:05
  Speaker 2: Yeah, I like that distinction a lot, particularly as it
• 26:07
  Speaker 2: hits it right, Like you know, there's there's just a
• 26:10
  Speaker 2: superficial of a deep internal transformation of this department among
• 26:14
  Speaker 2: so many others looking at AI journeys and the genders.
• 26:19
  Speaker 2: Coming back to your your real world, real world experience. Here,
• 26:23
  Speaker 2: can you give an example of an organization doing AI
• 26:27
  Speaker 2: adoption well and what this looked like in practice as
• 26:30
  Speaker 2: as opposed to in theory.
• 26:33
  Speaker 5: Sure, so, I guess real world is orefull without naming
• 26:37
  Speaker 5: the organization is by confidential they can't speak. Confidentiality matters,
• 26:43
  Speaker 5: but I think the patterns are widely applicable. So I
• 26:46
  Speaker 5: led an enterprise knowledge discovery program inside this global consulting
• 26:51
  Speaker 5: environment where you know the core journey was simple, right,
• 26:54
  Speaker 5: is I need a reliable answer fast from the organization's
• 26:58
  Speaker 5: global knowledge. And the underlying problem wasn't the model, it
• 27:03
  Speaker 5: was fragmentation. You know, knowledge and guidance lived across dozens
• 27:07
  Speaker 5: of systems and tools. So the employee experience was essentially search,
• 27:12
  Speaker 5: ask around, you know, wait a while for an answer,
• 27:16
  Speaker 5: and then hope that you've kind of found the latest
• 27:18
  Speaker 5: and greatest correct version of the truth. And you know what,
• 27:24
  Speaker 5: I think what worked really well was treating it as
• 27:27
  Speaker 5: a journey, not a feature. So you know, we defined
• 27:30
  Speaker 5: what good meant for that journey for different user archetypes.
• 27:34
  Speaker 5: We put foundations in place, taxonomy, single source of truth, governance,
• 27:39
  Speaker 5: clear operational ownership. Critically, we were relentless around accuracy, and
• 27:46
  Speaker 5: because going back to that trust issue, the last thing
• 27:48
  Speaker 5: you want is you know, you you switch the tool
• 27:52
  Speaker 5: onto users, they go and use it and the first
• 27:54
  Speaker 5: answer they get is evidently very very wrong.
• 27:58
  Speaker 3: Right.
• 27:58
  Speaker 5: So absolutely the relentless around answer accuracy before going out
• 28:04
  Speaker 5: to a broad user base, and we designed around real
• 28:09
  Speaker 5: user patterns, right gathered through voice of employee and interviews,
• 28:13
  Speaker 5: and then we we essentially ran a super tight improvement loop.
• 28:17
  Speaker 5: So ongoing quality evaluation, feedback capture and managed you know,
• 28:24
  Speaker 5: managed backlog, so that essentially the service kept getting better
• 28:28
  Speaker 5: and better after go like that, that continuous improvement loop
• 28:34
  Speaker 5: and throughout you know, we go back to human centered, right.
• 28:37
  Speaker 5: We applied this human centered lens. We when we were
• 28:41
  Speaker 5: designing and testing. We did so with policy SMEs with employees,
• 28:46
  Speaker 5: and I think what worked really well for me sort
• 28:50
  Speaker 5: of user based perspective is that we included both skeptics
• 28:54
  Speaker 5: and sort of keen innovators if you like. So you
• 28:57
  Speaker 5: had both ends of the of the adoption curve, and
• 29:03
  Speaker 5: we balanced speed and risk and secured you know, key
• 29:07
  Speaker 5: learnings including changing adoption earnings through controlled expansion so alpha,
• 29:13
  Speaker 5: small user base, Beta much bigger user base, through to
• 29:17
  Speaker 5: a full country rollout, and then into into global rollout.
• 29:21
  Speaker 5: So you know, success wasn't wasn't measured as usage alone.
• 29:25
  Speaker 5: We looked for repeat the usage, shifting the tone of
• 29:28
  Speaker 5: user anecdotes, you know, measurable reduction in query volumes to sermease,
• 29:33
  Speaker 5: amongst other things, and in other words, you know, whether
• 29:36
  Speaker 5: it was genuinely removing effort or or not all but
• 29:41
  Speaker 5: essent you to turn on whether it was generally removing
• 29:43
  Speaker 5: effort and making work feel simpler for the end user.
• 29:47
  Speaker 5: So I guess summarizing the playbook, we pick one or
• 29:52
  Speaker 5: two journeys right with obvious pain map that current state,
• 29:55
  Speaker 5: align on what the target experience needs to be, fix
• 29:59
  Speaker 5: knowledge and trust, f instrument experience and outcomes, and then
• 30:03
  Speaker 5: keep tuning quality and workflow fit as an ongoing capability.
• 30:09
  Speaker 5: And you know that's what really turned you know that
• 30:11
  Speaker 5: this this AI into a dependable way of working rather
• 30:14
  Speaker 5: than kind of you know, string and pilots if.
• 30:16
  Speaker 2: You like, a hundred percent, And this is this is
• 30:20
  Speaker 2: a personal curiosity of mine. Right with regards to AI,
• 30:25
  Speaker 2: over use is usually framed in terms of, you know,
• 30:29
  Speaker 2: I guess it's a risk issue. You know, in cautious
• 30:33
  Speaker 2: kind of usage is what your mind usually goes to.
• 30:35
  Speaker 2: But is there any perception or should there be about
• 30:38
  Speaker 2: the danger of overuse in terms of quality of work,
• 30:42
  Speaker 2: cognitive load, even dependency of whether that gets perhaps a
• 30:46
  Speaker 2: bit more niche I suppose my personal view, Anthony, is
• 30:50
  Speaker 2: that there's always a slight threat that with AI transformation,
• 30:57
  Speaker 2: there's always a threat of mediocrity in the way in work.
• 31:02
  Speaker 2: And is that something that you feel that like corporate
• 31:04
  Speaker 2: leadership is aware of or should be aware of, and
• 31:07
  Speaker 2: is there anything that can be done about that from
• 31:09
  Speaker 2: an adoption perspective.
• 31:11
  Speaker 3: Yeah, that's a that's a super super interesting question.
• 31:14
  Speaker 5: It's I'll tell you why it's interesting because I don't
• 31:17
  Speaker 5: think it's it's very much on the I don't think
• 31:22
  Speaker 5: it's on the minds of leaders. I think everyone everyone
• 31:26
  Speaker 5: is generalization, but but I think most organizations are so
• 31:30
  Speaker 5: they're so focused on how on earth do we make
• 31:33
  Speaker 5: this ai investment successful, how do we get people using it?
• 31:36
  Speaker 5: How do we drive that adoption? But actually, you know,
• 31:40
  Speaker 5: the question just ask is kind of what falls out
• 31:42
  Speaker 5: of the back end once you're successful, And a lot
• 31:45
  Speaker 5: of organizations aren't that mature yet, So that't me give
• 31:49
  Speaker 5: you my perspective. I think that the simple answer is, yeah,
• 31:53
  Speaker 5: I think there is a there is a case for
• 31:54
  Speaker 5: monitoring aiovius, but only if you're like crystal clear about
• 32:00
  Speaker 5: what you're monitoring for, and you do it in a
• 32:02
  Speaker 5: way that protects trust rather than eroding it. That won't
• 32:05
  Speaker 5: surprise you give me earlier points that they're made around trusts,
• 32:08
  Speaker 5: So you know, overuse, I guess what what does overuse
• 32:13
  Speaker 5: manifests itself. As you know, I think it's it's not
• 32:17
  Speaker 5: just oh, people are using AI a lot, right, I
• 32:20
  Speaker 5: think overuses when usage starts to create this sort of
• 32:23
  Speaker 5: downside risk. Right, So I guess three three obvious things
• 32:28
  Speaker 5: off the top of my head versus quality drift. Right,
• 32:30
  Speaker 5: people kind of accept low quality outputs. Right, critical thinking
• 32:34
  Speaker 5: from a humor perspective drops and therefore errors and rework rises. Secondly,
• 32:42
  Speaker 5: maybe hidden workload. Right this point I made earlier around
• 32:45
  Speaker 5: this sort of AI time tax. You know, if that
• 32:48
  Speaker 5: if that time tax is quietly expanding in the background
• 32:52
  Speaker 5: and people, you know, you're not getting the efficiency out
• 32:55
  Speaker 5: of that A fight a AI sorry.
• 32:56
  Speaker 3: Because people are essentially.
• 33:00
  Speaker 5: Going through twenty prompts to get an answer, and then
• 33:03
  Speaker 5: they're checking in and then they're quite believing it, and
• 33:05
  Speaker 5: they're validating it, and they're fixing it and going back
• 33:07
  Speaker 5: into it again, and then and then they're copying and
• 33:09
  Speaker 5: pasting it into the you know, the system that they're
• 33:12
  Speaker 5: actually using. That could, I guess become a downside risk.
• 33:19
  Speaker 5: And then the last one is dependency. So it was
• 33:23
  Speaker 5: super interesting one for you know, from a human perspective,
• 33:27
  Speaker 5: is people just you know, the gray matter starts failing,
• 33:31
  Speaker 5: a big will lose the capability or stop learning the
• 33:35
  Speaker 5: underlying skill. So I guess those are super real, But
• 33:41
  Speaker 5: I you know, I would say, if you are going
• 33:44
  Speaker 5: to go down that monitoring route, it has to be
• 33:46
  Speaker 5: handled super carefully.
• 33:47
  Speaker 3: Right.
• 33:47
  Speaker 5: If it feels like surveillance to your employees end users,
• 33:52
  Speaker 5: trust will collapsed and people would just work around it.
• 33:57
  Speaker 5: I think the healthy approach is to essentially define what
• 34:00
  Speaker 5: good looks like by roll and by task, and then
• 34:03
  Speaker 5: you know, monitor at the journey level. You know where
• 34:06
  Speaker 5: is effort rising, where are people bouncing between tools where
• 34:10
  Speaker 5: self service fails and escalations spike, rather than sort of
• 34:14
  Speaker 5: policing individuals. But there's an really interesting angle that I
• 34:20
  Speaker 5: discovered in my last program, and that is safeguarding. Right,
• 34:26
  Speaker 5: So we found that some people, maybe all people, will
• 34:30
  Speaker 5: disclose distress more readily to an AI assistant than to
• 34:34
  Speaker 5: a colleague. And you know that creates this really significant
• 34:39
  Speaker 5: duty of care when you're designing your AI. So take
• 34:44
  Speaker 5: it to the stream. What if an employee tells the
• 34:47
  Speaker 5: bot that they're they're having suicidal thoughts, You know you
• 34:52
  Speaker 5: need the assistant to respond appropriately both in terms of
• 34:57
  Speaker 5: you know, the content, but also the tone of voice.
• 35:01
  Speaker 5: And you need clear, you know, pre agreed trigger conditions
• 35:05
  Speaker 5: for when you know a trained human is alerted, and
• 35:08
  Speaker 5: you need to be really really thoughtful and specific as
• 35:11
  Speaker 5: to who that human is. You know, for example, if
• 35:14
  Speaker 5: in that example, I don't think it should be the
• 35:17
  Speaker 5: manager of.
• 35:18
  Speaker 4: It, it should be well bringing in despondent someone's who
• 35:23
  Speaker 4: trained in these things. So yeah, I digress a little,
• 35:26
  Speaker 4: but it's you know, it's important, and it's Yeah, it's
• 35:30
  Speaker 4: a key aspect of that monitoring perhaps a few people.
• 35:34
  Speaker 3: Really think about.
• 35:36
  Speaker 5: So I guess to wrap that point out around monitoring, Yeah,
• 35:39
  Speaker 5: definitely a case for it. I think the key is
• 35:41
  Speaker 5: designed transparency. Transparently explain what is and isn't monitored, protect privacy,
• 35:48
  Speaker 5: and make sure that you know escalation routes are a safe,
• 35:51
  Speaker 5: proportionate and governed.
• 35:54
  Speaker 2: I think you've touched upon so many interesting things in
• 35:57
  Speaker 2: this conversation, Anthony, and you've helped us help their post
• 36:02
  Speaker 2: tim Ra on the Deck show go off with a bang.
• 36:05
  Speaker 2: Really really appreciate it. Great to have you back on
• 36:07
  Speaker 2: and let people know where they can reach you, reach
• 36:11
  Speaker 2: out to you and read your very interesting articles as well.
• 36:15
  Speaker 5: Yeah, sure well, feel free to contact me Violin tain
• 36:18
  Speaker 5: That's probably the best route, but thank you for reviving
• 36:21
  Speaker 5: me back. I always love these conversations.
• 36:23
  Speaker 2: See likewise Likewise, thanks so much.
• 36:26
  Speaker 3: Take care of yourself.
• 36:27
  Speaker 6: Thanks up to make sure that you never miss an episode.
• 36:30
  Speaker 6: Subscribe to the show in Apple Podcasts, Spotify, or your
• 36:34
  Speaker 6: favorite podcast player, And if you're listening on Apple Podcasts,
• 36:37
  Speaker 6: make sure to leave a rating of the show. Just
• 36:39
  Speaker 6: tap the number of stars you think the podcast deserves.
• 36:42
  Speaker 1: If you'd love to learn more about how next Thing
• 36:44
  Speaker 1: can help me improve your digital employee experience, head over
• 36:48
  Speaker 1: to next think dot com. Thank you so much for listening.
• 36:51
  Speaker 6: Until next time,