Latest / Project Podcast / How AI is changing projects right now

Transcript

How AI is changing projects right now · Project Podcast

0:08
Welcome to the APM podcast. APM is the childhood body for
0:11
the project profession. My name's Emma DaVita.
0:14
I'm the editor of Project APM's quarterly journal and your host.
0:18
In this podcast, I'm speaking to Daniel Armanios, BT, Professor
0:21
of major programme Management at Oxford University's SCII
0:24
Business School, and Zach Swafford, Co founder of Dot, a
0:28
San Francisco based startup. He's a project management tool
0:31
uses AI to assist with brainstorming, road map planning
0:34
and report generation, among many other tasks.
0:38
We thought it's the perfect time to cast Sari over the rapid
0:41
developments in AI over the past year and speculate on what 2025
0:45
might hold with two guests with two different perspectives on
0:48
the world of work. Listen on to hear their thoughts
0:51
on the exciting potential of AI, their worries about the
0:54
technology, and their advice on what you should be doing to
0:56
practically get to grips with this work revolution.
1:00
What's really valuable, I think, is their insights into how AI is
1:03
being used in projects right now.
1:05
So do listen on. I'd like to welcome both of you
1:12
to the APM Podcast. Thanks for giving us your time
1:14
today. Let's begin by finding out a bit
1:17
more about your work and your area of interest in AI.
1:20
So perhaps, Daniel, do you want to tell us a bit about your work
1:23
and your area of interest when it comes to AI and project
1:25
management? I'm Daniel Armanios.
1:28
I'm the BT Professor and Chair of Major Programme Management
1:32
here at the Site Business School, University of Oxford.
1:35
I think my work is at the intersection of civil
1:39
engineering and sociology. So essentially I'm interested in
1:42
how organisations coordinate to underpin really big scale
1:46
initiatives. And so my interest particularly
1:49
around the AI facets, is thinking about where do you
1:53
deploy what kinds of algorithms in the organisation.
1:58
And so when you have a project with several organisations,
2:01
several stakeholders, where best to place what kinds of
2:04
algorithms. And so I've been spending much
2:05
more time on the organisational design programme, project design
2:11
kind of perspective to look at where you place these kind of
2:15
algorithms, knowing that AI is not all one thing, right?
2:18
It's been multiple things. And so that's where I've been
2:20
spending a lot of time. And the second is where is the
2:25
assistance of AI really good at processing a lot of information,
2:29
but also where do we need to build guard rails and caution
2:33
around the hallucinations? The ability for AI to shape the
2:37
informational landscape in ways that might not be fully
2:40
revealing of the truth. And so take its benefits, but
2:43
also be mindful of its places where it can go off, off the
2:47
rails, if you will. Thanks, Daniel.
2:49
So by what you're talking about, do you do you tend to focus on
2:52
mega projects, bigger projects? Usually large scale projects,
2:56
Yeah, usually large scale projects.
2:58
But I think often how it's being deployed in smaller projects can
3:02
be a harbinger of what could happen at scale in a larger
3:06
initiative. Is that how it tends to work out
3:09
in the real world, that that organisations will will
3:12
experiment with a smaller project and naturally that will
3:16
scale up or not depending on how successful it was or lessons
3:19
taken from that and shared? I mean, I'd be, I'd be
3:23
interested in Zach's corporate experience around this in terms
3:26
of what they're doing with this, the start up at Dart.
3:28
But I think what I would say is, is that ideally that's what
3:32
you'd like, like some experimentation on a sandbox.
3:36
But I think matching with, I think this year was, was really
3:41
a year of, Oh my gosh, we need to get on this.
3:43
And so what's happened, there's a lot of organisations that
3:46
feel, Oh my gosh, we just have to get everyone to do it.
3:47
And no matter what, deploy it. And so you would hope it's done
3:52
with strategic intense, you know, sandboxing, etcetera.
3:56
But I think there is a bit of, Oh my gosh, we're behind the
3:59
curve, let's just deploy it. We need to show that we're doing
4:01
something with AI. And so in some cases they're
4:05
just experimenting and you're seeing this with AI itself.
4:08
I mean, a lot of these products are being disseminated without
4:12
necessarily full on validation whereby it's like what we're
4:15
going to know what's the issues or know what is by people trying
4:18
it. And so I mean, even open AI when
4:19
they launch ChatGPT, other things, it would just let's just
4:22
see what happens. And you know, there's different
4:24
perspectives as whether that's OK or not, but I think it's a
4:27
mix of both, right? Some are sandboxing, trying
4:29
this, figuring out the kinks where others are just like,
4:31
let's just get it out there and see what happens.
4:34
Each has their pros and cons, if you will.
4:37
So a little bit messy in real life.
4:39
Yeah, unfortunately. Thank you, Daniel.
4:43
Zach, tell us about about your work and a bit about your
4:46
experience with AI. My name is Zach Swafford and I
4:49
am a Co founder at a start up called Dart.
4:53
We do project management with AI, so pretty relevant and we
4:59
got started a couple years ago basically because we saw in, we
5:04
saw a big sea change here basically in the project
5:07
management space with AI. We saw this coming a little bit
5:10
and wanted to build really from the ground up a new project
5:14
management tool that would incorporate AI and really
5:19
specifically when I say I, I mean LLMS on into every facet of
5:26
the project management experience.
5:28
Can you spell out what an LM is just for people listening?
5:31
You might not know. So an LLM is a large language
5:35
model and this is something like Jet GPT or Gemini is a Google
5:40
One Co pilot and Microsoft Claude is another one.
5:43
Those are some of the biggest models and those the idea is
5:47
that they use a lot of language, a lot of text and other
5:53
information from the Internet to create a chat like experience.
5:57
But really thinking beyond chat. I think in the longer run, these
6:01
are the models that will act more like a like a real AI like
6:06
a real Jarvis or whatever you want to think about.
6:11
You know, these these ideas of big AI models that can do almost
6:16
anything. These LLMS are the technology
6:19
that starts to enable that more so than traditional machine
6:23
learning, which can be great at answering specific questions and
6:27
is an excellent tool for lots of things.
6:30
But the LLMS are the side that I am the most excited about right
6:34
now. So there's lots of interesting
6:36
stuff we can talk about there, get into that.
6:38
We have tonnes of really interesting features in Dart.
6:43
But to answer sort of the question of how are, how are
6:49
organisations building up to this?
6:52
I've seen it, I've seen it all sorts of different ways.
6:54
I've seen, like Dan was saying, really diving right into a big
7:00
project and kind of jumping off the deep end, which can be
7:03
risky. Of course.
7:04
I've also seen really successful implementations where folks are
7:09
trying something on a smaller project and building up to it.
7:11
And I think there are there are pros and cons and there are also
7:16
different levels of risk that different organisations are able
7:20
to accept or willing to accept. And depending on the industry.
7:24
At DART, we work with a lot of different industries.
7:26
And I always say that some of our marketing agencies, right,
7:32
can accept a higher level of risk and a higher level of
7:36
incorrect answers or hallucinations or whatever than
7:41
a team that might be working in healthcare or, or construction
7:46
or something like that. So that's another perspective
7:49
on, you know, maybe different industries can move at different
7:51
speeds here. As well.
7:53
And Zach is have you had a background in project management
7:56
before? Yes.
7:58
Great question. Yeah, sorry, I should have,
8:01
should have answered the background question.
8:02
So, yeah. So I studied AI actually
8:05
specifically at Stanford a few years ago and then I went into a
8:10
company called Plenty and we did indoor farming.
8:14
So at Plenty and I worked on the hardware and software sides.
8:18
There are LED teams in AI software and hardware actually.
8:22
Would you mean by Intel farming? Is that literally what?
8:25
Oh yeah, what do you mean by that?
8:26
Absolutely. It's what it sounds like, Yeah.
8:29
So. Farming inside.
8:31
That's right. Yeah.
8:31
So you can think of it like a big automated greenhouse.
8:35
I think that's the simplest way to think about it.
8:36
We grew leafy greens, lettuce and arugula, kale, lots of fancy
8:43
herbs and the indoor farming process is.
8:47
The big idea was using automation and AI to really have
8:53
the perfect environment for plants and grow them really fast
8:57
and well and healthily. And when you have an indoor
9:00
farm, you can also place it right next to where it's going.
9:03
So you also don't have to deal with the supply chain logistics
9:07
and you get a lot of fresher food.
9:08
So yes, very interesting work and, and plenty in particular
9:14
was a really fast growing company.
9:16
So we, we grew a tonne and that meant that we had all sorts of
9:20
scaling problems, all sorts of project management needs.
9:23
So I filled the gaps in engineering management, but also
9:27
in implementing project management, both for software,
9:31
which is really my original background, and then also even
9:36
building Gantt charts and helping the construction and
9:39
then the builds we were, we were another startup.
9:42
So there was just a lot going on and, and I filled some of the
9:45
gaps there. And that's really how I got
9:47
started with project management formally.
9:50
OK, well that's really interesting.
9:52
So you 2 have very different perspectives.
9:56
Daniel Moore, the organisational take in the bigger picture and
9:58
Zach, really nitty gritty of making this work day-to-day.
10:03
I think there's the thing that I'd really like to hear about is
10:07
looking over 2024 comparing where the world is now compared
10:12
to a year ago. What would you say have been the
10:15
the biggest developments when it comes to AI and project
10:18
management specifically? Zach, maybe on the ground, what,
10:23
what would you say? Jump out at you?
10:27
I think that in the past year I've seen of new tools that
10:31
allow people to chat directly with their projects.
10:35
So this means asking questions about are there any blockers?
10:40
Are there what, what are the, what are the critical path in
10:43
this timeline? Those sorts of things.
10:45
Or sometimes the questions that project managers or potentially
10:50
even other stakeholders might formulate and then might want to
10:53
ask, but maybe would need to go through.
10:58
You need to have a bunch of messages about, about that and,
11:01
and go through a whole process to figure those sorts of things
11:04
out. Those are the kinds of questions
11:06
that if you have your project management tool or your system
11:11
in place to have all that data in good shape, then you know a
11:17
random stakeholder would be able to just ask the question what's
11:22
our critical path right now? What's our timeline and get a
11:25
chat based answer, which is particularly nice actually for
11:29
non-technical folks or people that aren't steeped in the
11:32
project management world like we are.
11:34
OK. That's great.
11:36
Daniel, you agree with Zach. Are there any other developments
11:39
that you've noticed that have been on your radar?
11:42
Definitely one of the trends I see very much in agreements with
11:45
Zach. So the ability to take large
11:50
amount of corpus of text because if you think of projects, they
11:53
fundamentally produce a tremendous amount of language
11:58
data, text data. And So what DART is doing other
12:03
organisations and plan others, they're really enhancing
12:06
drastically the processing capability.
12:10
So I mean, you can it's get, it's quite revolutionary project
12:14
world. I mean, the idea that you can
12:15
take a request for proposals, run it through these LLMS and
12:20
come up with an addition, an initial Gantt chart, a racy
12:25
diagram just from the data to be able to ideate things that you
12:29
know are doable, but usually took, you know, burned a lot of
12:33
time. And if you think of projects,
12:35
often the issues are like death by 1000 cuts.
12:38
You know, if you can save 30 minutes here, an hour here doing
12:41
this, the fact that you can see critical pass in your data run
12:45
last planner models. And this is where the Omni
12:47
models get into the way you can ensemble different models
12:50
together. Perhaps even now there's a lot
12:53
of interesting trends on, you know, reference class
12:57
forecasting, benchmarking, for example, right, how you come up
13:01
with the reference class, There's 20,000 different ways
13:03
you can come up with a reference class that can be done back end
13:06
automatically with some of these generative AI large language
13:10
models. So that's one dimension I think
13:12
is really interesting. The ability to take all of that
13:16
data and make sense of it, that would take hours, months of
13:21
really specialised time. And even if it's not perfect,
13:25
just getting you started, you've already saved an infinite amount
13:28
of time. So the ability to ideate on this
13:31
is, is is really revolutionary. I think where another trend is
13:35
happening is the visual turn. So there's now models that can
13:39
translate what you're doing or what you're thinking in text
13:42
into image. So there's some organisations,
13:46
some companies like Icon for example, that they, what they're
13:50
planning to do is take your ideas and then translate them
13:53
into initial construction documents.
13:56
That's an amazing turn and then schedule what would look like
14:00
that from that. So I think the visual turn, even
14:02
using video to do this, be able to OK, I have this idea.
14:07
Let me generate a video of what this could look like and immerse
14:10
someone in it. So the connections between large
14:13
language models and visual or even video motion kind of
14:17
imagery that could then be tied into immersive reality, other
14:21
things. So it's not just the text and
14:24
the visual, but the modality is really quite fascinating and
14:27
really exciting because especially at the early stages
14:30
where if you could plan and see much more of the information
14:33
around you both in Word but also image, that could save so much
14:37
cost at the back end because you've already kind of stress
14:40
tested. Red teamed a lot of different
14:42
scenarios that would have taken a lot of costs to do.
14:46
Because if you have to do that without this kind of assistance,
14:50
the issue becomes I have to then think, is this time to build
14:54
this scenario, to run this chart, to run these matrices?
14:59
Is that really worth the time to do so?
15:00
Then you have to spend a lot of time being very careful about
15:03
where we're going to search more, where we're going to plan
15:05
more. And with these tools, you can
15:09
run many more trials than you would otherwise.
15:12
And it has huge implications. I think it's just really
15:15
exciting to see that strain both in the text base term large
15:19
language models, but also now the different modalities, text
15:23
to visual. And we could talk about what I
15:25
think is happening more recently, which is even the
15:29
ability to train on really pretty interestingly accurate
15:32
synthetic data too. So it's not just looking at your
15:36
own prior project data and seeing it running synthetic data
15:41
that because of how they're trained, it could be quite
15:44
accurate. It may not be, but at least as a
15:46
starting point could be quite fascinating.
15:48
I mean, for example, in spatial computing, which is a lot of the
15:51
back end Nvidia's now training, you know, if you have a project
15:55
on autonomous vehicles or other things, they're actually
15:58
training on synthetic data because the lighting and stuff
16:02
now is so good. So autonomous vehicles, for
16:05
example, are being trained on synthetic data to some degree.
16:08
We get into the final one, I would say, and then I'll move
16:12
on. Is this.
16:12
So we have we talked about textual, we talked about visual,
16:15
but also the ability now to artificially or with AI create
16:21
kind of interactions of social interactions between people.
16:24
So we can take the next step and say if these different project
16:27
manager are working together, how would their interaction look
16:30
like? What pitfalls can we worry about
16:32
simply by embedding rules into simulated managers or agents?
16:37
This would be called deep reinforcement learning.
16:39
And they're remarkably adapt. So for example, they can run
16:44
scenarios like what would you know, birthday parties and who's
16:47
invited? I mean, the social interactions
16:48
you can do with just very basic rules.
16:50
So I could see in the future, and even now it's happening, run
16:54
different managers with very basic realistic assumptions what
16:57
would happen when they work and interact in this way over a few
17:00
months, what could be, what could get, what pitfalls could
17:03
we worry about, what not? And you could do that all at the
17:06
early stage. So it's really, I mean we're
17:09
only scraping the very initial surface and I think that's why
17:14
Dart and plan icon. These are just going to grow
17:18
because of their ability to really ride that initial wave.
17:22
And so with that comes amazing power.
17:24
And also, you know, we could discuss the perils later, But in
17:28
summary, I think three things. One is the massive ability to
17:31
read large scale text, the text to visual and motion right, It's
17:38
like video and so forth. And then the third one is even
17:40
to simulate what interactions of individuals in these projects or
17:44
large scale programmes or even small one could look like.
17:47
Is it realistic? Not necessarily, but it is at
17:51
least a really good way to ideate in very different ways at
17:55
scale. So what you're explaining there
17:58
sounds very exciting and we're on the cusp of big things.
18:01
I'm just, I've, I haven't heard anyone talk about the, the kind
18:05
of project manager interfacing with another project manager.
18:09
Do you mean that you could get AI to understand the characters
18:14
of individual people and how they might work together?
18:17
Is that what you're saying? Yeah, I think so.
18:20
So there is, and we can talk about the integration of how
18:24
these different algorithms work. But one set of algorithms are
18:28
what's known as deep reinforcement learning.
18:30
So essentially what happens is you populate with some very
18:34
basic rules so you can look at your prior work as a project
18:38
manager, otherwise say what are kind of heuristics, rules of
18:41
thumb or ways we operate that we can embed into an agent, right?
18:47
And then you say, and then how you, the learning aspect is when
18:51
it does really well, you say, yes, that's great.
18:54
When it kind of goes off the rails, you say, no, no, no,
18:56
that's not what I'm looking for. And as you train these agents
19:00
with these rules and let them kind of code another and deal
19:04
with the data, you could potentially simulate
19:06
interactions. There's some interesting work.
19:08
It's funny, Zach mentioned Stanford.
19:10
I graduated from Stanford as well.
19:12
There's a really interesting study at a basic level out of
19:14
Stanford where they developed this kind of, I don't know what
19:18
you call it, like a kind of, it's almost gamified, but it's
19:21
an interaction called Simorca. And essentially what they do is
19:24
they embed agents with kind of rules.
19:27
Now in that case, it's like we're hosting parties or general
19:29
social interaction. But if you use your insight and
19:33
acumen as a project manager to understand here's how I break
19:36
down sets of work, here's how I interact with with suppliers,
19:40
etcetera, you can simulate that interaction.
19:42
You can even build avatars to practise.
19:44
So for example, let's say I'm going to have a difficult
19:48
conversation with a supplier, but I want to practise what that
19:52
would be like. You could interact with an
19:54
avatar who has these kind of rules to help you train for this
19:58
difficult conversation I'm going to have about procurement.
20:01
Right. So the, the interaction ability
20:03
and that's only just started maybe within the last couple of
20:06
months, I would say in the project management space.
20:08
It's been longer in the CS space, but you're starting to
20:12
see that ability in the project management very recent.
20:16
So it's not just taking my data, rendering it, visualising the
20:19
Ida, but even practising how we even do the work is possible now
20:25
I think. OK, So what I think I'd like you
20:29
to do is if I take you away from from the, you know, what ifs and
20:34
the excitement about what could happen to actually bring us back
20:37
to how AI is becoming a facet of work life and projects these
20:43
days. So kind of practical examples.
20:48
So there are honestly lots of stories of success that I'm
20:53
familiar with. And I think one place to start
20:59
is where the newest technologies are at their best, where they
21:04
really shine and thinking about how that can be applied to PM as
21:09
a discipline. So one of those Dan just
21:12
mentioned is acting as a smart, I like to say sometimes a smart
21:19
intern almost giving you an 80% version of what you need to.
21:26
And maybe maybe it's almost like a team of interns where you can
21:30
send them out and proactively get all sorts of different first
21:34
stabs at a project or at A at any given task really.
21:38
So one place where I've seen that really successful a lot is
21:42
in initial phases is in brainstorming, ideating, trying
21:47
to come up with an initial plan. A lot of this happens in
21:50
software, maybe more so than construction, but at the
21:53
beginning of a software project we might be thinking about what
21:57
are the different ways we can tackle this.
21:59
Here's a bunch of the user feedback that we have.
22:02
We have a long list of tonnes of documents about how users act
22:06
and behave. And now what we need is some
22:09
solutions, some some ways to tackle that problem.
22:14
There are great tools really tactically, I might just put
22:18
that as a first stab. I would always say just put that
22:21
into Cha Chi PT. Just put all those documents
22:23
straight into Cha Chi PT if you can, if your organisation allows
22:27
that sort of, you know, security wise, but put all that straight
22:31
into the nearest large language model and see what happens,
22:34
right. And, and see, get some of that
22:36
back out. Ask, ask it, Hey, can you help
22:40
me solve some of these problems? Can we ideate about how to solve
22:44
some of this? And probably what you'll find is
22:47
that some of the answers are good, some of the ideas are
22:49
good. A lot of them lack the context
22:51
and awareness of your organisation, of your practises
22:54
of kind of what what you actually need to do.
22:57
You'd be surprised sometimes how that's not totally necessary.
23:00
But in a lot of cases, then you might need to iterate and start
23:04
to provide more of that context and start to think about more
23:07
advanced tools that would allow you to include that context.
23:13
So that's one way. Another way I would say is,
23:17
yeah, I mean, I guess I think an important thing that I wanted to
23:21
emphasise was figuring out the ways to work with your
23:24
organisation to have the right security posture to allow you to
23:28
use those tools. Because it is really important
23:32
that people at all levels of an organisation are able to
23:35
experiment with this stuff, in my opinion.
23:37
And there are ways now to allow you to not trained on any of
23:42
that data, right? You can get an agreement with
23:44
open AI to originally this was not the case.
23:49
And so people were worried about providing information to LLMS
23:52
and there was kind of a let's let's take this slow sort of
23:56
approach. Now.
23:57
I would say it's a lot easier for anybody to get the correct
24:01
kind of agreements in place to say, OK, our data isn't going to
24:04
be used. We can treat this more like any
24:06
other service provider or cloud service provider and let's you
24:09
know, let's get chat be set up in our organisation so that we
24:13
can start experimenting with this.
24:15
Do some of that brainstorming. Do some of that initial get.
24:20
Get some of the initial takes on how we can apply this.
24:24
Provide the documents, provide the context, start to get back
24:27
some of the answers. So it's really about using
24:31
another resource to make connections.
24:34
You might not necessarily. It's like having a whole nother
24:37
team of people in the room who might come up with some really
24:40
terrible ideas, but some of them might be the valuable ones that
24:43
you may never have picked up on in the first place.
24:46
That's right. Yeah.
24:47
And I think that that's an awesome opportunity for or folks
24:52
to take away some of the, the hard work, the brainstorming
24:58
work, the grunt work of just building, building a timeline,
25:01
building a Gantt chart, let's say, outsource that to some of
25:04
these tools. And then free you up as as a
25:10
honestly, as APM contributor at any level really to do some of
25:15
the higher level work. Even if you're junior in an
25:18
organisation. Now if you end up with a team of
25:20
interns that can come up with great ideas for you, then you
25:25
can start to build the muscle of discerning what the right course
25:29
is and then escalating that as your your work essentially.
25:35
OK. And what about a more
25:37
sophisticated level or a more senior level cross projects
25:41
where AI is taken more seriously beyond the kind of initial
25:46
experimentation as you've described?
25:49
Where do you see that happening right now in the real world?
25:54
Yeah. I think there it becomes more
25:57
about, I see a lot of the challenges at that level as more
26:05
interpersonal challenges of getting stakeholder alignment
26:09
and you know, you've got a plan or you've got some candidate
26:12
plans and you need to, you need to collect all the resources and
26:17
get everyone focused around a common goal, a shared goal.
26:22
That's the kind of thing that AI doesn't do as well right now.
26:27
So you can, but there are still ways that you can really apply
26:32
those tools. Daniel mentioned earlier the
26:35
ability to run scenarios right to you can you can say, hey, act
26:41
as this stakeholder. Let's, let's I, I need to
26:45
practise for a stakeholder meeting here where I'm going to
26:48
be delivering some tough news or I'm going to be pitching this
26:52
plan. Help me give me some feedback on
26:56
the ways that I could improve this project proposal, but also
27:00
just my demeanour and the way that I the way that I present
27:04
myself and present this project. That would be a very concrete
27:08
case and and anyone could just have that chat with ChatGPT
27:12
right now to to practise for a meeting.
27:15
You don't need some super hyper specialised tool.
27:18
Once you try ChatGPT and you see where it fails, you can move to
27:23
a more advanced tool for something like this specific to
27:26
the project management domain. But I would encourage everyone
27:30
to just give that a try and see if see if you can learn
27:35
something from it. See if you can get a takeaway
27:37
there that can help you to help even improve your your inner
27:42
human interaction. That's I haven't that's not
27:45
really something that we've covered much before around using
27:48
AI to improve your human skills. That's you always hit a negative
27:53
and how it's taken away from the world of humans.
27:57
But this is absolutely putting that on its head.
28:00
Daniel, I can see you nodding your head there in agreement,
28:03
but what have you seen AI used across projects, programmes over
28:11
the past year that would have been successful?
28:15
The way I generally characterise it, I think Zach is really spot
28:18
on. Maybe maybe to take a
28:21
background, I think it's important to understand how
28:24
these language, large language models work so that you can
28:28
understand what it's useful or not.
28:29
Essentially the best analogy I have is let's just use
28:33
generative AI large language models as the example.
28:36
It's kind of like a stochastic parrot.
28:39
So what does it do? It sees some words in a question
28:43
and it says, OK, I've seen these words Co located with each
28:47
other. I'm going to go to my training
28:49
data and model that I've developed on that training data
28:51
and respond back with something that seems probabilistically
28:56
related, right? So if I asked certain kinds of
28:59
questions, it knows I've seen that question before, I've seen
29:01
these words kind of matched together.
29:03
And so I can give you a response that would probably seem
29:07
similar. Now, why do I say this is you
29:10
have to realise that ChatGPT others have been trained on
29:14
essentially the realm of large scale corpus of data like
29:18
Internet level data. So why do I say this is that
29:21
it's good to start where I've seen it useful is good to start
29:25
for like generalist things to start things that you know are
29:29
very common in project management and are very like, I
29:33
would start with the thing What I've seen is you start with a
29:35
very mundane things that everybody with a project has to
29:39
do, a Gantt chart, a racing matrix, you know, like that
29:43
level and stuff that you have that you're willing to share
29:47
publicly, right? So you start there and you get a
29:51
sense of, and I think that's where the biggest, easiest low
29:54
hanging fruit is mundane, painful things that you have to
29:57
do. You want to have 80% trial like
30:01
Zach said. And then that way I could spend
30:03
my time finessing the details as opposed to building that up over
30:07
hours. I can just start there.
30:08
Where it gets like the added value is now if you have
30:14
something that's your specialist sauce as an organisation.
30:17
Now why did I mention all of this stuff about large language
30:20
models is they're chained on generalist data.
30:23
So now the question is, how do I get it to understand better my
30:27
refined thing? Let's say I'm doing healthcare
30:29
work and I'm looking at a specialised disease just using
30:34
ChatGPT, etcetera. It's going to give me everything
30:39
about medicine, but it's not going to have specialised
30:41
knowledge. And sometimes you want it to
30:42
train on smaller data. So there's two ways Zach brought
30:46
it up about this is that some companies, their large language
30:50
models, you essentially have to ping the server of that model
30:53
and that's how it works. Other ones, you can actually
30:56
download their neural model, bring it into your closed kind
31:01
of space, give it data that's only to you in your closed
31:07
firewalled environment, and essentially take the basis of
31:10
the algorithm existing and train it with more specialised data.
31:14
So now if you're taking the next step and say, well, I get a
31:17
sense of how I want to do this Gantt chart.
31:18
I get a sense of how I want to do the race, like general things
31:20
that just are pain points that are mundane.
31:23
And then I take the next step and say, hey, I want actually
31:27
more targeted knowledge for which we have the secret sauce.
31:31
You can train that model uniquely to your data.
31:36
And so that's called transfer learning essentially.
31:39
That would be the next step. Now I can get the really
31:41
specialised know how to develop it.
31:44
That's just for me. Now the point Zach made.
31:47
And I think there's concern that, Oh my gosh, if I use this,
31:50
the human will be displaced, etcetera, because AI can
31:54
hallucinate. You really need to have expert
31:57
knowledge of what is sensible and what doesn't make sense.
32:01
So there's a little bit of danger if you're trying to use
32:03
it to explore in some newer area you're not familiar with, you
32:07
need to make sure you're triangulating it with other
32:09
things and, and stuff, you know, it's your expert knowledge.
32:13
In other places where you're using it to experiment, you're
32:15
going to need to look at other sources just to make sure.
32:17
OK, that sounds interesting, but let me Fact Check it.
32:22
And there's ways to, there's settings actually you can do to
32:24
change that. There's something called
32:25
temperature. So if you actually make it, I
32:29
think it's really, really low. It's saying, I don't, I want you
32:32
to be absolutely certain about this.
32:34
I don't want you to be creative versus I want you to really
32:37
think crazy. You can set the temperature
32:39
really high and say, come up with the most creative, unhinged
32:42
thing. And even if it hallucinates,
32:43
maybe it's interesting, right? So it just depends on the nature
32:46
of the problem. But very quickly, generalist,
32:50
mundane pain points start there. As you build acumen, build
32:54
experimentation, you want to do something more specialised, you
32:56
can take in those models and train it specifically on your
32:58
data in a firewalled way. There's ways to do that now.
33:02
What benefits is AI bringing right now to to projects that
33:05
you've seen, Daniel? I think massive time benefits
33:09
already immediately, right, in terms of saving you all the
33:12
times you have to deal with things.
33:14
I think also it's saving you on cost later on because if you're
33:20
doing so much, the the theory is if I'm experimenting and
33:23
trialling things on several scenarios and getting a bunch of
33:26
these 80 percents and looking at it, in the long run, it's
33:29
helping me avoid issues that I would have that I wouldn't have
33:33
been able to recognise if I didn't try that experimentation
33:36
up front. But I think the most immediate
33:37
for me is the time benefits, things that used to take really
33:41
long time that have to be done are getting you most of the way.
33:47
And I think to Zach's point, as you use, if you do it with a
33:52
variety of LLMS, each trained in different ways, like Claude,
33:56
like Gemini, like Co pilot, like ChatGPT, you also can
33:59
triangulate pretty quickly as well.
34:02
And so I think there's there's like triangulation robustness
34:08
that I think in the immediate saves you time.
34:10
That's immediate. I think the cost stuff is, is
34:14
the things you would say based on comparison of what you had to
34:16
do in the past of mistakes you can avoid now because you've
34:20
you're able to experiment far more widely.
34:23
But I'd say immediately is the scheduling and time facets.
34:27
As a leader, so as a project leader, like one level up, how,
34:31
how do you lead on AI within, I know your department or you
34:37
programme or an organisation And how do you, I think, Daniel,
34:43
this is a point you, you made around making AI explainable in
34:48
terms of what it does and what assumptions it makes about
34:52
project delivery and outcomes. So, so to start asking people,
34:59
your workers, your Co workers to have that discernment, not to go
35:05
into this blind. I mean, it's pretty nuanced kind
35:07
of sophisticated stuff. If all you've been doing is
35:10
trying to experiment a bit with ChatGPT, you might not get all
35:13
of this stuff. So as a leader, as a project
35:15
leader, how should you talk about this?
35:17
How? Where should you be about the
35:19
language you're using and what you're asking people to to take
35:23
from what AI produces? I think you're really hitting a
35:29
profound nail on the head, Emma, because if you look at the
35:33
surveys with CEOs, executives, more senior leadership and
35:37
projects and organisations like Price Waterhouse Coopers did a
35:40
very nice one. And they basically said 61% are
35:43
like thinking this is going to solve a lot of issues.
35:46
They're going to increase productivity rapidly.
35:48
And then if you do, there's another interview by Upwork on
35:52
the day-to-day worker. Look at the, I call it the front
35:55
line. It's not a great word.
35:56
But let's just say the, the those who are nearer to the
35:59
locus of work and they find something like, I want to say 77
36:03
percent, 75 feel it's actually making them worse off and
36:06
burning them out. And what's happening is, is
36:10
you'd be sad. You know, we joke about the, I
36:11
mean, Zach mentioned the experimentation.
36:13
You would be surprised how many senior level people are just
36:17
saying this is going to solve everything without experimenting
36:19
with it themselves. And So what I think is
36:21
happening, the gap is, is an over expectation of what this
36:25
will achieve and how much judgement will be needed.
36:29
This is why I think it's really, really important.
36:32
First and foremost, senior leaders need to experiment with
36:34
it. And if you are actually, they
36:36
need to experiment this so they can calibrate their expectation
36:40
in judgement because they're looking at the kind of what they
36:44
see or hear or perceive and using that to drive.
36:49
Oh, yeah, yeah. Just do it for everything.
36:50
And that's creating a lot of burnout.
36:52
So that's one aspect. So the question is, and there's
36:54
different ways to think about it, Do you treat it like,
36:57
there's a very nice analogy by Ethan Malik at Gorton.
37:00
He argues, do you look at this as cyborgs or centaurs?
37:03
So the idea is, do you look at it as I even more pragmatically,
37:07
do I outsource bits and we look at it sequentially where AI does
37:11
something and passes it to me, etcetera?
37:13
Or are we doing it in parallel where you can't even see where
37:15
the human and AI end and there's different kinds of ways you can
37:18
use AI in your teaming to do that.
37:21
Do you treat it as another team member that you outsource stuff
37:23
and get back and sequence it? Or are they part of the ideation
37:26
where you don't even see where the the team comes and goes?
37:29
That's more like pragmatic. I think on terms of your
37:32
question about at the end of the day, we're dealing with a black
37:35
box where we don't know all the latent features.
37:37
There's a real push right now to get a sense of what is going
37:44
into the algorithms, how representative they are, how do
37:46
they change the data when they go in.
37:48
And there's some discussion about creating like prescription
37:52
labels for AI or nutrition labels with things like how
37:57
representative is the data? What data was this trained on?
38:00
Where is it vulnerable? How long is this?
38:03
How long is the insights I get from this algorithm worth useful
38:08
or not? Does it take?
38:09
Is it months? Is it days?
38:11
And I think even those simple things would really help people
38:14
understand. Yes, it's a black box, but given
38:16
what I know is going in and what's coming out, it gives me
38:19
enough kind of insight to say, OK, maybe I should try these
38:23
things versus another. To give a really tangible
38:25
example of management consulting you brought up, Emma, is that
38:29
they find they did a really interesting study at BCG and all
38:33
it was was literally cutting and pasting ideas into the into the
38:37
algorithm and there was a massive boost in performance.
38:41
I want to say like not massive, it was like 17% increase just
38:45
from copy and paste. But when they gave things to the
38:49
algorithm for which it was not equipped, sometimes these
38:53
algorithms have trouble triangulating data or even basic
38:56
math. They actually experienced a
38:58
percentage loss in performance. So part of the trick is they
39:02
have to understand what to give the algorithm and what is not
39:09
good to give. And so if we have these kind of
39:11
nutrition labels, prescription labels to say basically, yes,
39:15
these are the kind of things we're excelling at ideation,
39:18
brainstorming, recombining information.
39:21
And here's where it's not good, like triangulating between
39:23
spreadsheets and other things in interviews or even sometimes
39:27
basic math, then we're going to suffer.
39:31
So I think that and that's why it really it depends on senior
39:35
management experience. It would be great if they
39:37
experiment with their team so they can work on how they would
39:39
negotiate, how they would bring it into their protocols.
39:42
And from that they can learn more quickly what to share and
39:47
what does it get completely off so that you get the performance
39:49
benefits without the performance detriments, if you will.
39:54
That what are you talking about there is learning what
39:57
algorithms work for which data and that's the sort of thing
40:01
that. On a team, people individually
40:03
might be going off doing their own thing and not thinking
40:06
necessary strategically. So have you got any advice
40:09
around if there's a project team and everyone's doing their own
40:12
thing and they know they should be doing and it isn't the most
40:15
valuable thing there to share what you've learnt?
40:20
It is. And so as a leader, that's
40:23
something that you should be building into, building time
40:26
into regularly to make sure that everyone knows what everyone
40:29
else is doing and what works and what doesn't.
40:31
And somehow there is some kind of process around that so people
40:36
aren't wasting their time and everyone's reaping the benefits.
40:40
Because it's like you're very much working in, you know, with,
40:44
I imagine people who aren't that used to using this kind of stuff
40:48
in their projects or they're excited about it or, you know,
40:51
what advice do you give them? How do you see people use it and
40:55
experiment with it? And what advice would you give?
40:58
Yeah, absolutely. Yeah.
41:00
Well, I, I completely agree with all of the points that we've
41:04
talked about so far. I think that socialising, what's
41:09
working and what isn't within an organisation.
41:12
I'm personally a big believer and my background in startups
41:15
and perhaps show some of my bias, but I'm a big believer in
41:19
letting teams be independent and experiment with different things
41:23
and even in a bigger organisation and then have the
41:27
having the the successful strategies, the successful
41:31
projects rise up and then be disseminated.
41:35
And so I would encourage the way that I would think about it
41:38
would be to encourage teams to use different tools and
41:41
experiment and try, try out what the the new, the new breadth of
41:47
technologies, because we don't know what's going to work in the
41:50
long run. But we're still we're still in
41:52
the middle of this transition or even at the beginning.
41:55
And so there isn't one right answer that's going to at least
41:59
most organisations, all but the very luckiest probably are not
42:02
going to set one course through this storm and maintain that the
42:08
whole time, right. So the key instead would be to
42:11
develop systems, as you're saying I'm a really where the
42:14
best practises and the best new ideas coming up can be
42:19
socialised and spread more broadly and spread throughout an
42:22
organisation to have everyone on all the different teams follow
42:26
those best practises as they evolve.
42:30
I think it's really important to ask both of you where what you
42:33
think 2025 will hold. I mean, I know that it's hard to
42:39
look so too far ahead with this. And equally it can be fun to get
42:45
carried away the excitement to see what, what, what are the
42:49
possibilities, Zach? What, what would do you hope
42:53
2025 might bring? So what do you feel optimistic
42:57
about? And equally what what might keep
43:01
you awake in the middle of the night around AII?
43:05
Think that there's a lot to be excited about.
43:07
I, I personally believe that AI will be a paradigm shift across
43:14
industries. So in the long run, I think it's
43:18
of a similar scale to the Internet or personal computing
43:21
or these sorts of things where every industry, every vertical,
43:25
every specialisation, including of course PM is going to be
43:32
changed. And so I think that whether
43:34
that's in 2025 or in the next, in a few years after that, it's
43:38
a different question. But I do think that stay, I
43:42
mean, it's so exciting because there's so much possibility and
43:45
there's so much value from a business perspective to be
43:47
created and added and so much time to be saved and so many
43:51
more projects to be done. Just straight up, we can do so
43:55
much more. If we if we can work faster, we
43:57
can do so many more things, which is really exciting on a
44:00
human level. I think that just more and more
44:04
the ability to have AI agents are sort of these both somewhat
44:10
more autonomous tools that can have some long running context.
44:16
One of the biggest problems with some of our chat chat based
44:19
interfaces right now is that they lose that organisational
44:22
context. Today.
44:23
They don't know if you start a new chat, you maybe you told it
44:27
about some of your best practises and and that kind of
44:30
thing, but you have to start over if you start a new chat.
44:33
So more and more I think we're going to see specific dedicated
44:37
agents is sort of the term of art to have that long running
44:43
memory of here's how things work, here's how we do things,
44:47
here's how we did the last project even right, which is
44:51
something that short chat session would not have.
44:55
So an agent that has that long running context and they can
44:59
also bring to bear some of the general knowledge and general
45:02
skills to kind of get something done.
45:05
That's something I'm really excited about.
45:07
On the downsides, I think that we've already talked about it a
45:12
bit, but just there is risk with any, any paradigm shift this big
45:18
that some people in certain roles and even some entire
45:23
organisations that don't, all of which maybe don't adapt fast
45:29
enough to the, the new changes will be left behind.
45:34
And so I think that the biggest risk, I mean, the lucky thing
45:38
about this risk that I'm flagging is that we can all
45:42
still fix it. It's not the, it's not too late
45:46
to upskill yourself and your organisation and to adapt to
45:51
some of these tools and to keep adapting over the next 5 or 10
45:54
years and, and to, to weather that storm and learn what you
46:00
need to learn and build a more valuable and more powerful
46:04
really organisation in the longer run.
46:06
So there are risks there, but I think that we're it's still
46:10
early, so we can all we can all still weather this storm.
46:14
Thanks, Zach. That's so interesting.
46:17
Daniel what? What would be your take on what
46:21
what we can expect in 2025 and what might keep you awake at
46:24
night around AI? Yeah, it's hard to think of 2025
46:28
because, you know, if you talked McKinsey five years ago, they
46:31
were thinking, oh, it's 10 to 15 years from now.
46:33
We're going to have, you know, generative AI and language
46:35
learning. And then all of a sudden it
46:36
came. So I'm not sure when it will
46:39
happen. But my, my take away from this
46:41
is we got to think things that we thought were 5-10 years from
46:44
now could actually be next year. So I'm just going to say the
46:46
ones I think are interesting. I think there's a couple of
46:49
things. One is the powering of AI.
46:52
The amount of energy and water that's going to need to upkeep
46:56
the data is massive. So I'd be looking out for what's
47:02
going to happen in fusion energy, what's going to happen
47:04
in small modular nuclear reactors.
47:06
Because for example, the projections are one study out of
47:10
Yale basically said by 2026 even that the amount of energy that's
47:16
going to be needed for the servers is the same amount of
47:19
annual consumption in Japan. So we're talking massive energy.
47:24
The other one is cooling currently with just ChatGPT 3.
47:27
The the study I saw it uses something between 4.2 to 6.6
47:31
billion cubic metres of water. That's half the entire
47:34
consumption of the UK. That's nuts.
47:36
So look out of the technologies in terms of the cooling and the
47:41
powering. So that's one.
47:43
The other trend is how do I make AI work on small data?
47:46
Everyone's assuming big data, so I think you're going to see a
47:48
lot more on transfer learning. How do I use parsimonious data
47:51
that doesn't need an insane amount of energy?
47:55
So that's kind of 1, and I have some others, but you know, you
47:57
wanted to ask a question on this.
47:58
Yeah. No, no, no, that was that.
48:01
That was what I wanted to follow, follow up on.
48:03
What were the other things do you think are coming our way?
48:06
Not necessary, not necessarily for 2025, but on the agenda at
48:10
some point? I think an interesting one is
48:14
the developments in quantum. So machine learning right now
48:18
runs on what they call binary logic, right?
48:20
0 ones, typical computers, Quantum is game changing in this
48:24
regard because now assume instead of just having zero and
48:27
one, I'm looking at the probability of being zero and
48:31
one, now I'm thinking of a continuous distribution that
48:34
used to be binary and there is massive power that can be
48:37
developed from that. Now I think it's a ways away,
48:40
but look, if LLM happened earlier than we expected,
48:43
there's going to be interesting things of quantum.
48:45
I've already seen some of those methodologies used in marketing
48:48
etcetera. I think one that's more
48:50
immediate is AI going into the contracting or procurement
48:55
space. So the notion of smart
48:57
contracts. So imagine now instead of human
48:59
to human interaction or project, it's machine to machine.
49:03
So let's say company in my project, I'm a supplier A and
49:07
I'm the PMO and we've agreed that I want, I don't know, 100
49:11
widgets. I have an A sensor on the
49:15
machine building it that the minute that the widgets are
49:18
built, it immediately enacts the contract and delivers it.
49:21
It's entirely self governing. That's going to be interesting.
49:24
So you're going to see some interesting work on the
49:26
procurement space with smart contracts potentially on the
49:30
downside, the ability to discern between what is fake and what is
49:36
real is becoming really, really hard.
49:39
So there's going to be, imagine a world where someone has a
49:43
crisis, uses audio from a influential figure in politics
49:49
or corporation. You can foment crises quite
49:51
quickly. So we're going to have to build
49:53
rapidly our ability to better discern fake versus not.
49:58
And so looking at the regulatory space is going to be
50:01
interesting. I think the US, the EU and even
50:03
the Chinese are thinking differently about this.
50:06
I think the US is a market LED approach.
50:08
They want to see what the market does and let that kind of run.
50:11
I think the EU is taking a more safety approach.
50:13
They're worried about validating the data quickly.
50:16
This is why you see the EUAI Act versus how it is in the States.
50:18
And I think in China they're thinking about the integration.
50:22
They realise that some of this is going to be difficult, but
50:24
they realise they have to build their capabilities to discern
50:27
what's real or not. So I think the regulatory space
50:29
is going to be interesting to see about how we handle the, you
50:32
know, the proliferation of this stuff at scale, but also
50:35
recognising our inability to recognise synthetic versus not.
50:39
In some cases that may not be a problem.
50:41
In some cases you can obviously imagine it could be a very big
50:43
problem. I'm going to try and wrap this
50:45
up now, but are there any final thoughts that either of you want
50:49
to share? Or maybe I can ask you how
50:52
optimistical, pessimistic you feel about this right now?
50:56
I'm very optimistic. I think my my final takeaways
51:00
would be just to embrace, embrace this paradigm shift if
51:05
you can, as an individual contributor all the way up to an
51:09
executive. Anyone who's working on PM
51:13
problems, I would say just embrace, Embrace the change and
51:17
see how you can do your best to see how you can apply it to your
51:21
organisation over time. That's all that anyone really
51:24
can do. Daniel, do do you want to give
51:27
us your your final thoughts? How optimistic are you feeling?
51:32
Yeah, sure. That's, that's a hard question.
51:35
I would say I'm in the middle, right.
51:36
I'm measured about this. I think, I think there's a real
51:40
lot of potential. But we're also at the same time
51:44
really as a society, very ill equipped with the capabilities
51:50
to really harness that potential and where it where to understand
51:53
where it goes wrong. So my encouragement is very
51:55
similar to what Zach said, which is the need to experiment.
51:59
Everyone needs to experiment at a level of which to do this and
52:04
not to try to outsource it. The temptation is let me
52:07
outsource it to a technology unit.
52:09
Let me outsource it to my lower end.
52:11
You need to experiment it with yourself to calibrate your
52:13
judgement. And I think hopefully with that
52:15
as a society, if we can build the right labelling, the right
52:19
guardrails, I think it could really harness that potential.
52:22
But it's still early days. So I would say, you know, it
52:27
depends on what day you get me. Listen, it just needs to say
52:30
thank you to both of you for an interesting conversation that
52:34
could and should go on for another couple of hours.
52:36
So thank you again so much for your time.
52:39
It's been absolutely fascinating.
52:42
And I'd love to catch up with you again in the last six months
52:46
and see how far it's all moved since then.
52:49
So, so thank you very much for your time.
52:52
Yeah, thank you. This was a fun conversation.
52:54
Thanks, Ella. Thanks again to Daniel and Zach
53:04
for joining us and to you for listening to the APM podcast.
53:08
I don't think I've ever recorded such a mind expanding podcast
53:12
before, so I hope they've given you much food for thought and
53:15
practical advice on how you can use AI in projects and how to
53:18
lead on experimenting with it. But anyway, don't forget to look
53:22
out for more episodes or to rate and reviews.
53:24
Wherever you get your podcasts, we'd welcome you to get in touch
53:27
with your comments, feedback and suggestions by emailing us at
53:31
APM Podcast at syncpublishing.co.uk.
53:35
This podcast has been brought to you by APM, the chartered body
53:39
for the project profession. For more information on APM,
53:42
visit apm.org.uk.