Latest / Project Podcast / How to measure anything in project management

Transcript

How to measure anything in project management · Project Podcast

0:14
Welcome to the APM podcast. APM is the childhood body for
0:18
the project profession. My name is Emma David and I'm
0:21
the editor of Project APM's quarterly journal and your host
0:24
today. In this podcast, we're talking
0:27
about how to measure anything in project management.
0:29
Yes, you did hear me write anything.
0:32
It's the title of a new book that is co-authored by Douglas
0:35
Hubbard, Alexander Budzia and Andreas lead.
0:38
I have the pleasure of being joined by Doug and Andreas Now.
0:42
Doug is founder and president of Hubbard Decision Research, is a
0:46
management consultant and the author of a series of books
0:48
including the brilliantly titled How to Measure Anything.
0:52
Andreas is head of Data Science at Oxford Global Projects and is
0:56
a PhD Fellow at AAHSA University in Denmark.
0:59
Be prepared for mind expanding conversation about project
1:02
management and how any aspect of it can be measured, including
1:06
those things you thought might be immeasurable and why we
1:09
should all be engaged in the most important project of all,
1:12
the meta project of understanding how to manage
1:15
projects better. I'd like to begin by welcoming
1:19
Andreas and Douglas to the APM Podcast.
1:21
So thanks very much for finding the time to speak to us today.
1:25
Thank you for having us. You bet.
1:27
Let's begin by you telling us a little bit about your area of
1:32
work and interest within project management.
1:34
So Andreas, tell us a bit about what you're what you're working
1:38
on right now. Yes, Sir, of course.
1:41
So I'm, I'm Andreas lead and the head of Data Science at Oxford
1:44
Global Projects. And Oxford Global projects is a
1:47
small consultancy which is really about analysing vast
1:51
amount of vast amounts of project data and and using it to
1:58
to have a better decision making around projects.
2:01
Another of what we do is about, you know, debiasing estimates
2:05
and kind of assuring estimates, realistic, basing estimates on
2:09
historical data rather than bottom up type approaches.
2:14
But it's also kind of bigger than that.
2:17
So as the head of data science, I did some of the more
2:20
quantitative engagements, things like using AI for machine
2:25
learning for forecasting project outcomes or doing kind of
2:28
econometric cost driver analysis.
2:32
And the big goal with all of this is, is to better projects
2:35
because after having looked at all the project data we have
2:39
access to, which is now about 20 data from 20,000 completed
2:43
projects, the the big conclusion is that rojects aren't erforming
2:48
that well. O we'd like to understand, you
2:50
know, what leads to success and how can we imrove roject
2:53
erformance? Thank you.
2:55
So that's not necessarily a controversial finding, is it,
2:58
unfortunately. But good to know.
3:01
We're here today to talk about how we can make perhaps make
3:04
projects more successful through the way they're measured or what
3:07
what is in fact measured. So Douglas, tell us a bit about
3:10
your area of interest and and actually tell us how the book
3:13
came about as well, because that's always good to know.
3:17
Yeah, sure. So I'm Doug Hubbard.
3:21
I'm the president and founder of Hubbard Decision Research, and
3:24
for the past 36 or 37 years or so I've been in some form of
3:29
quantitative management consulting.
3:32
So all of my work, including all my books, this was my fifth book
3:37
actually, They've all been on some topic related to decisions
3:42
under uncertainty and difficult measurements.
3:45
So that includes risk analysis, making big bets, and lots of
3:49
different areas of forecasting problems.
3:52
And of course, that applies to projects in general.
3:57
My previous four books were not about projects specifically, but
4:00
certainly a lot of my work was about risk return analysis on
4:04
major new investments, which were usually projects in regards
4:08
to how we met. Actually, I'll let Andres
4:12
describe this because initially 2 of you called me.
4:18
Yeah, so. So I was writing an article
4:20
together with a colleague about cost, benefit, cost benefit
4:24
analysis. Now kind of alternatives to, to
4:27
typical inclusions in cost benefit analysis.
4:30
I, I also teach cost benefit analysis at the Department of
4:34
economics here in all who's where I'm based.
4:37
But we found that Doug's work was really interesting.
4:40
So we, we asked him for an interview and kind of some ideas
4:45
around what should we include in this article.
4:48
And I guess it was, it was a good match.
4:50
Doug was looking for his next book project and project
4:53
management was one of the topics he was considering.
4:56
And so because we had access to a lot of data and a lot of
4:59
analysis that kind of fit in really well with the book.
5:02
I think that's fair, isn't it, Doug?
5:04
Oh yeah, absolutely. In fact, I I chose this book
5:07
title based on analytics. I decided to go after How to
5:13
Measure Anything in Project Management because it had
5:16
previously tied as one of the top 2 with How to Measure
5:20
Anything in Cybersecurity Risk, which I've already wrote now.
5:24
So I wrote that one first. And what tipped the needle in
5:27
the in favour of the first book of the the 4th, my 4th book.
5:32
But the previous How to Measure Anything in Cybersecurity Risk
5:34
book was that I had a a well qualified, enthusiastic
5:38
co-author that said, Hey, I'll write this with you.
5:42
And I said, OK, well, then let's get started on that.
5:44
And so I was really looking for the right people.
5:47
I actually talked to a bunch of other possible candidates for
5:51
co-authors for how to measure anything in project management.
5:54
And my strategy is always to find somebody who's a
5:57
specialist, an expert in renowned in that field because
6:02
I'm always the outsider, I'm always the outside quant
6:04
analyst, applying it to different kinds of problems,
6:07
etcetera. So what I wanted to do a spin
6:09
off of my first book, my first book was just how to measure
6:12
anything. So that and I meant it
6:14
literally. So how to measure anything.
6:17
And I decided to do a series of spin off books like that.
6:20
And so I waited for the right co-authors and when I learned
6:25
about Oxford Global projects, I thought, well, that's ideal.
6:28
They've got all this data. They, they already have these
6:32
guys that are, that have been studying projects and all the
6:35
problems with projects for many years now.
6:37
So it seemed like the ideal co-authors.
6:42
That's really interesting. So Doug, can you measure
6:47
anything? Sorry, I've got to ask you.
6:50
Yeah, absolutely. Including the hardest most
6:52
possible things you've ever heard of.
6:54
Can you, I'll think of some ideas as I as as we progress
6:58
about what might be really hard to measure, but it sounds as
7:01
though the book then is a really nice mix of the quality of
7:04
quantitative, someone who's very deep with people who are very
7:07
deeply involved in the profession and then more of an
7:10
outsider. And I think that always adds a
7:13
really valuable perspective on the same topic.
7:16
I'm going to dive in there and ask you a really hard question
7:20
and probably very hard to answer in not very much time.
7:24
But but what's new about what you're saying in project
7:28
management and how it's measured?
7:30
Because I've read in the forward something very ambitious that
7:34
you wrote that was that you don't just want to challenge
7:37
traditional project management practises, but replace them with
7:40
something more of rigorous, transparent and effective.
7:44
This sounds really interesting. So.
7:46
So tell me more. I don't know who'd like to go
7:49
press. Doug, would you like to share
7:52
that with us? So I approach everything as a
7:56
sceptic and in every field I've looked in, whether it was
7:59
cybersecurity risk or operational risk management or
8:02
portfolio prioritisation methods that businesses use or just
8:06
decision making in general, I tend to find that there's a lot
8:10
of, a lot of assumptions they make about what works and what
8:14
does it. They don't really know, they
8:16
just assumed it always has worked.
8:19
They they never really measured it and they're often surprised
8:22
when they do measure it. So it's the sort of thing where
8:25
it's not really obvious though, like if you're, if you're
8:29
golfing, it's going to be obvious whether or not you're
8:32
good or bad at it, right? You get your score right at the
8:34
end of things, right, other kinds of sports, etcetera.
8:38
But in a lot of management decision making, you have this
8:41
really inconsistent and highly delayed and ambiguous feedback
8:45
cycle. So it's not really necessarily
8:49
obvious that what you're using is really one of the better
8:53
methods. There's probably many other
8:54
methods. People tend to be very
8:56
overconfident about their own judgement.
8:58
We know this from a lot of other research that's been, you know,
9:01
spearheaded by many others. We also know that they're highly
9:05
inconsistent, that they're influenced by irrelevant random
9:08
external factors, and they dismiss too many things as
9:13
immeasurable when all the research shows that you're
9:16
better off doing the math on a wide variety of problems.
9:20
Even relatively simple statistical models seem to
9:23
outperform human experts in a variety of different fields.
9:26
You can't trust the humans. Well, you can't trust the humans
9:30
for some things. It's really important you figure
9:32
out how to use the humans, what to use them for, and what not to
9:37
rely on them for. Quantitative methods in
9:40
mathematics are really important.
9:42
Human invention, slash discovery.
9:45
All right. As important as language itself.
9:48
OK. And it solves a lot of problems.
9:52
It's a tool that we should use more often.
9:54
We we revert too frequently to just rely on our own judgement.
9:59
We're having too much trust in other human beings when in fact
10:02
the data shows that simple statistical models are
10:05
outperforming them. So does that pass the argument
10:09
in the book? Oh yes, absolutely.
10:12
We, we take a sceptical view of, of a lot of the methods that are
10:15
used in project management and Andres can talk a little bit
10:18
about that. But we did, in addition to the
10:21
analysis of the 20,000 or so projects that they had in their
10:24
database, we did our own original survey of little over
10:28
200 project managers and projects and pretty extensive
10:32
literature review, more than 100 different, you know, peer
10:36
reviewed articles, standards, other sources and so forth.
10:40
And we were looking for what's the evidence that some methods
10:43
are measurably better than others.
10:46
And that was one of the surprising things.
10:50
I'll let Andreas talk about some of that.
10:51
But really, what works doesn't is will be a surprise to many
10:57
project managers I think. Well, what?
10:59
What were these discoveries you made, Andreas?
11:03
So I, I might just, I might just take a, a step back because one
11:06
of the first things we did when we approached the book was,
11:10
well, can can we say something about some of the, do some
11:14
projects perform better than others?
11:15
Do projects use specific types of methodologies or frameworks
11:20
or different types of tools for estimating how do they perform
11:24
compared to others? And so the the first thing that
11:27
we actually did was to look at our large database of projects,
11:31
which goes back to we have some historical projects from the
11:34
1800s. And what we see there is kind of
11:38
a very consistent trend in terms of projects not delivering to
11:42
expectations in terms of cost and schedule and also benefits.
11:47
And and that's despite the fact that since at least since the
11:51
90s, we've gotten more and more kind of methodologies and
11:54
frameworks and all these things that supposedly help us with
11:56
projects. So we, if these frameworks and
12:00
these tools we've developed works, we would expect to see,
12:04
you know, budget overruns have gone down over time.
12:07
We, we don't see as big delays. We are better at getting the
12:11
benefits that we, we expect we will from projects.
12:14
But when we look at kind of project performance over time,
12:18
we don't see that a performance improvement since 1990s despite
12:23
all this, all this work and improving projects in terms of
12:28
methodologies and certifications and all this.
12:30
So that was kind of the first clue.
12:34
Afterwards, we, we launched this survey where we looked into,
12:38
well, what, what do projects use?
12:40
And can we see some kind of statistical differences between
12:44
different types of approaches to delivering projects?
12:49
And, and I think the, the evidence, what we found was was
12:54
very contrasting because project management is a field where
12:56
there's just a lot of claims, There's lots of methodologies
12:59
that promise, you know, if you use this approach to deliver
13:03
your project, you're going to get twice the benefits in half
13:07
the time. But we didn't see that for any
13:09
methods. We didn't see any methods really
13:11
outperform others. We, we saw very few actually
13:15
effects of applying any kind of methods at all to projects.
13:20
I think that the one very consistent finding we had was
13:24
that projects have spent more time in the front end
13:27
development of projects. They they tend to perform better
13:31
and and secondly projects that apply more quantitative and
13:34
evidence based methods. They also tend to perform better
13:39
in terms of adhering to their estimates compared to projects
13:42
that use more qualitative and subjective estimating
13:46
techniques. One of the things I thought of
13:50
as I was going through the book was that I thought listeners
13:53
might be interesting. Was was I, I?
13:57
What can be measured in projects?
13:59
Can everything be measured and what are the things you should
14:04
be measuring? So the methodologies you're
14:06
saying out there, many of them don't deliver any notable
14:09
results. And is part of that because
14:11
they're measuring the wrong things or what do you do with
14:15
the measurements once you've got them?
14:17
So big, a big couple of big questions there.
14:20
So I guess if you want to take, can you measure everything in
14:25
projects and then which are the ones you should be measuring?
14:29
I think that would be really interesting.
14:32
Well, the simple answer is yes. There's literally nothing,
14:37
literally nothing that has any consequences at all for the real
14:41
world. Every project that anybody has
14:43
ever proposed had some expected consequences, right?
14:49
And every consequence is necessarily observable.
14:53
Otherwise, it wouldn't be something that everybody cared
14:56
about, right? It would have to have observable
14:58
consequences. No matter what you're thinking
15:00
of, it's really hard to measure if you have a project that's
15:03
supposed to improve collaboration or innovation or
15:06
the environment or customer satisfaction, or some sort of
15:10
social welfare or children's education, it doesn't really
15:13
matter. In each of those cases, there
15:16
are expected observable outcomes.
15:18
Now, you might be highly uncertain about them.
15:21
That's a different issue, but you can identify the
15:23
observations first, then the rest is about observations and a
15:29
little bit of math to reduce your uncertainty.
15:32
Does that mean you need to know what you're observing before you
15:37
start measuring? Do you know?
15:39
Do you have to have an expectation of what there will
15:41
be to observe before you can start thinking about measuring
15:44
that thing? You don't have to, but you won't
15:48
know the value that measurement until you know the reason for
15:50
it. You can have purely exploratory
15:53
discovery driven observations. A lot of science is this way.
15:56
You you don't necessarily know what you're looking for, right?
16:01
If I go to a, a, a, a new city and I'm walking around downtown,
16:06
I don't know what I'm looking for necessarily, right?
16:08
I might discover something. But if I'm going to spend a lot
16:12
of time and effort measuring something in particular, I have
16:15
to know what it's for, right? And all measurements are meant
16:21
to reduce uncertainty, to make better bets.
16:24
You're making better decisions. That's what all of this is
16:28
about. Every project manager, what
16:31
management means is all about making decisions before and
16:35
during a project, sometimes even after a project.
16:39
So you can think of projects, project decisions during the
16:42
project as sort of intervention decisions.
16:46
So what are they measuring if they're not going to do anything
16:49
with the measurements during a project?
16:53
That is, they can't think of anything they might do
16:55
differently at all. If it was surprisingly high or
16:58
surprisingly low, well then that has a information value of 0.
17:03
This is a computable value, by the way, the value of
17:05
information. So it's sort of like the cost of
17:08
being wrong times the chance of being wrong.
17:11
It can get a little bit more elaborate than that.
17:13
But generally speaking, if there's, if you can't think of
17:16
any consequence that would be different at all, or if there's
17:20
no uncertainty about it already, well then there's no value to
17:22
that that information. Do you think that project
17:26
professionals. Oh, know that deep down that
17:30
they do question why they're measuring something or is it
17:33
just we're just measuring these things?
17:36
So it's there's enough thought given to or consideration given
17:39
to what's being measured and why, what the outcome you want
17:43
from that measurement to be Andreas?
17:47
So I think there's a lot of culture and project management.
17:50
We typically measure the things we've measured before or that
17:54
that other projects measure. And it's actually one of the
17:58
things we go into that, you know, there's all these standard
18:01
setting bodies, there's all these kind of influential
18:03
organisations in, in the space of project management.
18:06
And a lot of them just kind of repeat.
18:08
This is how we've delivered projects before.
18:10
This is how we've done project management and how we've thought
18:13
about, you know, quantifying risk in some way.
18:16
But they, they typically don't really distinguish between what
18:20
are the methods that are proven to actually give you better
18:25
results versus the ones that in, in the best case don't do
18:30
anything, but at, at worst, they might add uncertainty.
18:34
We use kind of a risk matrix as one of the examples of, so a way
18:39
of we, we call it analysis placebo in the book, but it's,
18:43
it's the idea that you, you feel you're kind of measuring a risk
18:48
by using risk matrices to kind of quantify the impact versus
18:52
the likelihood. But there's multiple problems
18:56
with, with that type of more subjective scale approach where
19:01
all the research from, from not just product management, but
19:05
also different fields show that these types of quantifications,
19:09
they, there's a lot of error with them and they don't
19:12
actually contribute anything good.
19:15
But in projects, we, we apply some of these tools and because
19:18
we've been applying those tools, we feel that we've actually
19:21
we've done some kind of assessment of a risk even though
19:25
we haven't really, we call it analysis placebo that because
19:28
we've spent time applying A structured framework, now we
19:32
feel better about the situation even though we haven't reduced
19:35
uncertainty at all. So, so I think that's a lot of
19:40
that kind of culture. But another thing that's we
19:44
found was pretty surprising in the book was that projects say
19:47
spend a lot of time measuring things that have the lowest
19:50
information values. If you want to stick to that
19:53
framework of the value of information in terms of reducing
19:56
uncertainty for our decision making projects, they spend a
20:00
long time kind of estimating or measuring the CapEx cost, but
20:05
they don't spend as much time measuring the OpEx cost of the
20:09
kind of the maintenance of the the infrastructure, the project
20:13
that they build afterwards. Where typically the
20:16
uncertainties is typically in the latter and the Opecs the
20:21
same with benefits. Often times big projects,
20:24
especially big public projects have wider societal benefits and
20:28
those are really uncertain. But projects they don't spend as
20:32
much time trying to measure and reduce the uncertainty about the
20:35
benefits side of the project as they do cost because they're the
20:39
cost is the big typically it's, it's the big component that they
20:45
could ask questions about. They have to deliver too.
20:48
So should project professionals be focusing more on the benefits
20:56
that will come from a project and measuring those?
20:59
Yeah. Actually, when we compute the
21:02
value of information, and we've done this many times on many
21:05
different types of investments, including lots of different
21:07
categories of projects, we tend to find the high information
21:11
value variables. So I mentioned that's an
21:13
algorithm you can actually run. These are more sensitive
21:16
variables. They change the outcome of the
21:19
decision more and they're more uncertain, right, than other
21:22
variables. The high information value
21:25
variables tend to be the things they haven't measured before,
21:28
and that often includes benefits.
21:30
They spend more time historically spending things
21:33
that spending their time on, things that have lower
21:38
information values, things that are statistically less likely to
21:41
actually improve decisions. So we call this the measurement
21:44
conversion in our books. I think this is a pervasive
21:49
issue in every industry and projects and also outside of
21:54
projects. I I don't know how it couldn't
21:56
affect the GDP of countries. It would kind of have to.
22:00
Why have we ended up this way? What?
22:03
Why are we not measuring benefits better or measuring
22:07
them at all? There's a general, there are a
22:11
few different reasons I think and Andreas, you can comment on
22:15
this. But First off, I think there's a
22:16
habit of when you label something as intangible and
22:20
immeasurable, it's never considered again.
22:24
Once it's labelled intangible or immeasurable, no one ever goes
22:27
back and ask, is it really immeasurable?
22:30
Are there really no observable consequences at all that could
22:34
reduce our uncertainty about this?
22:35
Have we even figured out what it is that we're measuring right?
22:38
Often things seem immeasurable because we have an ambiguous
22:41
label. We're using terms like
22:42
collaboration or innovation, but there's observable consequences
22:46
that you can't identify out of each of those things.
22:48
So there's that. Things get labelled too quickly
22:54
or at all as immeasurable. That's one problem.
22:57
They're all measurable. The other problem is that people
23:02
are just as Andre has pointed out, it's just a matter of
23:05
culture. They measure what they know how
23:07
to measure. They don't really look at a
23:09
problem and say, what should I measure given this problem and
23:13
then learn how to measure those things.
23:15
That's not the approach. The approach is I know how to
23:17
measure these things. That's what I'm measuring.
23:21
So what needs to change to make projects delivered more
23:27
successfully or deliver better benefits?
23:29
Is is part of that looking at benefits and realising that
23:34
these can be measured in a useful way?
23:39
Yes, that's part of it. The first big project management
23:44
decision is approval and prioritisation.
23:47
Should you even do the project in the first place?
23:51
In those cases, even the decision of doing the project in
23:56
the 1st place is. And that first big decision,
23:59
modelling it quantitatively, is what's needed.
24:02
Because when we start looking at that problem by itself, we find
24:06
out that really most decision makers probably would not have
24:10
accepted most of their projects. Seriously.
24:14
Oh yes. So when you look at the data,
24:17
there's a way to quantify risk aversion as well.
24:20
And when and when you look at all the research on what
24:23
executives say about how risk tolerant they are, how risk
24:26
averse they are to, and then compare that to the actual
24:30
uncertainties and risk of the big projects, they wouldn't have
24:34
accepted almost any of them. So fewer projects, bigger wins.
24:41
Is it one approach, right. And also spending more time on
24:45
that early selection, you know, if somebody says, hey, this
24:49
project is a good deal because we make more than our cost of
24:53
money, that's not nearly good enough.
24:57
If your return is just a little higher than your discount rates,
25:00
you say, well, it's a good public project.
25:02
You know, we're going to make the the cost of this big
25:06
infrastructure project is justified or the cost of this
25:09
big software project or Rd project is justified because we
25:13
expect these returns. If those returns are modest,
25:17
like modest improvements over your cost of money, your
25:19
discount rates, that's not nearly good enough.
25:23
If a project is expected, If we look at just all the projects in
25:27
their database, if a project is expected to have a lot twice the
25:32
benefits as costs, and this is present value discounted to
25:35
current dollars, current units of monetary units.
25:39
If you expect twice the benefits as you have cost, there's still
25:43
so much uncertainty on benefits and costs.
25:45
There's about a 14% chance of losing money still.
25:49
In fact, there's even a chance of losing more than the money
25:51
you put into it. And that's.
25:55
Quite something, yeah. Yes, if you spend, if you spend,
25:59
you know, £100 million on some big infrastructure project, is
26:04
it possible you could lose more than 100 million?
26:08
Yeah, actually there is. One of the cost that are not
26:11
typically included in cost overruns is other process
26:15
disruptions. So you've seen that's when
26:18
there's a big infrastructure project, there's a bridge being
26:20
built or a new road being repaid for something like this, and
26:23
then there's a traffic congestion for a while because
26:26
traffic has to be rerouted. Well, the project takes longer
26:29
than expected. So that cost is the congestion
26:33
cost is longer than expected, maybe even greater than
26:36
expected. That's how you actually have
26:40
cost that aren't in your budget. There's a other disruption cost
26:45
in software cases. There's been cases of software
26:49
where a lot of money has been spent to implement something.
26:52
And then they figured out that this software that was supposed
26:54
to improve productivity made things worse.
26:58
So they had to unravel it. They had to undo it and take it
27:01
out, go back to the old way. That's more that cost more than
27:06
the cost of the project to do all of that.
27:09
Now that doesn't happen every time, but it happens frequently
27:13
enough that it's a risk we should consider and it's often
27:18
not even acknowledges a possibility at the beginning of
27:21
this approval process. So that's one big thing.
27:24
Fewer projects, bigger wins, you know, a 20 or even 30 or 40%
27:29
return. Not nearly good enough.
27:31
You got to go bigger, spend more time in that upfront planning
27:36
and look for better options. Keep exploring.
27:40
There's one of the issues we talked about in the book is
27:42
exploration versus exploitation. So this is kind of a classic
27:47
measurement problem. It's a operations research
27:49
problem. How long should I generate
27:52
solutions before I pick the best one so far and and build on that
27:57
one? Because there's a chance the
27:59
next solution you generate is going to be better than any of
28:01
the previous ones. On the other hand, you keep
28:04
deferring the project, which has benefits itself presumably,
28:08
right? So that's something that isn't
28:13
typically looked at and it's something you could spend more
28:15
time on in front end loading is you could spend more time
28:19
generating solutions. It appears that people converge
28:24
too quickly on a preferred solution.
28:27
They should think of many more solutions and they should
28:31
generate many more decision models and options to be used
28:35
during the project. So it's not just the solutions,
28:37
it's not the just the various designs of bridges, for example,
28:42
that you're looking at. If you're also designing the
28:46
measurement programme during the project, you're also designing
28:50
the intervention options. These are decision models that
28:53
run during the project. You're designing things.
28:57
It's like what method should I use?
28:58
Should this be an agile project or some modified version of
29:01
that? That's all part of front end
29:03
loading. And it's usually those things
29:06
are they're not only not spend enough time on they're, they're
29:10
just ignored as choices. They're just assumed that we're
29:13
going to deal with this project with this way.
29:15
We're going to follow this particular standard and we're
29:18
going to measure these particular things and maybe
29:21
after looking at a few solutions, we're going to pick
29:23
that solution. That's a big problem.
29:27
Just to add, when we talk about solutions, we mean solutions in
29:30
a much broader kind of definition of that word.
29:34
So when we talk about iterations of bridges, we really should
29:38
spend more time thinking about alternatives to bridges as well
29:41
that fulfil the same kind of requirement for the bridge.
29:44
So thinking in fairies or tunnels or other types of
29:49
solutions, not just iterations of bridges, because that's one
29:52
thing that we saw too, is that we we fixate on a type of
29:56
solution fairly quickly in projects and then we do
29:59
iterations of that kind of design.
30:01
How should that bridge look or how should that nuclear power
30:04
plant be? Whereas we could consider much
30:07
different types of solutions as well.
30:11
I don't know where to start with everything I want to ask you,
30:13
because that was quite macro. It's quite philosophical about
30:17
what? What do we do and how do we do
30:20
it in other alternatives, let alone the complication of
30:25
political feeling or government. When you hit government mega
30:29
projects for example, where the politics behind it can make it
30:34
something happen without really being questioned why or is this
30:38
the right thing to do. So for people listening, what is
30:44
the how can they imagine someone just in a how I get a job
30:49
working on a project? How can you begin to deliver,
30:55
execute, think about your projects in a in a better way
30:59
along the lines he suggests. And are you saying, let's look
31:04
at the data, that let's make data more of a priority than
31:08
these humans who might be too might be prone to optimism bias
31:12
or and then with the rise of AI is, is this going to nudge
31:18
projects more towards the way you're thinking that that if
31:22
it's not measured, it should be measured, Everything could be
31:24
measured and we should listen to the data and see the evidence of
31:27
what actually works and be more objective about it.
31:33
I think this is an interesting, interesting thing that comes up
31:36
quite a lot. So many might say, but what
31:38
about somebody who's working on a project right now?
31:42
I think that's already too far down in the weeds because just
31:46
organizationally that person doesn't really have the position
31:49
or authority to make some of the changes we're talking about.
31:52
We have to start much higher and much earlier in the
31:55
organisation. We have to talk about
31:57
stakeholders changing the way that they approach things,
32:00
right? So it's stakeholders that are
32:02
making ultimately making choices about where they should allocate
32:07
funds in the first place. And something has to be
32:10
communicated there that it's to their benefit to spend more time
32:15
in this analysis before you make big bets.
32:18
You know, if you're buying a house, you look at a few houses,
32:21
right? If you're buying a car, you look
32:23
at a few cars. You don't just pick the first
32:25
one you see, right? And I think decision makers
32:30
often are too quick to say we got to start coding or we got to
32:34
start pouring concrete, right? I, that's a, that's a, this is a
32:40
different way of thinking about it.
32:41
We have to spend more time in that upfront analysis, analysing
32:45
our choices. Again, fewer projects, bigger
32:48
wins. That's the, the optimistic part
32:51
here. I think it's the bigger wins
32:52
part there. There's room for really
32:56
impactful mega projects. That's where we should be
32:59
thinking those really fundamentally life changing
33:05
kinds of projects, socially changing kinds of projects.
33:09
Like like what? Such as?
33:12
Oh well, we have a couple of examples in the book, but at
33:15
least in the states here, is it possible?
33:19
Ideas have been proposed and it should be analysed.
33:21
Is it possible to stop hurricanes?
33:25
That's a hypothesis. And even if there's, if you work
33:28
out the information value that if there were only a 5% chance
33:31
of that being true, it'd be worth investigating just because
33:35
it's so impactful. The cost of hurricanes is starts
33:40
to compete with the cost of our entire aircraft carrier fleet,
33:44
you know, in the US, right, or the entire Interstate system of
33:49
expressways in the US. All of a sudden, when you add up
33:54
a few big hurricanes like that, you say, well, yeah, actually,
33:57
if it were even possible to even slightly reduce of the impact of
34:01
hurricanes, what are the different ways we could do that?
34:04
That's a big project. That could be a very big
34:06
project. It might not even be a big
34:08
project, Might be a bunch of little projects, by the way.
34:12
But also, you think about AI itself.
34:15
We talk about AI in the project and we tell people that you're
34:18
single most important project, and this is the biggest project
34:21
of all of them. By the way, the single biggest
34:24
project the world is facing right now is how to do projects.
34:29
That's the single biggest project.
34:32
That's the most important mega project that any country or
34:35
company is dealing with is how they should do projects.
34:39
And this is a key message in the entire book.
34:42
There's a lot of ways to run projects and a lot of ways to
34:44
measure projects, and we're just not spending enough time
34:47
figuring out which ways work better.
34:50
Well, some are better than others, and we're still stuck on
34:53
some, you know, habits that don't work as well as other
34:57
things. Thanks Doug.
35:00
Andreas, what? What are those better ways of
35:02
running projects then what? What actually has does give some
35:07
success. So we do have a lot of, I think
35:11
one of the positive messages of the book is that there is a lot
35:14
of potential, there are a lot of things you could be doing to
35:18
improve the projects you're doing at the moment.
35:21
So far we've talked a lot about, well, if we, we're able to
35:25
reduce uncertainty about the bets that we're making, we would
35:28
be making better bets, probably fewer bets, but the portfolio
35:32
projects would be entirely different to what we're
35:34
investing in now. And it would probably be better
35:37
because a lot of what we, a lot of the projects we invest in now
35:41
are the ones that look best on paper, but aren't necessarily
35:44
the the best projects in in the book.
35:49
Obviously the book is about measurement.
35:50
So we talk about like what kind of measurement should you be
35:53
making? Why are you doing?
35:55
Why are you measuring in the 1st place?
35:57
How should you be doing measurement?
36:00
And one of the things when it comes to information value is,
36:04
is often, if you don't know anything, almost anything will
36:09
tell you something, right? So if the uncertainty is really
36:14
high about something you haven't measured in your project, you
36:16
don't need to collect a lot of data to reduce that uncertainty.
36:22
And that's also part of the measurement inversion we talked
36:25
about. We spend a lot of time fixating
36:27
on the cost of the project. And sometimes to marginally
36:30
reduce the uncertainty, we have to do a whole whole bunch of
36:33
engineering in a project that might be really expensive and
36:37
there's still a lot of uncertainty after we've done
36:39
that detailed engineering. Whereas when it comes to some of
36:43
the benefits coming out of projects, since we haven't
36:46
observed it so far, we, we don't really need to make a lot of
36:50
observation to be in a better place than we were before, to
36:53
have reduced the uncertainty of what can we expect to get out of
36:56
it and how can we manage that better.
36:58
So we, we talk about, you know, you probably need less data than
37:04
you think you need and you probably have more data than
37:07
than you think you do. And that's generally going to be
37:11
the case for organisations and for projects, especially on the
37:14
things they aren't measuring so far.
37:17
What what other things we talk about that you should be
37:20
measuring as well as the the outside environment of projects.
37:23
You mentioned politics as as an example of how it can add
37:27
uncertainty to projects. And obviously that should be
37:30
something if you're doing a big project that runs over multiple
37:33
years, you should be measuring kind of the political context,
37:36
political support of a project, because that's going to add
37:39
uncertainty and it might mean that you're building in more
37:42
decision options along the way of a project.
37:47
But that's also one of the recommendations of the book
37:49
that, you know, the projects, they aren't in their own bubble.
37:53
They they do engage with and they are impacted by the
37:56
external environment. So that's also some other things
37:59
you should be measuring that you haven't been before.
38:03
Thank you. Could you give an example of a
38:05
project that perhaps does take on board the things you're
38:09
telling me the recommend, the recommendations that you make in
38:12
the book? Is there anything out there that
38:14
people could find out more about?
38:17
I've run into a couple of examples in in industries that
38:21
do some of the things we're talking about.
38:23
So for example, the option to cancel a project, that should be
38:30
an option that's defined very well in advance.
38:33
It's a part of a decision model that's run every day.
38:36
Now nobody's doing quite that just yet, but in the
38:39
Pharmaceutical industry and some related industries, they have a
38:42
stage gate process. So only about 8% of all
38:47
pharmaceutical products that are investigated ever make it to
38:50
market. So how do they make any money?
38:53
Well, when they quit, they quit early.
38:57
And when you look at a lot of big projects that cancelled, and
38:59
a lot of them do, they cancel with nothing to show for it
39:03
after spending a lot of cost and having a lot of external
39:05
consequences that are costly after.
39:09
If you look at those kinds of projects, you often find out
39:12
that they could have and should have cancelled it quite a long
39:16
time ago. It they had all the data that it
39:18
would would have justified cancellation quite a long time
39:21
ago. It was just running on inertia.
39:24
But also there's that feeling of, I think is it called sunk
39:27
cost, sort of sunk cost of allergy where you've put all
39:31
this money in already so it's too late to turn back.
39:36
Right. It should always be on the
39:38
forecast of the future. Here's what we here's our
39:41
uncertainty about what there is remaining to be spent given what
39:45
we've learned so far about the complexities of the project,
39:48
because that might update our uncertainties about what's
39:52
remaining to be spent. And we're comparing it to the
39:56
outside world. It might not just be the
39:57
project, it might be that something in the outside world
39:59
has changed that changed the benefits of this thing.
40:04
If I'm. 35% of the way into a project for some big
40:08
infrastructure project and AI at the same time is improving
40:13
because the and it will, because these are multi year projects,
40:17
right? So there will be changes in not
40:20
just AI, but materials and things like this and fabrication
40:23
methods during the project. At what point during the project
40:27
do we say there's enough that's changed about the outside world
40:31
that I should now re engineer something about this project?
40:35
That's a intervention decision. That's an intervention option
40:39
that we should define in advance.
40:41
So the pharmaceutical companies are one, but also aerospace does
40:46
a little bit of something that we talked about where they do
40:50
anticipate changes in technology.
40:52
Because if they're developing a new crew re entry vehicle from
40:56
the space station or something like this, they start designing
41:00
it without necessarily assuming that the only materials they
41:04
have are the current materials. Because it'll take a decade or
41:07
more to design these things. And during that time there will
41:11
be technological improvements. They're kind of counting on them
41:14
in some ways. So we should take that into
41:17
account. If we have a big infrastructure
41:19
project, should we be saying asking questions like if AI
41:23
could have changed this design if only I waited 2 1/2 years
41:27
from now, what would I do differently?
41:31
Is is there a point at which it might be worth waiting?
41:34
There's something we introduced in the book called Technology
41:37
Regret. Technology regret is includes
41:41
things like you invest too soon in a technology that's rapidly
41:44
changing, so you fixate on a bridge design or fixate on some
41:49
software features at the same time while you're pouring
41:54
concrete and writing code. AI is getting better.
41:59
Not only AI, but materials and fabrication methods, 3D
42:03
printing, hole bridges and so forth.
42:05
That's getting better during this.
42:07
Now AI is actually, I think improving faster than the other
42:10
two items I mentioned. So we should at least be looking
42:14
at there. There's not too many big
42:16
projects that are going multiple years that shouldn't also be
42:19
tracking what's going on with AI and asking questions about.
42:24
Is there a point in time I should consider the possibility
42:27
of right now and design an intervention option where I'm so
42:31
far into the project and something the outside world has
42:34
changed? And given how far I am into the
42:36
project, and given the benefits of switching gears at that point
42:40
in time, does it make sense to change something during the
42:43
project? And you're saying that the
42:46
answer could be yes, it could be, it could be beneficial to.
42:50
So, so you're, you're planning in uncertainty, you're kind of
42:56
planning that. That's quite probably quite a
43:00
mindset shift for lots of people working on projects like this.
43:06
Yeah, I think people are used to the concept of options, though
43:10
some are. They're used to the concept of
43:12
real options too. You know what's the what's the
43:15
value of keeping an option open, right?
43:19
Actually, even things like the option to cancel a project.
43:23
This surprises people, but it de risk a project.
43:27
Risk of a projects go down when you build in intervention
43:30
options because in the case that you do need to cancel a project,
43:35
you'll at least be able to cancel it sooner.
43:38
So your expected losses would be lower if you do cancel a
43:41
project. Now I've had lots of clients
43:44
that would say, yeah, I've seen projects get cancelled, but
43:47
prior to us developing decision models for them, they never saw
43:51
a chance of cancellation on the business case for a project.
43:56
How often do you see that right, if you're if you're presenting
44:01
to a government agency or someone else in the corporation.
44:04
I've got this idea for a project.
44:07
Should I include the fact that there's based on historical
44:10
data, there's about a 15% chance of cancellation with nothing to
44:12
show for it. Some we've heard this, Andres
44:16
and I were just talking to a group of the last time I was in
44:18
the lending group of, you know, project managers and stuff like
44:21
this. They'd say, well, you know,
44:23
things wouldn't get approved if that were the case.
44:25
If you're I said, well, if that's what you're looking at,
44:27
then just leave out half the cost too.
44:30
Why not? So the no, it's, it's about
44:34
being transparent about these issues.
44:38
Also, I think there's a fundamental conflict of interest
44:42
here if the person opposing the project is the only one building
44:46
the decision model for it, right, and the only one
44:50
presenting it to decision makers.
44:53
I do think there's room for some independent body of auditors.
44:57
Actually, Oxford Global Projects does some of this right now
45:00
because they audit people's projects, project plans and
45:03
costs and schedules. That's part of the services that
45:05
they do. And one of the values of having
45:09
an outside consultant come come in, in general is that, hey, we
45:12
don't have any skin in the game here.
45:14
We're just going to do our analysis and give you an honest
45:19
representation of the outcomes. Because usually we're working
45:21
for the check writers, the people who were allocated funds,
45:25
not necessarily the project manager or the person who wanted
45:28
to promote the project. We're working for the people who
45:31
have have to commit to the funds in the 1st place.
45:34
So they want an honest answer and I think that's part of it.
45:39
We have to think about some built in conflicts of interest
45:42
in the way that we approach even proposing projects to begin
45:46
with, but and then everything else that we just talked about.
45:51
South more rigour what? Andreas, what would you like to
45:53
add to that? I love this conversation has
45:56
been about you know what, What type of projects should we be
45:59
making? How can we plan projects better?
46:03
But we do spend quite a long time in the book talking about
46:06
different types of measurements for projects that are already in
46:09
flights. What should you be start
46:11
starting to measure and how can you make, how can you make kind
46:15
of metrics more, more decision driven?
46:20
So we, we talked a little about, we, we have specific chapters
46:24
about, you know, cost and schedule and benefits progress,
46:27
typical things that you need to measure on, on projects and that
46:31
it's really useful to measure. But one of the things we
46:34
observed was that a lot of the metrics that are currently being
46:38
used and on all sorts of dashboards, they're very kind of
46:44
exploratory, as Doug would put it.
46:46
So they're there because decision makers are project
46:50
managers, are risk managers. They feel that once they know
46:53
that once they look at the dashboard full of metrics and
46:56
they see like the right combination of different figures
47:01
and charts and, and metrics, they'll have some kind of
47:04
they'll know what to do in that kind of instance.
47:06
They'll know how to react on the metrics on the dashboard.
47:09
But a lot of the time what we discovered was that a lot of the
47:12
metrics aren't geared for decision making really, and not
47:16
about, you know, what intervention decision or how
47:19
should we react if we see a specific metric.
47:23
So the book spends a lot of time thinking about, you know,
47:27
progress metrics like cost performance indicators or
47:32
scheduled performance indicators and so forth, and thinking
47:35
about, well, how could we make it reframe that into something
47:38
that is easier to understand. And a lot of that is reframing
47:43
it into, well, how much productivity increase would we
47:46
need to have to actually catch up on our delay, for example?
47:51
So we do give a lot of advice in terms of how should you be
47:56
measuring, how should you be presenting metrics?
47:59
And I think the other thing in the book that we do is we be
48:02
sure that because this is a book about measurement, there is some
48:07
maths in it. But we do show that, you know,
48:10
all this maths isn't complicated.
48:12
So alongside the book, we have a website where you uploaded a
48:16
whole bunch of spreadsheets to show that all these things.
48:19
We talk about the technology regret example, we have an
48:23
example of using machine learning for predicting
48:26
outcomes. We have different types of
48:28
simulation exercises and we run it all in Excel, which you can
48:33
download and apply it to your projects just to show that.
48:36
It sounds pretty complex if you're used to using more
48:39
subjective scores and scales and risk matrices, but it actually
48:43
isn't that difficult and there's a lot of already built solutions
48:48
out there that you can just apply.
48:52
If there's one thing you'd like readers to take away from your
48:56
book, what what might that be, Andreas?
49:02
So if, if I can positively frame it, because you've talked about
49:05
issues a lot on this on this podcast so far, is that there is
49:09
actually a lot of evidence, even though there's in the industry,
49:13
there's a lot of claims around or if you just apply our methods
49:17
or our tools or our software, you're going to better projects.
49:21
That's typically what we see. If we go out to conferences and
49:24
we look at like vendor stands, it's all the same software
49:26
companies with some kind of reframing of if you use our
49:30
products, you're going to deliver to cost into schedule,
49:33
right? That'll be the typical
49:35
experience. But there's so many claims out
49:38
there can be quite difficult to find out what works and what
49:41
doesn't. And what we've tried to do is
49:44
actually just to look at the data.
49:46
The research has already been done.
49:49
We've done some of our own research and we've identified
49:52
that, you know, some things are better than others in terms of
49:56
measurement or estimating or delivering projects.
50:01
So that there's there's an opportunity to apply the things
50:05
that work and kind of toss the things that don't work.
50:09
Fantastic. That's very well put.
50:12
Thank you, Doug. Any any one message you'd like
50:16
to get across to to listeners? Yeah, we, we refer to the the
50:22
most important project, the one that is about making projects
50:26
better. We call that the meta project in
50:29
the book. And everybody ought to start
50:32
thinking about their meta project.
50:33
You know, no matter how big your project portfolio is, no, no
50:39
matter how big the individual projects are, some time is worth
50:43
being spent on the meta project. Your most important decision is
50:48
how to make decisions. By the way, that's in life and
50:52
business and government and everything.
50:53
Your most important decision is how to make decisions.
50:56
So we should apply that here. And so the meta project is, is
51:01
the project about making all the other projects better.
51:04
What's a great way to spend 1% of a whole project portfolio
51:09
trying to optimise the other 99%?
51:11
That's the best way to spend 1%, right?
51:14
And it might be about that much for a lot of project portfolios.
51:18
If you look at some of the infrastructure portfolios, you
51:21
know, even spending 1% of those portfolios is a significant meta
51:25
project where a lot of analysis could be done.
51:29
And so we recommend having a project called the Meta Project
51:34
and Everybody's portfolio. Thank you, Doug.
51:38
I feel as though we've just scrape the surface of what we
51:42
could talk about in your book. I feel like I might have to do a
51:45
follow up podcast sometime soon. But thank you very much for your
51:49
time. Thank you for your thoughts.
51:51
I wish you best of luck with the book, but it's been really
51:55
fascinating talking to you and I'm sorry we couldn't cover much
51:59
more of what you've what you've put in the book, but thanks very
52:01
much for your time today. You best text.
52:05
Thank you. Thanks again to Doug and Andreas
52:09
for joining us and to you for listening to the APM podcast.
52:12
I hope you've been left with much food for thought, so don't
52:16
forget to look out for more episodes, alterate and reviews.
52:18
Wherever you get your podcasts. We'd welcome you to get in touch
52:22
with your comments, feedback and suggestions by emailing us at
52:25
apmpodcast@thinkpublishing.co.uk. Spotify and YouTube users,
52:30
please do also leave us your comments.
52:33
This podcast has been brought to you by APM, the chartered body
52:37
for the project profession. For more information on APM,
52:40
visit apm.org.uk.