Latest / Project Podcast / The origins of agile project management

Transcript

The origins of agile project management · Project Podcast

0:09
Welcome to the IBM podcast. AIM is the chartered body for
0:12
the project profession. My name is Emma De Vita and I'm
0:15
the editor of Project APM's quarterly journal and your host.
0:19
In this podcast, I'm speaking to Darren Dalcher, Professor of
0:22
Strategic Project Management at the University of Lancaster.
0:25
Few people have started project management as closely or for as
0:29
long as Darren, who's also a director of the National Centre
0:31
for Project Management and Co editor of the 7th edition of
0:34
OPM's Body of Knowledge. In a career spanning more than
0:38
25 years, Darren has become very interested in why projects fail
0:42
when it comes to Agile. Darren was there from the start
0:45
when it was being experimented with in software development.
0:48
Who better than to explore the origins of Agile and its impact
0:51
on projects? Listen on.
0:58
Welcome Darren, Thank you for your time.
1:01
Could you tell us how you originally got interested in
1:04
Agile, what its origins were? I was involved in software
1:09
development originally as an engineer developer, and then I
1:14
moved away from that and at some point I developed an interest in
1:19
failures. As a practitioner, I'd worked on
1:23
a number of on IT projects. They're all kinds of challenges
1:26
around them, and personally I was trying to make sense of some
1:30
of the challenges, some of what I was seeing around me.
1:33
The whole notion of success and failure were a little bit
1:36
dubious, as far as I could see. We had fixed deadline, fixed
1:41
ideas about what what we needed to be produced, and sometimes we
1:46
delivered on time, sometimes we didn't, Probably around
1:49
19818283. I was working on a number of
1:54
projects. We were using this new idea
1:57
called prototyping when we were spending a lot of time with the
2:00
users. We were trying to understand
2:02
what they were after and essentially a lot of people
2:05
didn't have an idea what computer technology could
2:07
deliver. So working with the users was an
2:09
opportunity to introduce to them different ways, different
2:13
options. Very often we didn't know that
2:16
much about the users world. We would have a client coming in
2:20
getting us to do a system. We came from the technical side,
2:23
so we had an idea about how to develop things, but not
2:26
necessarily the user's world. And essentially it was a way of
2:31
interacting with people. So for me, prototyping was a way
2:34
of interacting. Prototyping made the project
2:37
longer because once you started interacting with the users, once
2:41
you spend time with them, it was all about the dialogue, all
2:44
about the understanding. This is where you'd realise that
2:47
the requirements that you were given or the the need statement
2:51
that was passed along wasn't exactly what the users needed.
2:56
But we've kind of gone the complete circle now and we
2:59
recognise that we need to get the users on board.
3:01
There's no point delivering if nobody uses the system.
3:04
We deliver systems, we deliver projects so that it can be used.
3:08
So the results need to be deployed, the results need to be
3:11
used. There's no point delivering a
3:13
bridge if no one's going to go across it if no one lives.
3:15
In the area so, so it's all about the use, but we were
3:18
learning that as we work it to computer technology was becoming
3:23
more and more enabling. There was more that you could do
3:26
with it. And it became all about creating
3:29
the interaction, creating the relationships and understanding
3:32
what people wanted. And how they could do their job
3:36
better and what we could achieve for them through computers.
3:39
So really my journey there starts with an understanding
3:44
that we need to gain a better insight into what people are
3:48
trying to achieve. And it's only when we understand
3:50
what we're trying, what they're trying to achieve, that we can
3:52
deliver something meaningful and useful to them.
3:56
So is that really the core of what Agile meant?
3:59
So this to a large extent predates Agile.
4:02
This is prototyping, It's experimentation, It's something
4:06
that people have been doing for a long time.
4:09
We have been experimenting, we've been trying things out.
4:13
So rather than have a fixed idea about what we could deliver,
4:16
what needs to be done, it's playing with it a little bit and
4:20
and trying to make sense of what we're getting and seeing if we
4:23
can, if we can find a better solution.
4:26
Tinkering with the solution being empiricist, being playful
4:32
developers. The way we develop things has a
4:38
has the sense of a problem solving cycle.
4:40
We move some problem towards solution and we try not to
4:43
engage with this solution too early.
4:45
We go through different modes of development and to a large
4:48
extent computers enabled us to do a lot of things that we
4:50
couldn't do before. So they they enabled us to to
4:54
process large volumes of data, to do so very quickly, to do
4:58
amazing calculations, to change how we how we implement systems,
5:03
how we deal with people. They enabled us to do a lot, so
5:07
I was involved initially in trying to utilise computers in
5:11
useful ways, but also trying to make sense of what is success
5:14
and failure. And for me, prototyping was very
5:17
much an early precursor of of Agile.
5:20
We were doing exactly what our journey is doing.
5:22
The challenge was as soon as you introduced prototyping, as soon
5:27
as you engaged with the users, the user they wanted more.
5:30
So the challenge was not not a technical challenge, but there
5:34
was a management challenge. How do you control this, this
5:37
effort? It can go on forever.
5:40
And then the real challenge is how do you pay for it?
5:44
How do you structure it? So it became about.
5:48
How? How do we control this
5:49
prototyping activity? Do we only allow?
5:54
3 increments however long does increments are.
5:57
So we could say, well we will do 3 versions of the system we will
6:01
go through in three times. Or we can just do it for two
6:04
months and at the end of the two months what we have is really
6:08
the the answer unless you want to pay more and then we we will
6:12
do another increment. So it was really imposing some
6:15
kind of a managerial structure around it, and I was trying to
6:19
make sense of what seemed rather arbitrary at the time, the
6:23
notions of success. Failure, because if you say,
6:26
well, I'm going to deliver a system in six months.
6:29
And then you spend 10 months with the users because they're
6:32
engaging with it and you're delivering a better system.
6:35
Are you failing or are you succeeding?
6:39
More so? This is kind of in the 80s.
6:44
The Agile Manifesto was published in 2001.
6:47
So what was it trying to achieve and how did it fit with what was
6:51
going on with prototyping? You know, if you cast your mind
6:54
back to them, what what? What did you think its value
6:56
might be? To set the context in a way that
7:02
we are still dealing with project failures, they're quite
7:06
a few challenges around projects, primarily because we
7:09
are doing things that have not been done before.
7:11
So computers enable us to seventies 80s, really.
7:16
Even before that, some amazing things are happening through
7:18
computers. They're really exciting
7:20
projects. The problem is it's they're
7:22
quite difficult to predict. They're quite difficult to
7:24
control. It's back to this control
7:26
question. So there is a perception of
7:28
failure. There's a perception that
7:31
software projects fail a lot. And something needs to be done
7:35
about it. And they're various efforts,
7:38
various communities are formed and they're looking at different
7:41
kinds of solutions. So some of them are looking at
7:44
various ways of introducing quality.
7:46
So all kinds of quality structures, some of them are
7:49
looking. At what would later become the
7:51
idea of benefits. And some of them are looking at
7:56
ways of creating a craft, a discipline that is more
8:01
controlled. So software engineering is born
8:04
around 1969. In the NATO conference, which
8:09
brings together a lot of scientists who concerned about
8:13
suffer being out of control. So the idea of engineering
8:16
software engineering is to impose engineering structures.
8:19
So bring in that control over what we're doing and software
8:22
engineering allows us. To plan, go through the
8:27
requirements, design, develop and deliver.
8:32
And deploy meaningful software almost in an engineering kind of
8:36
way. So there are different
8:37
communities all over the place and the different communities
8:40
are engaging with different types of approaches.
8:43
Some of them more formal. Software Engineers will try and
8:45
find formal methods, but within that some would argue we still
8:49
need to work with people. Back to this if you like the
8:53
fight between. Creativity and control.
8:58
Various methods are created and throughout the Seventies, 80s
9:03
and 90s there are all kinds of structured approaches.
9:07
To obtaining requirement and structured approaches to design.
9:11
So the structured requirements, structured design and our
9:14
communities. Now those methods are becoming
9:16
more and more structured and more and more controlled in
9:19
response to the failures. We need to control the effort.
9:22
We need to know what we're doing.
9:23
We need to start here and finish over there.
9:25
And suffering engineering becomes quite controlled and at
9:30
a time I'm teaching systems and software engineering, so I'm
9:33
doing, I'm doing a lot of that. I've shifted from doing to to
9:39
teaching, researching, making sense of things.
9:42
And we we are particularly focused on some of the methods
9:47
that are emerging. So I had a strong interest in
9:50
methods. I had a strong interest in what
9:54
could be achieved and in this shift between control and
9:58
creativity and they have various efforts that are going on.
10:01
So in Japan, Takeuchi and Nonaka are saying, well to do, you
10:05
know, when you look at innovation, when you look at
10:08
creativity and how people work, it's not really a relay race
10:12
where you hand over to someone else.
10:15
So this is I think 1996, they're this, they're saying it's more
10:19
for. It's more like a rugby match.
10:22
When you pass the ball backwards and forwards that they come up
10:25
with the notion of it's a rugby match and this is the idea of
10:28
scrum being born. It goes in all kinds of
10:31
directions and at the same time other people are coming up with
10:36
other lighter methodologies. They're calling them, but
10:40
they're really methods. Lighter approaches to
10:42
development, lighter life cycles.
10:45
So what happens in 2001 is 17 people involved in the creation
10:52
and the development of of flight and methods.
10:56
Get together. What they have in common is that
10:58
they all believe that there's too much, if you like, structure
11:01
and control. And we need to be we need to
11:05
liberate ourselves a little bit. And we need to recognise these
11:08
are the people that come into the meeting, they've all created
11:12
their own methods or selling particular approaches.
11:16
That emphasise working with users.
11:20
Little bits of prototyping. Some of the methods can be
11:22
traced back to prototyping. Some of them are.
11:27
Simply lighter approaches. They're not project managers,
11:30
they're software developers. They're interested in software
11:33
and what they do is they they get together for a weekend.
11:37
There were a number of similar get togethers and this is this
11:40
was one particular group and they do want a lot of groups to
11:44
get together. Do they?
11:45
They come up with a manifesto, of course.
11:48
Wouldn't it be great if we could shape the world in the way we
11:51
like it and everyone used like methods?
11:55
So they come up with a statement, and a statement
11:58
reflects the kind of thinking at a time.
12:00
It comes a little bit from manufacturing, a little bit from
12:04
innovation. They're inspired by other
12:07
developments that are going on at the time.
12:09
And they're really saying we think we should be working in
12:12
different ways. We should be using lighter
12:16
methods. We should be doing things with
12:18
those lighter methods that without the control, without the
12:22
structure, without the imposition of of the.
12:25
Bureaucracy and they're all trying to sell their own
12:28
individual methods at the time. So the challenge for them, for
12:31
17 people in that in that room is to to come up with a coherent
12:36
statement that reflects. The flexibility, the freedom,
12:41
the that are common across the all the methods that they're all
12:46
representing. So.
12:47
So how do we create a sort of a statement of independence?
12:50
What is our our common cause? Did you know any of them
12:56
personally? Had you met any of these people
12:58
in that room? Yes, I knew a few, quite a few
13:01
of them at the time. I've met since.
13:05
I've done keynotes and I've spoken at conferences with many
13:10
of them, so I've so, yes, very much so.
13:13
And. I knew many others liked them.
13:16
It was pretty much the prevailing kind of approach.
13:20
Programmers don't like structure.
13:23
They don't like controls. So having come from that
13:25
background, I was familiar with. I was familiar with the
13:29
constraints. I was familiar with the ways of
13:31
thinking. So they basically work on the
13:34
manifesto over the weekend and the manifesto is 4 statements of
13:38
value. So we prefer something over
13:41
something else. We prefer people over tools,
13:45
essentially. The relationships and working
13:49
with people of the tools and methods and approaches, we
13:52
prefer delivering working products over comprehensive
13:57
documentation. So so we like to do things.
14:01
So the first one is saying we like to work with people.
14:04
And I suppose that's quite a strong statement for
14:06
programmers. Remember, programmers are the
14:08
people you keep in the basement. Now they're coming out.
14:11
They're saying we want to work with people.
14:12
Let me work with people. So they're working with people
14:15
now. And like me, I suppose they got
14:18
excited by the fact that ohh, I can shape things for you, I can
14:21
shape the world for you, I can use the computer to create a
14:24
better world for you. So.
14:25
So we work with people we prefer.
14:28
Delivering, working things, doing rather than documenting
14:33
and documentation wasn't a popular part of the software
14:37
process. It's the thing that came at the
14:38
end and it was you think you've done the thing.
14:41
Now go and test and then go and do the documentation.
14:45
We prefer customer collaboration over really contract
14:49
negotiation. And we prefer responding to
14:53
change over just following a plan.
14:55
If something changes, let's go with it.
14:58
Let's flow with it. So it represents an attitude, it
15:03
represents an age, it represents the people in the room, it
15:06
represents a long weekend in the cold February holiday resort.
15:11
It's a group of people saying wouldn't it be great if we could
15:14
change the world? So they come up with those
15:16
statements. Later on they will add the the
15:20
12 principles, but those four values really about 30, I don't
15:26
know, 40 words, something like that.
15:29
This is our manifesto. This is what we think we'd like
15:31
the world to look like. And one of them goes and posts
15:34
it on the web because you can now and reasonably quickly
15:38
there's some traction. Various communities realise
15:42
that. This is going on.
15:43
And they say, yeah, yeah, yeah, me too, me too.
15:45
I want to be part of this movement.
15:47
I believe in those principles as well.
15:50
So it becomes something within the software development
15:54
community, becomes a way of thinking about how we do, how we
15:59
develop software. I don't want to say how we do
16:01
projects, but certainly how we develop software, because there
16:03
is an implication that we're back to this creativity and
16:06
control. So it becomes perhaps an
16:09
alternative way of looking at how we develop.
16:14
I mean, I'm understanding that it was never intended to be a
16:18
management method or a project management method, but it has
16:23
permeated not only project management but management in
16:26
general. What's your take on this?
16:29
Do you think the people who originally created this are
16:32
surprised that this has happened?
16:34
What's your view on how Agile has influenced project
16:39
management and what it's become now?
16:43
Right. So when they post the manifesto,
16:45
it's the manifesto for Agile software development.
16:48
So that's quite explicit that that that's the title of it and
16:51
and and it remains like that. The document hasn't changed
16:55
since 2001. There was a conscious decision.
16:58
It reflects a point in time when they got together.
17:02
So what essentially was the the manifesto for Agile software
17:06
development became known as the Agile Manifesto.
17:10
So at some point we we dropped the software development.
17:12
But when I first knew it, it was this manifesto for Agile
17:16
software development. And within the software
17:18
development community there was a mixed response.
17:22
Some people got excited about what Agile?
17:24
And deliver. About the fact that it could
17:27
liberate you from some bureaucracy, it enabled you to
17:31
do things differently. It was quite an exciting
17:34
development certainly for programmers and a lot of the
17:38
agile literature from that time is saying we don't need project
17:41
managers, It's the end of the project manager.
17:45
Project manager can can disappear.
17:48
One of them is saying the role of the project manager is to
17:51
deliver pizza. So, so there are different
17:54
perspectives around that, but somewhere along the line.
17:57
People get excited by the concept.
18:00
Now. The software development
18:01
community, to a large extent, has moved on from Agile.
18:05
This suffering the software development community, if I
18:08
reflect back over the years, roughly every 10 years there's a
18:12
new trend and a new trend coming comes in.
18:14
And it could be structured systems analysis or it could be
18:17
structured design or software engineer.
18:20
Something will come along and it will grasp everyone and they'll
18:24
get excited by it. And then there'll be a new toy.
18:27
And we are looking at roughly every 10 years being there's a
18:31
new exciting development and to a large extent software
18:37
development and software engineering have moved on to
18:40
thinking about the delivery of services rather than products
18:45
and looking at continuous delivery, continuous testing,
18:48
continuous delivery, continuous deployment, So delivery becomes
18:53
more constant. The agile community has in
18:56
software development has certainly moved on.
18:59
But a lot of others, as you said, have have picked up on on
19:03
Djali. Do we need to be agile?
19:05
And that's without fully understanding what our journey
19:08
is. Our journey is this mythical
19:11
thing that we can do so we can be agile.
19:13
That that sounds good. We can be.
19:15
I want to be able to do magic. I want to do all kinds of
19:17
exciting things. So, so we can be agile, we can
19:20
do things, and we can pivot. We can respond.
19:24
And a lot of what our child gives you is the ability to.
19:28
Ready to to respond to change, Going back to where I started
19:33
and those experiments. Agile allows you to do little
19:36
experiments and on the basis of those experiments to amend to
19:40
some extent what you're doing. So you're being more responsive,
19:44
you're being more adaptive. And in terms of management,
19:48
management goes back to the bureaucracy to control the.
19:52
So having this magic ability to, yeah, do you want to be agile?
19:55
Yes, of course I do agile. Agile is exciting.
19:58
It's not boring old structure, it's I can be agile, I can do
20:02
what I like. It allows me to do all sorts of
20:05
amazing things. So there's a lot of traction,
20:07
There's a lot of people are impressed by it.
20:10
Some of the origins, as I pointed out, also go back to
20:14
production and manufacturing. And innovation management.
20:19
And there's a challenge around finding the balance between
20:23
product development and project management.
20:26
And is it a technical development that we are doing?
20:30
Can we get, can we do away with the project manager?
20:33
Can we? So the project management
20:36
community has found agile. Now they're very it seems like
20:39
an exciting development, and there are all kinds of
20:43
conversations around what replaces what does all this
20:48
waterfall V agile discussion everywhere, and which is
20:53
relatively meaningless because neither of them is a project
20:56
management approach. We're APM, the only chartered
21:01
membership organisation for the project profession.
21:04
When you become an AP member, you'll receive the resources and
21:07
support you need to make an impact, delivering better
21:10
projects with better outcomes. Plus you'll access exclusive
21:14
training and benefits to support your ongoing career development.
21:17
Find out how we can help you reach your potential by visiting
21:20
apm.org.uk, Because when projects succeed, society
21:26
benefits. Is the project management
21:31
profession using Agile? However, they define it as an
21:35
effective way to have fewer project failures.
21:39
Is it a useful tool? Interesting question.
21:43
We have fewer of the old failures, but our job will
21:46
introduce inevitably introduce its own kinds of failures.
21:49
I've been collecting failures for literally all of my career,
21:52
not just my own. I'm I'm less fussy nowadays.
21:56
I'm I'm happy to welcome anyone to the club. 10s of thousands of
21:59
failures have made it in and there are some major failures
22:03
and they're very different. What what Agile allows you to do
22:07
is you've got small, small teams doing things and this is what
22:11
the essence going back to 2001. This is the essence of that
22:16
meeting. This is what it was all about.
22:19
It was a small solution to a small problem of a small group
22:24
of people working together to do small things.
22:26
This is what our job enables you to do and this is what they were
22:31
talking about. So this is it liberates a small
22:35
team to do things far more creatively.
22:38
And at the time we were doing some research around Agile which
22:44
the view of determining whether it had, whether it had values.
22:48
We were looking at the quality of what the teams were producing
22:52
and we did a one year study where we we looked at multiple
22:56
projects and we looked at the outputs of teams in the banking
23:00
sector and we we measured the the output and the agile teams
23:06
produced far more code which means they produced far more
23:11
functionality. The obvious question would be
23:13
was that useful functionality? The answer would be to spend a
23:16
lot more time with your clients. So they did a lot more.
23:19
They delivered a lot more. But it introduced different
23:23
challenges. The fact that the software
23:25
community has moved on and is interested in dev OPS reflects
23:30
an interest in continuing operations.
23:34
It reflects a need to integrate other activities all the time to
23:39
to to continue to develop and to have backlogs that allow you to
23:44
bring in items from different areas and integrate them into
23:49
your deliverables. It means that agility.
23:53
So make the chef now. Agility allows you to do things
23:56
differently. Agility allows you to deal with
23:59
uncertainty. Agility allows you to see what
24:01
you can do. Action has been really exciting
24:05
during the Pandemic actually allows you to start.
24:08
You have a small team run with it, go with it.
24:11
Here's the ball run. So Agile allows you to enables
24:15
you to do all kinds of things. It's up to the project manager
24:19
to decide whether it is the most useful way of deploying off
24:24
using their team and the most useful way of doing things.
24:27
It's still the skills of the project managers.
24:29
I don't think the world is just to deliver pizza.
24:32
The role is to decide what you're hoping to deliver and to
24:35
say, well, we can do things together, we can do exciting
24:39
things and agile will allow us to experiment going back to
24:43
where we're starting, you can do your experiments.
24:45
You can see whether it makes sense.
24:47
During the pandemic, that was amazing.
24:49
You actually said to the team, so we need to start.
24:54
Rules are changing from This is a Friday afternoon announcement.
24:57
Rules are changing from Monday. Go run with it.
25:00
We don't really know what it means.
25:02
Let's go and explore the boundaries and it it liberates
25:06
teams. It enables them to do things but
25:09
it it's something that allows a small a small team to do things.
25:14
Somewhere along the line we realised, wait a minute, if one
25:18
team can be our gel, can we have lots of teams playing in the
25:22
same field being agile? So then we started talking about
25:24
scaling agile, and now we're talking about business agility.
25:27
And now we're talking about all kinds of other notions which are
25:31
far more complex. The problem is, if you allow one
25:35
team to, I'm not gonna say misbehave.
25:37
If you allow one team to have fun, you have to allow all of
25:40
them to have give everyone the ability to have fun together.
25:44
And that's where we come back to that good old balancing act
25:49
between creativity and structure and control.
25:55
One team having fun is exciting enough and noisy enough, but can
25:58
you let hundred teams, thousand teams do that?
26:03
So we are back to OK, now we need to introduce scaling
26:07
methods and approaches to to to getting multiple teams work
26:11
together. So we are beginning to put some
26:13
certain controls around that. If you ask the original people
26:16
who attended the the meeting and created the Agile Manifesto what
26:20
they think about it, They hate it.
26:22
Their view is that Agile was this liberating activity and
26:26
they don't like the many of them, don't like the scaling
26:29
approaches. There are various methods that
26:31
have been designed to do that and they feel that they are
26:33
adding a lot of structure, which you would need to do when you
26:37
have multiple teams playing and having fun together at the same
26:40
time. You want to control the noise,
26:41
you want to control the space, you want to control what's going
26:44
on. You need to manage it, so the
26:47
challenge for project managers is to think what it can achieve
26:50
through our journey. It goes back to the skills of
26:53
the project manager. We if we don't assume that all
26:56
the knowledge is there, it's really how do we deploy, how do
27:00
we create, how do we become pragmatic managers use their
27:04
professional judgement and decide how to deliver, how to
27:08
shape, how to deploy in meaningful ways.
27:12
So it's creating strategies that will allow the mix that will
27:16
allow the different activities that need to take place to
27:19
happen together at the same time.
27:22
Darren That's a. Fantastic way to end this
27:24
podcast. Thank you so much for your time.
27:27
Really, really interesting speaking to you about this.
27:29
So thank you again. It's a pleasure.
27:32
Thank you very much. Thanks again to.
27:40
Darren for joining us and to you for listening to the APM
27:42
Podcast. Don't forget to look out for
27:45
more episodes or to rate and review us wherever you get your
27:48
podcasts. We welcome you to get in touch
27:50
with your comments, feedback and suggestions by emailing us at
27:53
apmpodcast@thinkpublishing.co.uk. This podcast has been brought to
27:59
you by APM, The Chartered Body for the project profession.
28:03
For more information on APM, visit to apm.org.uk.