Latest / The Book Brief Project / How Big Things Get Done | The Numbers Were Never Real

Transcript

How Big Things Get Done | The Numbers Were Never Real · The Book Brief Project

• 0:00
  Speaker 1: San Francisco, 2013. Willie Brown, former mayor, former state assemblyman,
• 0:07
  Speaker 1: the man who oversaw construction of the downtown transit terminal,
• 0:11
  Speaker 1: sits down to write a newspaper column about that terminal
• 0:14
  Speaker 1: running roughly $ 300 million past its budget. He does not apologize.
• 0:20
  Speaker 1: He does not blame contractors or weather or the unforeseeable.
• 0:24
  Speaker 1: He explains the method. Nobody involved ever believed the original estimate.
• 0:29
  Speaker 1: he says. There was never a real number for the
• 0:32
  Speaker 1: terminal or for the subway before it, or the Bay
• 0:35
  Speaker 1: Bridge before that. The first budget is a down payment
• 0:39
  Speaker 1: on the eventual price. And if the eventual price had
• 0:42
  Speaker 1: been printed on the page at the start, the city
• 0:45
  Speaker 1: would have built nothing at all. So you get the
• 0:47
  Speaker 1: contract signed, and you get shovels into the ground fast,
• 0:51
  Speaker 1: and you make the hole large enough that filling it
• 0:54
  Speaker 1: becomes the only decision anyone has left. That column sits
• 0:58
  Speaker 1: inside a book about why big projects go wrong, and
• 1:02
  Speaker 1: I think it is the most honest passage in the book.
• 1:05
  Speaker 1: The book is How Big Things Get Done, published in
• 1:09
  Speaker 1: 2023 by Bent Flievoobjerg and Dan Gardner. And I should
• 1:13
  Speaker 1: say early that it is the best thing I have
• 1:15
  Speaker 1: read on its subject, because I'm going to disagree with it,
• 1:19
  Speaker 1: and I would rather do that from inside admiration than
• 1:22
  Speaker 1: from outside. Flievoobjerg quotes Brown as evidence of a pathology,
• 1:27
  Speaker 1: Brown is not describing a pathology. He is describing a
• 1:31
  Speaker 1: machine that works. Flavie Jurg came to all of this
• 1:36
  Speaker 1: through a bridge. In the early 1990s, Denmark was building
• 1:40
  Speaker 1: the Great Belt Link between two of its islands, an
• 1:43
  Speaker 1: enormous piece of engineering for a small country. And the
• 1:47
  Speaker 1: costs were behaving strangely. And a young Danish researcher started
• 1:51
  Speaker 1: keeping his own record of them. The official numbers and
• 1:55
  Speaker 1: his numbers did not agree. Then the National Audit Office
• 1:59
  Speaker 1: looked into it and confirmed his. That is a small
• 2:03
  Speaker 1: moment in a career, and it set the direction of
• 2:06
  Speaker 1: everything after it. Because what he learned was that the
• 2:10
  Speaker 1: numbers surrounding a big project are not one set of figures.
• 2:14
  Speaker 1: They are a blizzard of figures, produced at different moments
• 2:18
  Speaker 1: by different parties with different interests. And choosing which ones
• 2:22
  Speaker 1: to compare is already half the argument. So he kept counting.
• 2:27
  Speaker 1: That habit became a database, and the database is now
• 2:31
  Speaker 1: the largest of its kind anywhere. More than 16,000 projects
• 2:37
  Speaker 1: across more than 20 types of work in 136 countries. Rail, roads, tunnels, dams, ports,
• 2:46
  Speaker 1: defense systems, software, nuclear plants, the Olympic Games. For each
• 2:53
  Speaker 1: one he recorded the budget at the moment the decision
• 2:55
  Speaker 1: to build was taken immediately. and the number at the
• 2:58
  Speaker 1: end in real terms. That second detail is less dull
• 3:02
  Speaker 1: than it sounds. Projects quietly re-baseline themselves partway through, adopting
• 3:08
  Speaker 1: a new and larger budget as though it had always
• 3:11
  Speaker 1: been the plan. And measuring against the re-baseline figure makes
• 3:15
  Speaker 1: most of the damage vanish. He refused to let it vanish.
• 3:20
  Speaker 1: Here is what the pile says. Just under half of
• 3:23
  Speaker 1: those 16,000 projects came in on budget. Only 8.5% came
• 3:29
  Speaker 1: in on budget and on time, and the fraction that
• 3:33
  Speaker 1: came in on budget, on time, and delivered the benefits
• 3:36
  Speaker 1: promised when the thing was approved is one half of 1%,
• 3:41
  Speaker 1: 1 in 200. Flyvoob Jerg has a phrase for this,
• 3:45
  Speaker 1: and the whole book stands on it. Over budget, over time,
• 3:50
  Speaker 1: under benefits, over and over again. He calls it the
• 3:54
  Speaker 1: iron law. And the word iron earns its place, because
• 3:58
  Speaker 1: a pattern that holds across 16,000 cases in 136 countries
• 4:04
  Speaker 1: is not a story about carelessness in any particular country.
• 4:09
  Speaker 1: The prescription is four words long. Think slow. Act fast.
• 4:15
  Speaker 1: Divide any project into planning and delivery. Then notice that
• 4:20
  Speaker 1: almost everyone runs those two phases in the wrong proportion.
• 4:24
  Speaker 1: Planning feels like delay, and delay feels like weakness, so
• 4:28
  Speaker 1: planning gets compressed. Delivery feels like progress, so delivery gets
• 4:34
  Speaker 1: started early and then drags for years, while the unresolved
• 4:38
  Speaker 1: questions from the planning phase come due one at a time,
• 4:42
  Speaker 1: in public, at full price. The Empire State Building is
• 4:46
  Speaker 1: the counterexample the book keeps returning to, 21 months from
• 4:50
  Speaker 1: sketch to opening ceremony. estimated at $ 50 million and delivered
• 4:56
  Speaker 1: for 41, which is 17% under, and the construction finished
• 5:00
  Speaker 1: several weeks before anyone was scheduled to cut a ribbon.
• 5:04
  Speaker 1: Sydney runs the same clock in reverse. In 1957, a
• 5:09
  Speaker 1: 38-year-old Danish architect named Jorn Utzon won an international competition
• 5:15
  Speaker 1: with drawings for a building that nobody yet knew how
• 5:18
  Speaker 1: to construct. That alone was survivable. What was not survivable
• 5:24
  Speaker 1: was the politics around the schedule. The Premier of New
• 5:27
  Speaker 1: South Wales, Joe Cahill, was near the end of his career,
• 5:31
  Speaker 1: and he wanted the work visibly underway before any future
• 5:35
  Speaker 1: government could think about cancelling it, which meant construction started
• 5:39
  Speaker 1: while the design was still an open question. They thought fast,
• 5:44
  Speaker 1: and then they were forced to act slow. The building
• 5:46
  Speaker 1: was planned for five years and took fourteen. It came
• 5:50
  Speaker 1: in at 1,400% over cost. Utzon was removed in 1966
• 5:55
  Speaker 1: by the elegant method of the government simply ceasing to
• 5:59
  Speaker 1: pay him, and he went home to Denmark and never
• 6:02
  Speaker 1: returned to Australia, and he never stood inside the finished thing.
• 6:07
  Speaker 1: Notice what the rush was for. Cahill was not being
• 6:11
  Speaker 1: reckless about the whole. The whole was the strategy, the
• 6:16
  Speaker 1: same one Brown would describe in a newspaper half a
• 6:18
  Speaker 1: century later. Flybutjer calls the general case the commitment fallacy,
• 6:24
  Speaker 1: a decision locked in before the alternatives have been examined
• 6:28
  Speaker 1: and then defended long after the evidence has turned, because
• 6:32
  Speaker 1: so much has been spent that stopping looks like waste
• 6:35
  Speaker 1: rather than like the end of waste. He puts a
• 6:38
  Speaker 1: Pritzker-winning French architect on the page saying much the same
• 6:43
  Speaker 1: thing about his own country, that the opening budget is
• 6:46
  Speaker 1: a political figure rather than a technical one. Released because
• 6:50
  Speaker 1: it is the sum that can be approved, with the
• 6:53
  Speaker 1: real price disclosed later by politicians at a time of
• 6:56
  Speaker 1: their choosing. Two continents, two industries, the same confession. Nobody
• 7:02
  Speaker 1: in these quotations sounds guilty. They sound like professionals explaining
• 7:06
  Speaker 1: a procedure. And this is the point where the book
• 7:10
  Speaker 1: turns into something harder than a management argument. Overruns are
• 7:14
  Speaker 1: not spread the way intuition expects, in a gentle curve
• 7:18
  Speaker 1: around some average with a little bad luck at the edges.
• 7:22
  Speaker 1: The distribution has a tail, and nearly all the money
• 7:25
  Speaker 1: lives in the tail. Software is the clearest case. Most
• 7:30
  Speaker 1: technology projects overrun by an irritating but survivable amount. Then
• 7:35
  Speaker 1: about 18% of them go more than 50% over. And
• 7:39
  Speaker 1: within that, 18%, the average overrun is 447%. Not 50. 447.
• 7:49
  Speaker 1: Which means the average project, the one every forecast is
• 7:54
  Speaker 1: quietly describing, does not exist in any useful sense. What
• 7:59
  Speaker 1: you are buying when you approve a large one is
• 8:02
  Speaker 1: mostly a normal outcome plus a small probability of an
• 8:05
  Speaker 1: event that can take down the organization that approved it.
• 8:09
  Speaker 1: Dams behave this way and so do the Olympic Games,
• 8:13
  Speaker 1: which is why host cities keep discovering the same surprise
• 8:16
  Speaker 1: with the same expression on their faces. Roads and pipelines
• 8:20
  Speaker 1: and solar installations behave much better and the difference is
• 8:25
  Speaker 1: neither ambition nor competence. The cure is smallness. Repeated smallness
• 8:33
  Speaker 1: at scale. The projects that behave well are the ones
• 8:37
  Speaker 1: built out of one modest thing produced over and over,
• 8:41
  Speaker 1: where the thousandth unit is cheaper and faster than the
• 8:44
  Speaker 1: first because a thousand units of learning sit behind it.
• 8:49
  Speaker 1: A solar farm is a panel, repeated. Madrid built metro
• 8:53
  Speaker 1: line after metro line at remarkable speed and cost under
• 8:57
  Speaker 1: a director who standardized the stations and refused custom architecture.
• 9:02
  Speaker 1: and the system got better at building itself each time.
• 9:05
  Speaker 1: What has no repeatable unit, the one-off cathedral of a project,
• 9:11
  Speaker 1: the Olympics, the bespoke nuclear plant, is precisely what occupies
• 9:16
  Speaker 1: the tale. And the Empire State Building belongs in this family,
• 9:21
  Speaker 1: which is the detail I keep turning over. The men
• 9:24
  Speaker 1: who put it up said afterward that they had not
• 9:26
  Speaker 1: really built a skyscraper. They had built one floor a
• 9:30
  Speaker 1: hundred and sometimes, and stacked it. And the most famous
• 9:34
  Speaker 1: act of architectural ambition in the 20th century turns out
• 9:38
  Speaker 1: to have been a manufacturing exercise in a limestone coat.
• 9:42
  Speaker 1: That idea has changed how I look at work I
• 9:45
  Speaker 1: am responsible for. It gives you one question to ask
• 9:49
  Speaker 1: before committing to anything, and the question is whether the
• 9:52
  Speaker 1: thing has a unit. If it does, the risk is
• 9:56
  Speaker 1: roughly knowable and the learning curve is on your side.
• 9:59
  Speaker 1: If it does not, you are standing at the edge
• 10:01
  Speaker 1: of a distribution with no meaningful center, and confidence is
• 10:05
  Speaker 1: no defense against that. I did not have this distinction
• 10:09
  Speaker 1: before the book, and it is worth the price of
• 10:12
  Speaker 1: admission by itself. Which brings me back to that column
• 10:16
  Speaker 1: in the Chronicle. Flyvoobjerg knows exactly what he has in
• 10:20
  Speaker 1: his hands there. He has a technical name for it,
• 10:23
  Speaker 1: strategic misrepresentation, the deliberate distortion of information to serve an interest,
• 10:29
  Speaker 1: and he sets it beside the psychological explanation, Kahneman's optimism bias.
• 10:35
  Speaker 1: He and Kahneman argued about the balance between the two
• 10:38
  Speaker 1: for years. The book records how that argument resolved, and
• 10:42
  Speaker 1: the resolution is a lovely piece of intellectual honesty. Flavogerg
• 10:47
  Speaker 1: had been pressing for politics and Kahneman for psychology, and
• 10:51
  Speaker 1: each ended up conceding the other's ground. Flavogerg moved toward
• 10:56
  Speaker 1: the psychology. I think he moved the wrong way. If
• 11:00
  Speaker 1: Brown is telling the truth, and nothing in 16,000 projects
• 11:04
  Speaker 1: suggests he is not, then the forecasts in that database
• 11:08
  Speaker 1: did not fail. They performed well. A number manufactured to
• 11:12
  Speaker 1: clear an approval committee has done its entire job the
• 11:15
  Speaker 1: moment the committee approves, and the overrun that arrives four
• 11:18
  Speaker 1: years later refutes nothing. It is the plan, arriving on schedule.
• 11:23
  Speaker 1: You cannot fix that with a better estimating technique, because
• 11:27
  Speaker 1: it was never an estimate. Optimism is an error, and
• 11:31
  Speaker 1: errors can be trained out of people. What Brown is
• 11:34
  Speaker 1: describing has no error in it at all. And the
• 11:38
  Speaker 1: form of the book cannot carry this, which is the
• 11:41
  Speaker 1: real cost of writing it as a trade book. It
• 11:44
  Speaker 1: closes with 11 heuristics addressed to you, the reader, the
• 11:49
  Speaker 1: person running a renovation or a software migration. Take the
• 11:53
  Speaker 1: outside view. Know that your biggest risk is yourself. All
• 11:57
  Speaker 1: of it sound and all of it written on the
• 12:00
  Speaker 1: assumption that the person receiving the advice wants a true number.
• 12:05
  Speaker 1: The people who produce the false numbers wanted the false numbers.
• 12:09
  Speaker 1: Fleivjerg's most effective remedy is not a heuristic at all.
• 12:14
  Speaker 1: It is institutional. He got reference class forecasting written into
• 12:18
  Speaker 1: British government practice so that departments are obliged to apply
• 12:23
  Speaker 1: real base rates whether the minister enjoys it or not.
• 12:26
  Speaker 1: That is a political fix for a political disease, and
• 12:30
  Speaker 1: it sits nearly hidden inside a book selling a cognitive
• 12:34
  Speaker 1: fix for a cognitive one. There is an older argument
• 12:38
  Speaker 1: on the other side of this that deserves its hearing.
• 12:41
  Speaker 1: Albert Hirschman, the economist, proposed what he called the principle
• 12:46
  Speaker 1: of the hiding hand. We badly underestimate how difficult things
• 12:50
  Speaker 1: will be, he said, and we equally underestimate our own
• 12:54
  Speaker 1: ingenuity once we are trapped inside the difficulty and the
• 12:58
  Speaker 1: two errors roughly cancel. On that view, ignorance is providential.
• 13:03
  Speaker 1: because nobody would start anything if they could see the
• 13:06
  Speaker 1: whole of it in advance. Flivebjerg took that claim to
• 13:10
  Speaker 1: his data and found it false. The underestimation of difficulty
• 13:15
  Speaker 1: is vast and reliable. The creative rescue is occasional and
• 13:20
  Speaker 1: far too small to pay the bill. He is right.
• 13:24
  Speaker 1: And being right leaves me standing in front of the
• 13:26
  Speaker 1: Sydney Opera House with nothing useful to say. It is
• 13:30
  Speaker 1: the book's central case of failures. And it is also
• 13:33
  Speaker 1: the most loved building in the Southern Hemisphere, the picture
• 13:37
  Speaker 1: an entire country uses to say its own name. Vlaivo
• 13:41
  Speaker 1: Juric answers this, and he answers it well. Survivorship? He says,
• 13:47
  Speaker 1: you see the opera house, and you do not see
• 13:50
  Speaker 1: the hundred ruined projects that never became postcards. And Utzon
• 13:55
  Speaker 1: paid the real bill. A career ended young. his life's
• 13:59
  Speaker 1: greatest work finished by other hands while he sat in Denmark.
• 14:03
  Speaker 1: That is a cautionary tale that got lucky with its architecture.
• 14:08
  Speaker 1: I believe him. I also noticed that his answer only
• 14:11
  Speaker 1: works from a long way up, where you are counting
• 14:14
  Speaker 1: outcomes across populations. Bring it down into the room in 1957,
• 14:19
  Speaker 1: with the drawings on the table and a young man
• 14:21
  Speaker 1: who has never built anything at this scale, and the
• 14:24
  Speaker 1: correct application of everything in this book returns one instruction.
• 14:29
  Speaker 1: which is, do not do this. The method that would
• 14:32
  Speaker 1: have saved Utsun's career is the method that would have
• 14:34
  Speaker 1: left the harbor empty. If this kind of thinking is
• 14:38
  Speaker 1: useful to you, the slow kind rather than the quick kind,
• 14:42
  Speaker 1: follow the show. No fast summaries here. Books taken seriously.
• 14:48
  Speaker 1: So I hold both, and I suspect FlyVubeJerk does too,
• 14:53
  Speaker 1: somewhere he does not put in print. The numbers are true.
• 14:56
  Speaker 1: The building is beautiful. Those two facts are not going
• 15:00
  Speaker 1: to reconcile, and a book honest enough to quote Willie
• 15:04
  Speaker 1: Brown at full length is honest enough to have known that.
• 15:08
  Speaker 1: What I am left with is the shape of the trade,
• 15:11
  Speaker 1: and it is not a comfortable shape. The honest world,
• 15:15
  Speaker 1: the one where every forecast carries its real base rate
• 15:18
  Speaker 1: and every committee sees the true figure before voting, is
• 15:22
  Speaker 1: a world with fewer catastrophes in it and fewer harbors
• 15:26
  Speaker 1: worth photographing. Underneath almost everything worth crossing a city to
• 15:30
  Speaker 1: look at, there is a hole that somebody dug on purpose,
• 15:35
  Speaker 1: using a number they knew was wrong. That is the scandal,
• 15:39
  Speaker 1: and Flavjerg has measured it more carefully than anyone alive.
• 15:44
  Speaker 1: It may also be the only reason any of it
• 15:47
  Speaker 1: is standing still.