Latest / Masters in Business / Barry Ritholtz's Masters in Business: Philip Tetlock Interview

Transcript

Barry Ritholtz's Masters in Business: Philip Tetlock Interview · Masters in Business

• 0:03
  Speaker 1: This is Masters in Business with Barry Ridholts on Bloomberg Radio.
• 0:08
  Speaker 1: This week on the podcast, I have a really fascinating guest.
• 0:11
  Speaker 1: His name is Philip Tetlock. Here's a professor of loosely
• 0:16
  Speaker 1: let's call its psychology and political science at uh both
• 0:20
  Speaker 1: Wharton and the Arts and Sciences School at the University
• 0:23
  Speaker 1: of Pennsylvania. Professor Tatlock is really a fascinating guy. I
• 0:28
  Speaker 1: first got to know of his work through a book
• 0:32
  Speaker 1: he wrote well over a decade UH ago called Expert
• 0:36
  Speaker 1: Political Judgment, and it really cast an enormous shadow of
• 0:40
  Speaker 1: doubt on all of these self proclaimed experts and and
• 0:45
  Speaker 1: he used the field of political science as his basic
• 0:49
  Speaker 1: investigation arena, but really it applied to everything from investing
• 0:55
  Speaker 1: to economics to anywhere people prognosticate about the future and
• 1:01
  Speaker 1: make um confident sounding assertions about what's going to happen
• 1:05
  Speaker 1: one to five ten years in the future. He pretty
• 1:09
  Speaker 1: much destroyed that entire line of thinking and showed that
• 1:14
  Speaker 1: the average so called expert is no better than the
• 1:17
  Speaker 1: average person walking down the street. That was well over
• 1:20
  Speaker 1: a decade ago, and he tells an absolutely fascinating story
• 1:25
  Speaker 1: of how we're familiar with DARPA, nette the Defense Research
• 1:29
  Speaker 1: Projects invented the Internet and all these other really cool things. Well,
• 1:33
  Speaker 1: the intelligence community has their own version of DARPA and
• 1:36
  Speaker 1: it's called IARPA. And they basically approached Professor Tetlock and said,
• 1:41
  Speaker 1: we're fascinated by your work and we'd like to find
• 1:43
  Speaker 1: out if there are a way to to either identify
• 1:48
  Speaker 1: or develop those outlawyers in your studies who actually turned
• 1:53
  Speaker 1: out to be pretty good forecasters. And as it turns out,
• 1:57
  Speaker 1: forecasting is a humble skill that can be I don't
• 2:02
  Speaker 1: want to say learned by just about anybody, but you
• 2:06
  Speaker 1: can undertake a number of steps to make your own
• 2:11
  Speaker 1: forecasting better than it might have been otherwise. And it's
• 2:16
  Speaker 1: fairly rational and fairly straightforward and really really fascinating. And
• 2:21
  Speaker 1: so you'll hear the story of of not only how
• 2:24
  Speaker 1: the original book came about, but how the new book
• 2:28
  Speaker 1: came about and how he modified the previous view. It
• 2:32
  Speaker 1: turns out, looking twelve months and further out, it's still
• 2:36
  Speaker 1: pretty much a crapshoot. No one knows what's going to happen.
• 2:39
  Speaker 1: But on shorter periods of time and on very very
• 2:42
  Speaker 1: specific things, you can undertake a number of steps that
• 2:45
  Speaker 1: will help your own ability to discern probable outcomes be
• 2:52
  Speaker 1: much better. I think if you're at all a statistics
• 2:55
  Speaker 1: or probability want you're gonna find this absolutely fascinating. Anyone
• 2:59
  Speaker 1: who's an investor, a trader, who who deals with market
• 3:04
  Speaker 1: or economic forecast might also find this to be really
• 3:07
  Speaker 1: really interesting. So, without any further ado, my conversation with
• 3:11
  Speaker 1: Professor Philip Tetlock. This is Masters in Business with Barry
• 3:18
  Speaker 1: Ridholts on Bloomberg Radio. This week on Masters in Business
• 3:22
  Speaker 1: on Bloomberg Radio, I have a special guest, Philip Tetlock,
• 3:26
  Speaker 1: professor at the University of Pennsylvania, where he is cross
• 3:30
  Speaker 1: appointed at both the Wharton School and the School of
• 3:32
  Speaker 1: Arts and Sciences. Perhaps best known as the author of
• 3:37
  Speaker 1: Expert Political Judgment, How Good Is It and How Can
• 3:40
  Speaker 1: We Know? Which has won a number of awards. We'll
• 3:43
  Speaker 1: talk about that in a little while. His most recent book,
• 3:47
  Speaker 1: also winning accolades, is called super Forecasting, The Art and
• 3:51
  Speaker 1: Science of Prediction. The Economist magazine named it to one
• 3:55
  Speaker 1: of its best book lists of Professor Tatlock is also
• 4:00
  Speaker 1: a co principal investigator of the Good Judgment Project, which
• 4:04
  Speaker 1: is a multi year study looking at improving the accuracy
• 4:09
  Speaker 1: of probability judgments of real world high stake events. Professor
• 4:13
  Speaker 1: tat Luck, Welcome to Bloomberg. Oh, thank you, great to
• 4:16
  Speaker 1: be here. So I'm familiar with you from your your
• 4:20
  Speaker 1: earlier work, some of your earlier books, which I found
• 4:23
  Speaker 1: absolutely fascinating. But let's just start this segment talking very
• 4:28
  Speaker 1: very generally. How come there's such an enormous appetite for
• 4:32
  Speaker 1: political and economic punditry, including all the predictions and forecasts
• 4:38
  Speaker 1: that go with that. Well, it would be deeply dissonant
• 4:41
  Speaker 1: for people to um to come to the conclusion that
• 4:46
  Speaker 1: it doesn't matter who's elected president. It doesn't matter whether
• 4:49
  Speaker 1: we raise taxes or cut them, whether we help the
• 4:51
  Speaker 1: Ukraine or don't help the Ukraine, or intervene in Syria,
• 4:54
  Speaker 1: or don't intervene in Syria, or assign free trade packs
• 4:56
  Speaker 1: or don't do that. Um, we could just toss a coin,
• 4:59
  Speaker 1: because nobody the is on either side of the political
• 5:02
  Speaker 1: spectrum has any demonstrated ability to assign good probabilities to
• 5:06
  Speaker 1: the consequences of those policy options. Uh So it's to
• 5:09
  Speaker 1: say that sounds nihilistic, it's dissonant. Um people just don't
• 5:14
  Speaker 1: want to live that way. People don't want to live
• 5:16
  Speaker 1: that They want structure. And for the same reason that
• 5:19
  Speaker 1: people have turned to psychics and which doctors and all
• 5:22
  Speaker 1: sorts of things over the past. We we we we
• 5:25
  Speaker 1: we we see guidance, predictive guidance, and we seek it.
• 5:29
  Speaker 1: There's a demand for it even when the supply of
• 5:33
  Speaker 1: good forecasting is extremely limited. So we're recording this in
• 5:38
  Speaker 1: the month of March not too long ago, the cover
• 5:42
  Speaker 1: of Barons magazine has a picture of Hillary Clinton and
• 5:47
  Speaker 1: Donald Trump, which one is better for investors, and the
• 5:50
  Speaker 1: usually conservative Barons grudgingly says, we think Hillary will be
• 5:56
  Speaker 1: better for the stock market then than that Donald will.
• 5:59
  Speaker 1: Don't we run the risk of putting way too much
• 6:02
  Speaker 1: credit for good markets and good economies on the president
• 6:05
  Speaker 1: and vice versa. When things are bad. Don't we blame
• 6:08
  Speaker 1: them too much for what's gone wrong. I think there's
• 6:11
  Speaker 1: a great deal of superstitious reasoning about the impact that
• 6:14
  Speaker 1: that that leaders have on organizational performance and also the
• 6:17
  Speaker 1: impact that presidents have on on the economy. The number
• 6:21
  Speaker 1: of leaders that presidents have to influence economic outcomes is
• 6:24
  Speaker 1: quite limited um, but there is this deep intuition we
• 6:27
  Speaker 1: have that leaders should be accountable for what happens on
• 6:29
  Speaker 1: their watch, like the captain of the ship. It really
• 6:32
  Speaker 1: doesn't matter whether it's a storm suddenly emerges out of nowhere. Uh.
• 6:38
  Speaker 1: You you, you want to hold somebody accountable and and
• 6:40
  Speaker 1: that that is also deeply wired into us as it's
• 6:43
  Speaker 1: how someone has to be accountable for it. So, so
• 6:45
  Speaker 1: let's ask the obvious question, why are experts so often
• 6:49
  Speaker 1: so wrong in their forecasts. Well, there are a couple
• 6:52
  Speaker 1: of theories about that. One is that the problem lies
• 6:55
  Speaker 1: in the experts and how the experts think, uh, and
• 6:58
  Speaker 1: they could do better if they had used better analytical
• 7:02
  Speaker 1: tools and we're more self critical and creative and thoughtful.
• 7:06
  Speaker 1: And the other theory is that we just live in
• 7:08
  Speaker 1: a radically unpredictable world in which originally nobody can do
• 7:12
  Speaker 1: appreciably better than chance over extended periods of time. It's
• 7:15
  Speaker 1: a world royaled by black Swanish dark gray Swanish events
• 7:19
  Speaker 1: uh and uh, com buyer beware. Fair enough, So let's
• 7:24
  Speaker 1: talk about good judgment in predicting future events, or at
• 7:29
  Speaker 1: least good analytical processes. And along that that line, I
• 7:33
  Speaker 1: would be remiss if I didn't ask what is the
• 7:35
  Speaker 1: Briar score and why is it so important the Briar score. Well,
• 7:40
  Speaker 1: the Briar score was originally developed by some statisticians UH
• 7:44
  Speaker 1: working with meteorologists who wanted to keep score. And this
• 7:48
  Speaker 1: goes all the way back to and it's a very
• 7:51
  Speaker 1: simple idea. UH. You want to minimize the gaps between
• 7:54
  Speaker 1: probability and reality. So your code reality is either zero
• 7:58
  Speaker 1: or one, depending on whether the event occurred or didn't occur,
• 8:01
  Speaker 1: and you have probability judgments that range from zero to one,
• 8:05
  Speaker 1: and you you get a really good Brier score if
• 8:08
  Speaker 1: you assign probabilities very close to zero to things that
• 8:11
  Speaker 1: don't happen, and probability is very close to one point
• 8:14
  Speaker 1: zero two things that do happen. So it's your ability
• 8:17
  Speaker 1: to be justifiably decisive to make appropriately extreme judgments as
• 8:21
  Speaker 1: the circumstances dictate, but to avoid driving off a cliff
• 8:25
  Speaker 1: in the pursuit of that objective by making extremely overconfident
• 8:29
  Speaker 1: judgments and saying ntent of things that don't happen and
• 8:32
  Speaker 1: of things that do well. Once you're thinking in terms
• 8:35
  Speaker 1: of probability, aren't you already light years ahead of the
• 8:39
  Speaker 1: folks who are that decisive in making these bolds outlawyer declarations.
• 8:44
  Speaker 1: But isn't that much more nuanced than thoughtful? Then here's
• 8:47
  Speaker 1: what's gonna happen A And of course there's a big
• 8:51
  Speaker 1: possibility that it just doesn't go that way. That's right.
• 8:54
  Speaker 1: The probability scale and principle, if you want to look
• 8:56
  Speaker 1: at it where a mathematician would is infinitely divisible, an
• 9:00
  Speaker 1: infinite number of points between zero and one point zero.
• 9:02
  Speaker 1: And obviously people can't distinguish that many levels of uncertainty. Now,
• 9:06
  Speaker 1: when IBM S. Watson was playing in the Jeopardy competition
• 9:10
  Speaker 1: and beat the best human players, uh, you might have
• 9:14
  Speaker 1: noticed that under its answers, occasionally there would be this
• 9:16
  Speaker 1: little uh Baysian probability estimate of how confident Watson was
• 9:20
  Speaker 1: in his answer at point eight seven three six two
• 9:22
  Speaker 1: or something. Uh So, these these types of form forms
• 9:27
  Speaker 1: of artificial intelligence do try to uh make extremely granular
• 9:31
  Speaker 1: distinctions among degrees of uncertainty. Human beings can't make that
• 9:34
  Speaker 1: many uh distinctions among degrees of uncertainty in most environments.
• 9:38
  Speaker 1: I mean, there's some environments where we can pull out
• 9:40
  Speaker 1: a calculator and do it. If you can do it,
• 9:42
  Speaker 1: with card tricks and so forth um or poker um.
• 9:45
  Speaker 1: But there are there are real limits on how granular
• 9:48
  Speaker 1: you can become when it comes to whether there's going
• 9:50
  Speaker 1: to be a country leaving the Eurozone in the next year,
• 9:52
  Speaker 1: or whether there's going to be a violent Sino Japanese
• 9:54
  Speaker 1: class in the East China Sea, or things of that sort.
• 9:58
  Speaker 1: How many degrees can you distinguish their Is it just
• 10:00
  Speaker 1: yes or no? Or that yes maybe no? Or can
• 10:02
  Speaker 1: you make finer distinctions than that? I'm Barry Ridhults. You're
• 10:05
  Speaker 1: listening to Masters in Business on Bloomberg Radio. My special
• 10:08
  Speaker 1: guest today is Professor Philip Tetlock of the University of Pennsylvania.
• 10:12
  Speaker 1: He is the author of a number of books, but
• 10:15
  Speaker 1: the one I want to talk about right now was
• 10:18
  Speaker 1: the award winning Expert Political Judgment, How good is it
• 10:22
  Speaker 1: and how can we know? Uh? This won a number
• 10:24
  Speaker 1: of awards, the Graumeyer Award for Ideas Improving Political Order,
• 10:30
  Speaker 1: the Woodrow Wilson Foundation Award for Political Science, and the
• 10:33
  Speaker 1: Robert E. Lan Award for a Political Psychology. Let's let's
• 10:39
  Speaker 1: start with a quote of yours that I want to
• 10:41
  Speaker 1: get some some feedback on people who make predictions in
• 10:46
  Speaker 1: their business who appear as experts on TV get quoted
• 10:50
  Speaker 1: in newspaper articles advised, governments and businesses are no better
• 10:55
  Speaker 1: than the rest of us at making forecasts. How is
• 10:59
  Speaker 1: that possible? It turns out that you reached the point
• 11:02
  Speaker 1: of diminishing marginal returns for knowledge quite quickly and a
• 11:06
  Speaker 1: lot of the domains we care the most about. Now
• 11:08
  Speaker 1: what does that mean? UM? When I started off doing
• 11:12
  Speaker 1: research on political judgment, UM, one of the greatest psychologists
• 11:16
  Speaker 1: and on the planet advised me, Daniel Kahneman and the
• 11:20
  Speaker 1: Nobel Prize award winning psychologists most recent book, Thinking Fast Right.
• 11:27
  Speaker 1: And it was a lunchtime conversation about thirty years ago
• 11:31
  Speaker 1: in which he said rather casually that he thought that
• 11:34
  Speaker 1: the experts I was interviewing from my early work on
• 11:36
  Speaker 1: expert political judgment would have a hard time doing better
• 11:39
  Speaker 1: than UM an attentive reader of the New York Times. UM,
• 11:43
  Speaker 1: which is, you know, a kind of a fancy way
• 11:45
  Speaker 1: of saying more or less what you just said. Uh. Now,
• 11:49
  Speaker 1: he he didn't know that as a fact. He was
• 11:51
  Speaker 1: offering that as an hypothesis. Uh. And I think the
• 11:54
  Speaker 1: right way to look at this is, UM. It is
• 11:57
  Speaker 1: an hypothesis we can be continually testing. It's it's not
• 12:00
  Speaker 1: always going to be the case that experts are going
• 12:02
  Speaker 1: to fall short, but they're going to fall short much
• 12:05
  Speaker 1: more often than we would expect. Um well, how much
• 12:10
  Speaker 1: of that is random? And when they're right and at
• 12:13
  Speaker 1: a certain point don't don't the either investing or let's
• 12:18
  Speaker 1: call it, voting public have a reasonable belief that the
• 12:22
  Speaker 1: supposed experts know what they're talking about. When they make
• 12:25
  Speaker 1: a forecast. They expect them to be considerably better than
• 12:30
  Speaker 1: I think as as someone once called it a dart
• 12:33
  Speaker 1: throwing monkey. Right, Well, there's a there's the big question
• 12:36
  Speaker 1: that we want the answer to, and then they're all
• 12:39
  Speaker 1: these proxy cues that we kind of latch onto in
• 12:43
  Speaker 1: the hope that those proxy cues will get us closer
• 12:45
  Speaker 1: to the answer. So the big question we want the
• 12:47
  Speaker 1: answer to say is whether the U. S. Economy is
• 12:49
  Speaker 1: going into her session next year, whether that the Dow
• 12:53
  Speaker 1: is going to be over twenty thousand or under fifteen thousand.
• 12:56
  Speaker 1: There there's some big questions that people in the financial
• 12:59
  Speaker 1: community or political community want answers to, And there are
• 13:03
  Speaker 1: various people who passed through our lives passed through your
• 13:06
  Speaker 1: radio station, passed there in the op ed pages of newspapers,
• 13:09
  Speaker 1: on television and so forth, who offer opinions on these things,
• 13:13
  Speaker 1: and they come with various types of credentials. You might say,
• 13:16
  Speaker 1: so and so is the muckety muck professor a bl
• 13:18
  Speaker 1: bludy blum, or you might say that so and so
• 13:21
  Speaker 1: when a Nobel prize. You might say that so and
• 13:22
  Speaker 1: so is worth ten billion dollars, or you might do
• 13:25
  Speaker 1: a lot of things you could say about that and um.
• 13:29
  Speaker 1: The The interesting question is, uh, do those things give
• 13:32
  Speaker 1: us much guidance on how accurate what the person is saying?
• 13:35
  Speaker 1: So we're hoping that they do. We're we're hoping to say, well,
• 13:38
  Speaker 1: this person must know what he or she is talking
• 13:40
  Speaker 1: about by virtue of the fact that this person has
• 13:42
  Speaker 1: done X, Y or z. But the relationship between having
• 13:46
  Speaker 1: done X, Y or Z and accuracy is unknown. And
• 13:49
  Speaker 1: the more honest we are about our ignorance, the more
• 13:52
  Speaker 1: honest we are about when we're using proxy cues for
• 13:55
  Speaker 1: judging how credible source of advice is, the better off
• 13:58
  Speaker 1: we're going to be in the long term. And and
• 14:00
  Speaker 1: by the way that applies to me too, that's fascinating.
• 14:03
  Speaker 1: So so let me ask, um, when you put out
• 14:07
  Speaker 1: expert political judgment, had anyone really done a full on
• 14:12
  Speaker 1: quantitative analysis of how accurate experts were, at least in
• 14:17
  Speaker 1: the political field. Had had anyone tried to figure out, Hey,
• 14:21
  Speaker 1: let's figure out exactly how right or wrong these folks are? Before? Interestingly,
• 14:27
  Speaker 1: very little work had been done. There was a little
• 14:29
  Speaker 1: bit of work assessing the accurate Well, there's quite a
• 14:31
  Speaker 1: bit of work assessing the work of weather forecasters. There
• 14:34
  Speaker 1: was some work assessing the accuracy of expert bridge players, uh,
• 14:38
  Speaker 1: And there was some work assessing the accuracy of economist.
• 14:41
  Speaker 1: The Federal Reserve in Philadelphia and elsewhere had been doing
• 14:45
  Speaker 1: some some studies along those lines. But as for assessing
• 14:49
  Speaker 1: the accuracy of political pundits, at the time my book
• 14:52
  Speaker 1: came out, I think there was extremely little work on
• 14:55
  Speaker 1: that subject. So here's a quote from the book, and
• 14:59
  Speaker 1: I want to get some um feedback from you on this.
• 15:02
  Speaker 1: When they're wrong, they're rarely held accountable, and they rarely admitted.
• 15:07
  Speaker 1: They insist they were just off on timing, or blindsided
• 15:11
  Speaker 1: by an improbable events, or almost right or wrong for
• 15:15
  Speaker 1: the right reasons. Well, if you're a pundit, You're playing
• 15:21
  Speaker 1: a complicated game. Uh. If I'm a pundit on your
• 15:25
  Speaker 1: show or anyone show, and I need I need to
• 15:27
  Speaker 1: make it sound as though I know what I'm talking about.
• 15:30
  Speaker 1: I need to make it sound as though I'm telling
• 15:32
  Speaker 1: the listeners something they didn't know before. I also need
• 15:36
  Speaker 1: to preserve my long term credibility, which means I have
• 15:39
  Speaker 1: to have some escape clauses. So if the claims I
• 15:42
  Speaker 1: make about the future turn out to be wrong, I
• 15:44
  Speaker 1: need I need some way about of walking away from it.
• 15:47
  Speaker 1: So when you have this problem of um, you have
• 15:50
  Speaker 1: a career as a pundit, you need to be saying
• 15:53
  Speaker 1: something surprising, but you also need to preserve your long
• 15:56
  Speaker 1: term credibility. That's a real dilemma the pundit is in.
• 16:00
  Speaker 1: So the typical way pundits cope with this is by
• 16:03
  Speaker 1: saying something dramatic, uh, like Canada will disintegrate or the
• 16:07
  Speaker 1: Eurozone will disintegrate, or Pudin will reinvade the Ukraine. But
• 16:11
  Speaker 1: build in some waffle words like this could happen, or
• 16:15
  Speaker 1: this might happen, or there's a distinct possibility this will happen.
• 16:18
  Speaker 1: Now distinct possibility is one of those wonderful phrases, because
• 16:22
  Speaker 1: if it happens, I can say, hey, I told you
• 16:25
  Speaker 1: there was a distinct possibility. If it doesn't happen, I
• 16:27
  Speaker 1: can say, hey, I just that was possible, So I'm
• 16:30
  Speaker 1: covered either way. Uh, And that UM, it helps to
• 16:33
  Speaker 1: explain why pundits um and and indeed why traditionally people
• 16:37
  Speaker 1: in the US intelligence community as well UM have relied
• 16:40
  Speaker 1: so heavily on vague verbiage forecasting because they need to
• 16:43
  Speaker 1: be saying something that sounds informative, but they need uh
• 16:47
  Speaker 1: strategy for preserving their long term credibility at the same time.
• 16:50
  Speaker 1: So that explains why they don't admit error. Although I
• 16:54
  Speaker 1: would argue nobody bats a thousand, admitting error shows that
• 16:58
  Speaker 1: you have a little humility and recognize that it's not easy.
• 17:03
  Speaker 1: In the last minute we have in this segment, the
• 17:05
  Speaker 1: real question I have is why don't we hold these
• 17:08
  Speaker 1: folks accountable. Well, here's the thing, Barry, they don't even
• 17:11
  Speaker 1: think that they're wrong. If I if I say there
• 17:13
  Speaker 1: was a distinct possibility that Putin is going to invade
• 17:16
  Speaker 1: Estonia this this coming year, and he doesn't do it,
• 17:20
  Speaker 1: I'm gonna interpret distinct possibilities having met a very low probability,
• 17:25
  Speaker 1: and if he does do it, I'm going to interpret
• 17:27
  Speaker 1: distinct possibilities having med a very high probability. I'm gonna
• 17:30
  Speaker 1: we we tend to be somewhat self serving in our
• 17:32
  Speaker 1: own mental calculus. I say, well, you know, we we interpret,
• 17:35
  Speaker 1: we give ourselves a benefit of the doubt. On how
• 17:37
  Speaker 1: we interpret distinct possibility adjusted, we give the benefit of
• 17:40
  Speaker 1: the doubt. You know political pundits who favor our political party.
• 17:43
  Speaker 1: I'm Barry rid Helts. You're listening to Masters in Business
• 17:46
  Speaker 1: on Bloomberg Radio. My special guest today is Professor of
• 17:49
  Speaker 1: Philip Tetlock. He is the author of Expert Political Judgment,
• 17:53
  Speaker 1: How Good Is It? As well as Super Forecasters, a
• 17:56
  Speaker 1: new book that just came out to great acclaim. He
• 17:58
  Speaker 1: teaches at the Universe City of Pennsylvania, Wharton And and
• 18:02
  Speaker 1: let's jump right in to the Hedgehogs versus Fox's discussion.
• 18:07
  Speaker 1: You referenced this throughout really throughout um the second book
• 18:11
  Speaker 1: a lot, and if I recall correctly, you mentioned it
• 18:14
  Speaker 1: a few times in the first book, which I've read
• 18:17
  Speaker 1: a while ago. Uh. For those people who may not
• 18:20
  Speaker 1: have read Isaiah Berlin's essay explain to us what is
• 18:25
  Speaker 1: the hedgehog and the fox? So Isaiah Berlin was British
• 18:31
  Speaker 1: um scholar, political philosopher, political historian, philosopher who um took
• 18:38
  Speaker 1: a quote from the Greek warrior poet or Kilicus from
• 18:43
  Speaker 1: years ago and he built a really interesting essay around
• 18:47
  Speaker 1: it and the quote. One of the few surviving fragments
• 18:50
  Speaker 1: of this man's work was that the fox knows many things,
• 18:54
  Speaker 1: but the hedgehog knows one big thing. And he intended
• 18:57
  Speaker 1: that to capture different styles of things. King he thought
• 19:00
  Speaker 1: that um, some some thinkers were much closer to being
• 19:04
  Speaker 1: um uh foxes. Shakespeare, I think was one of his
• 19:07
  Speaker 1: classic examples of a fox who could just a very
• 19:11
  Speaker 1: multifaceted view of human nature, and other writers um he
• 19:15
  Speaker 1: thought could be could be pigeonholed better as hedgehogs. Now
• 19:19
  Speaker 1: we use this fox hedgehog distinction um in the work
• 19:24
  Speaker 1: on political Judgment because it rather captures rather well uh
• 19:29
  Speaker 1: different styles of thinking. Um. You could be, for example,
• 19:33
  Speaker 1: a hedgehog of very many different political sorts. You could
• 19:36
  Speaker 1: be a free market hedgehog, or you could be a
• 19:38
  Speaker 1: Marxist hedgehog. You could be an environmentalist, uh, doomster hedgehog,
• 19:45
  Speaker 1: or you could be a boomster um utopian kind of
• 19:51
  Speaker 1: you techno utopian sort of hedgehog. You're going to find
• 19:54
  Speaker 1: a cost effective substitutes for whatever we're running out of.
• 19:56
  Speaker 1: So there there are many different forms of hedgehog left right, pessimistic, optimistic,
• 20:03
  Speaker 1: and we identified many of them in the early work,
• 20:07
  Speaker 1: and we tracked their their their accuracy, and we compared
• 20:09
  Speaker 1: their accuracy to that of more fox like forecasters, and
• 20:12
  Speaker 1: we found a couple of things. One is that the
• 20:14
  Speaker 1: hedgehogs tend to have pretty bad batting average when you
• 20:17
  Speaker 1: look at all their predictions, uh, their their overall batting
• 20:20
  Speaker 1: average is pretty bad. We also found that the hedgehogs
• 20:24
  Speaker 1: tended to be more prominent, They're more attractive to the media.
• 20:27
  Speaker 1: The media like the kinds of sound bites that hedgehogs
• 20:30
  Speaker 1: can deliver. And we found that the hedge hugs um
• 20:34
  Speaker 1: also tended to have at least a few home runs. Matt,
• 20:37
  Speaker 1: if you're making a lot of pretty extreme predictions on
• 20:40
  Speaker 1: a wide range of subjects, at least a few of
• 20:41
  Speaker 1: them are almost by chance going to be accurate, whereas
• 20:45
  Speaker 1: the foxes are more making more moderate probability judgments, and
• 20:48
  Speaker 1: they have less claim on home runs. So um, you
• 20:52
  Speaker 1: get on somewhat ironic situation that the worst forecasters have
• 20:56
  Speaker 1: the greatest media prominence. Isn't that inherent to the process
• 21:00
  Speaker 1: of not only having a real specific expertise in one
• 21:05
  Speaker 1: area as opposed to being a generalist, but also making
• 21:09
  Speaker 1: those outlier forecasts. I use a slide in my presentation
• 21:13
  Speaker 1: about a particular pundit who every year for the past
• 21:17
  Speaker 1: seven years has forecast a seven like crash every year,
• 21:23
  Speaker 1: and you would think the media would eventually say, hey,
• 21:26
  Speaker 1: this guy is just consistently wrong. But it's such an
• 21:29
  Speaker 1: outrageous forecast and it gets people so excited they love
• 21:33
  Speaker 1: to bring them back on. Isn't that the nature of
• 21:37
  Speaker 1: sensationalism that the hedgehogs are going to be more, especially
• 21:41
  Speaker 1: today where everything is on clicks and views. Who's going
• 21:45
  Speaker 1: to generate more clicks a rational sober well, we don't
• 21:48
  Speaker 1: really know what's gonna happen, versus the sky is falling
• 21:52
  Speaker 1: and everybody loves that. Well, I would just suggest that
• 21:55
  Speaker 1: that people be better off if they were more honest
• 21:58
  Speaker 1: about the functions that are served by consuming different types
• 22:01
  Speaker 1: of information. So you if you said to yourself, look,
• 22:04
  Speaker 1: I want to be entertained. I want I want to
• 22:06
  Speaker 1: see somebody who's saying outrageous things and I'm gonna be
• 22:08
  Speaker 1: But but I'm not gonna base my probability judgments on them.
• 22:11
  Speaker 1: I just going to want to be entertained by these
• 22:13
  Speaker 1: amazing stories as person is going to tell about how
• 22:15
  Speaker 1: the Saudi regime is going to disintegrate and how we're
• 22:16
  Speaker 1: on the verge of World War three. Uh, this is
• 22:19
  Speaker 1: this is really entertaining stuff as opposed to listening to
• 22:21
  Speaker 1: this much more at tentative, nuanced fox like forecaster who's
• 22:24
  Speaker 1: saying on the wine hand, on the other hand, drones
• 22:27
  Speaker 1: on and on. Why there's really only about probability of
• 22:31
  Speaker 1: the Saudi regime changing in the next twenty four months.
• 22:33
  Speaker 1: Can we really view the world through the lens of
• 22:37
  Speaker 1: a of a single defining idea or is that, as
• 22:40
  Speaker 1: the disclaimer says, for entertainment purposes only. Well, we need
• 22:45
  Speaker 1: to be very clear about the functions that are being served. Uh.
• 22:49
  Speaker 1: Some of these big ideas are very useful lenses for
• 22:52
  Speaker 1: viewing the world um at particular moments in history and
• 22:56
  Speaker 1: in conjunction with other ideas. So I'm not saying that
• 22:59
  Speaker 1: the intellectual apparatus is useless. I'm saying that it's what's
• 23:03
  Speaker 1: really dangerous is when you have a smart person who
• 23:07
  Speaker 1: runs too far with a big idea and fails to
• 23:10
  Speaker 1: see that the complexity of the world puts a lot
• 23:13
  Speaker 1: of breaks on it. So one of our rules of
• 23:15
  Speaker 1: thumb for distinguishing better forecasters and worse forecasters on the
• 23:18
  Speaker 1: media is the ratio of the number of times they
• 23:21
  Speaker 1: say however versus moreover. So if you have a high
• 23:24
  Speaker 1: however over moreover ratio, that means you're a fox. That
• 23:27
  Speaker 1: means you're boring. That means are probably going to be
• 23:29
  Speaker 1: kicked off the show. And if you have more more accurate,
• 23:32
  Speaker 1: more likely to be more accurate. But that's right, you're
• 23:34
  Speaker 1: gonna have better briar score. I'm Barry Ridholts. You're listening
• 23:37
  Speaker 1: to Masters in Business on Bloomberg Radio. My special guest
• 23:41
  Speaker 1: this week, professor of phil Tetlock of the University of Pennsylvania.
• 23:45
  Speaker 1: His most recent book, super Forecasting, The Art and Science
• 23:50
  Speaker 1: of Prediction. So let's jump right into this because I
• 23:53
  Speaker 1: have so many questions about this, and let me start
• 23:56
  Speaker 1: out by just asking how many piano tuners are there Chicago,
• 24:01
  Speaker 1: UH somewhere between about eight and I think so. So
• 24:05
  Speaker 1: it's a fascinating question because your initial reaction is to
• 24:09
  Speaker 1: shrug and say, I don't know. The variation I've heard
• 24:12
  Speaker 1: of that is how many cardiac surgeons are there in London?
• 24:16
  Speaker 1: UM or what's the empire state building way? That's another
• 24:19
  Speaker 1: good question. So so let's talk. Let's talk a little
• 24:22
  Speaker 1: bit about what do most people do when presented with
• 24:26
  Speaker 1: a question like that? Interesting? You know, some of the
• 24:29
  Speaker 1: high tech firms in Silicon Valley were quite fon fond
• 24:32
  Speaker 1: of asking off off the out of left field questions
• 24:36
  Speaker 1: of this sort because they thought it was a great
• 24:38
  Speaker 1: way of testing how well people think on their feet. UM.
• 24:41
  Speaker 1: In the book, we call these Ferremi questions, named after
• 24:44
  Speaker 1: the great Italian American physicist and Rico Ferremi, who developed
• 24:47
  Speaker 1: the first um UH nuclear reactors keep part of the
• 24:50
  Speaker 1: Manhattan Project developing the bomb, and Ferremy was fond of
• 24:55
  Speaker 1: UH posing these oddball questions to his students. He what
• 24:59
  Speaker 1: do he wanted to the students to do was to
• 25:02
  Speaker 1: take an seemingly intractable problem and break it down into
• 25:06
  Speaker 1: parts or components that were more tractable, So you might
• 25:09
  Speaker 1: not have any idea how many. No one has any
• 25:12
  Speaker 1: idea initially on how many how many piano tuners that
• 25:14
  Speaker 1: might be in Chicago. But you make guestiments about the
• 25:17
  Speaker 1: population of Chicago. Walk us through that because you you
• 25:20
  Speaker 1: go through about seven steps and you get pretty close
• 25:23
  Speaker 1: to the correct answer. Right. Well, you're you're making a
• 25:26
  Speaker 1: lot of guests and it's not just the breaking down
• 25:29
  Speaker 1: and get trying to get the answer. What you're doing
• 25:32
  Speaker 1: in the process is you're revealing sources of ignorance, and
• 25:35
  Speaker 1: your colleagues on your team, for example, if it's because
• 25:38
  Speaker 1: our forecasters often work together on teams, uh, your colleagues
• 25:41
  Speaker 1: can help you correct help correct your errors. So what's
• 25:44
  Speaker 1: the population of Chicago, I don't really know, between two
• 25:47
  Speaker 1: point five and four million, and maybe you know some
• 25:51
  Speaker 1: somewhere in the middle there, what proportion of the population
• 25:54
  Speaker 1: would have a piano and so forth? You see, if
• 25:57
  Speaker 1: you would, you would break it down and you you
• 25:59
  Speaker 1: you try to mind eventually how how how many people
• 26:02
  Speaker 1: could conceivably make a living working as as piano tuners
• 26:06
  Speaker 1: given the number of people who own pianos in Chicago
• 26:09
  Speaker 1: and their willingness to pay for the services of piano
• 26:12
  Speaker 1: tuners um and so so Faremy did this. He One
• 26:16
  Speaker 1: legend is that he tried to uh infer the strength
• 26:20
  Speaker 1: of the first atomic blast um by dropping little pieces
• 26:24
  Speaker 1: of paper when the when, when the when the winds
• 26:26
  Speaker 1: came in front, and by estimating how far the winds blew.
• 26:29
  Speaker 1: And I think he was off by about forty or
• 26:32
  Speaker 1: fifty percent. But you know that for Faremy estimates, that's
• 26:35
  Speaker 1: not too bad. It's it's a lot better than simply
• 26:37
  Speaker 1: shrugging your shoulders and saying I have no idea. I
• 26:39
  Speaker 1: say a ten Kelton glass as post of twenty. But
• 26:43
  Speaker 1: that's that's fascinating. I'm absolutely entranced by Firm's paradox, which says,
• 26:48
  Speaker 1: where where is everybody? You know, it's a giant universe
• 26:51
  Speaker 1: filled with different galaxies and hundreds of billions of stars?
• 26:55
  Speaker 1: And are we really the only intelligent life here? And
• 26:58
  Speaker 1: I found most of the various arguments both ways to
• 27:02
  Speaker 1: be lacking. It's really it's really a fascinating, fascinating debate.
• 27:06
  Speaker 1: But let's let's stick with this. So so looking at
• 27:10
  Speaker 1: what the empire state building ways, or or how many
• 27:13
  Speaker 1: piano tuners in Chicago show us how to break down
• 27:19
  Speaker 1: unknown questions into component parts and make reasonable assessments and
• 27:24
  Speaker 1: reasonable valuations on each of those segments. Um. So, so
• 27:29
  Speaker 1: what do you find about teams of super forecasters? How
• 27:33
  Speaker 1: much better are they at at these sort of predictions
• 27:37
  Speaker 1: than the average person, or prediction markets or just regular pundits. Well, Um,
• 27:46
  Speaker 1: the teams have super forecasters truly astonished us because the
• 27:51
  Speaker 1: statisticians were telling us the right thing to do here
• 27:55
  Speaker 1: was to have each individual top performer make judgments completely
• 27:59
  Speaker 1: independ at lay of the others, um, rather than allowing
• 28:02
  Speaker 1: them to contaminate each other. And you get conformity, and
• 28:05
  Speaker 1: you get group think, and you got to get kind
• 28:07
  Speaker 1: of a blur rather than a number of distinct points
• 28:09
  Speaker 1: of view, and then you can combine them statistically somehow.
• 28:12
  Speaker 1: So Um, we were the only competitor in the forecasting
• 28:16
  Speaker 1: tournament sponsored by the U S Intelligence community that used
• 28:19
  Speaker 1: teams um And but we we hedged our bets. We
• 28:22
  Speaker 1: weren't sure that teams would work. We ran an experiment
• 28:25
  Speaker 1: and we randomly assigned people to teams, and we random
• 28:27
  Speaker 1: and had other people work as individuals. Uh. And we
• 28:30
  Speaker 1: were truly surprised that the teams functioned as well as
• 28:33
  Speaker 1: they did UM, and it's an interesting question of why
• 28:37
  Speaker 1: our teams were so dynamic and open minded relative to
• 28:41
  Speaker 1: many teams you see in actual organizations. But before you
• 28:45
  Speaker 1: answer that question, let's let's put a little flesh on
• 28:47
  Speaker 1: the bone with some numbers. So teams of ordinary forecasters
• 28:51
  Speaker 1: beat the wisdom of the crowd by about ten. They
• 28:55
  Speaker 1: were bested by prediction markets UM and addiction markets beat
• 29:01
  Speaker 1: ordinary teams by about and then the super teams of
• 29:05
  Speaker 1: the best forecasters beat the prediction markets by anywhere from fifteen.
• 29:12
  Speaker 1: So these folks working in groups really are the outliers.
• 29:16
  Speaker 1: None of the other groups are even close to them
• 29:19
  Speaker 1: in terms of of accuracy. Why do you think that is, Well, yeah,
• 29:24
  Speaker 1: and I'll just add one other thing to that. They
• 29:26
  Speaker 1: weren't just outperforming prediction markets in the public sphere, they
• 29:31
  Speaker 1: are also outperforming intelligence analysts who were working, you know,
• 29:34
  Speaker 1: behind a veil of of of classified information. And it's
• 29:38
  Speaker 1: just a remarkable thing. And I think US intelligence community,
• 29:40
  Speaker 1: which is much maligned for many things, deserves some credit
• 29:43
  Speaker 1: for its willingness to sponsor of forecasting tournament that has
• 29:46
  Speaker 1: the potential to be embarrassing for government bureaucracy. How often
• 29:49
  Speaker 1: do you see a government bureaucracy, uh, spend money, a
• 29:52
  Speaker 1: lot of money on a project that has the potential
• 29:54
  Speaker 1: to be to yield results that you're fairly embarrassing. How
• 29:57
  Speaker 1: hard is it to cultivate these skills or is it
• 30:00
  Speaker 1: just a matter of internalizing these ten bullet points? I
• 30:04
  Speaker 1: wouldn't say just a matter. It's it's no, it's an
• 30:08
  Speaker 1: it's a non trivial thing to do. Uh it's it's
• 30:10
  Speaker 1: it's pretty hard. Um. And uh so let's let's take
• 30:15
  Speaker 1: get an example of one. And what do you think
• 30:17
  Speaker 1: is an important um, an important commandment of super forecasting?
• 30:23
  Speaker 1: You want to just pick up one at random? Sure? Okay, Um, well,
• 30:29
  Speaker 1: one of them has to do with granularity and uh
• 30:32
  Speaker 1: it's it's it's actually grounded in a story that about
• 30:36
  Speaker 1: President Obama and how are he reacted to his advisors
• 30:39
  Speaker 1: who were um offering him um somewhat conflicting probabilities on
• 30:45
  Speaker 1: how likely it was that Osama bin Laden was residing
• 30:48
  Speaker 1: in a particular compound in Abadabad, Pakistan. As we all
• 30:51
  Speaker 1: know now, he was indeed residing there, and the President
• 30:55
  Speaker 1: did authorize a n a vcal mission, and that resulted
• 30:58
  Speaker 1: in Osama bin Lad's death. UM. Now, when the President
• 31:03
  Speaker 1: was confronted by these probability estimates from really smart people
• 31:07
  Speaker 1: at the top of the intelligence apparatus, ranging from about
• 31:11
  Speaker 1: you know, thirty five or up to about um the
• 31:16
  Speaker 1: President's reaction was an interesting one. If if you Hee
• 31:19
  Speaker 1: had computed the average of the median estimate of the
• 31:22
  Speaker 1: advice he was getting, he would have said, looks like
• 31:24
  Speaker 1: about a s probability. But instead he said something interesting.
• 31:28
  Speaker 1: He said, well, look, look, guys, this is a coin flip.
• 31:31
  Speaker 1: It's a fifty fifty thing. Um. Now, the President, I
• 31:35
  Speaker 1: think is a very intelligent person, and I think he's
• 31:38
  Speaker 1: capable of being very granular in his assessments of uncertainty.
• 31:42
  Speaker 1: And if you doubt it, think about the following thought experiment,
• 31:44
  Speaker 1: which is appropriate given it approaching March Madness. He follows
• 31:48
  Speaker 1: March Madness he uh, and basketball. He's a basketball fan.
• 31:52
  Speaker 1: Imagine he'd been sitting around with friends waiting for Duke
• 31:55
  Speaker 1: to play some team in March Matt Martin the March
• 31:58
  Speaker 1: Madness tournament, and uh, they offered him exactly the same
• 32:02
  Speaker 1: probabilities about whether Duke would win the win the game.
• 32:05
  Speaker 1: You know, somewhere between thirty five and ninety five, with
• 32:08
  Speaker 1: the center of gravity of opinion around seventy, would you
• 32:11
  Speaker 1: have said it sounds like a fifty fifty thing or
• 32:13
  Speaker 1: would you have said, MM sounds like about UM three
• 32:17
  Speaker 1: to one, Duke UM, I think to ask the question
• 32:21
  Speaker 1: is to answer it. He would have seen opportunity for
• 32:24
  Speaker 1: being much more granular in making bets about sports than
• 32:27
  Speaker 1: he would and making estimates about the likelihood of a
• 32:30
  Speaker 1: particular terrorist being in a particular location. UM. Now it
• 32:34
  Speaker 1: turns out that UM for many categories of problems where
• 32:38
  Speaker 1: we think it's impossible to be more granular, it is possible.
• 32:43
  Speaker 1: And that's one of the things super forecasters have learned
• 32:45
  Speaker 1: that there's a difference between fifty and sometimes they can
• 32:49
  Speaker 1: even make distinctions between fifty and now we quote that
• 32:53
  Speaker 1: the Chief Risk Officer of UM a q r M,
• 32:57
  Speaker 1: the hedge fund UMAST is the head of that, and
• 33:02
  Speaker 1: the chief risk Officer is Aaron Brown. And when we
• 33:06
  Speaker 1: talked with with Aaron, he you know, he he's also
• 33:09
  Speaker 1: a really serious poker player, UM, and he said, well,
• 33:13
  Speaker 1: he can tell it different. World class poker player and
• 33:15
  Speaker 1: a talented amateur on the basis that the world class
• 33:18
  Speaker 1: player UM knows the difference, and then he paused that
• 33:23
  Speaker 1: maybe it's more like UM or indeed two, how granular
• 33:30
  Speaker 1: can you get in poker? Well, poker is a game
• 33:32
  Speaker 1: with repeated play, quick clear feedback. It's possible to get
• 33:35
  Speaker 1: more granular on poker than it is about the location
• 33:38
  Speaker 1: of terrorists or about whether countries are going to leave
• 33:40
  Speaker 1: the Eurozone. But it's an open question of how granular
• 33:43
  Speaker 1: you can get UM. And you need to grapple with
• 33:46
  Speaker 1: this distinction between precision and pseudo precision UM. And that's
• 33:51
  Speaker 1: one of the things. Super forecasters are just very thoughtful
• 33:54
  Speaker 1: people who pushed the frontiers of knowledge as far as
• 33:57
  Speaker 1: they can, and that means sometimes pushing them a little
• 33:59
  Speaker 1: too far, in which case they retreat. If people want
• 34:02
  Speaker 1: to find your work, just google Philip Tetlock and they'll
• 34:04
  Speaker 1: be able to dig up all of your various publications, books, writings, etcetera. Yeah,
• 34:09
  Speaker 1: Google scholars probably a little faster, but all right. If
• 34:12
  Speaker 1: you've enjoyed this conversation, be sure and check out all
• 34:14
  Speaker 1: eighty three or so of our prior conversations. Be sure
• 34:18
  Speaker 1: and follow me on Twitter at rid Halts and check
• 34:21
  Speaker 1: out my daily column on Bloomberg View dot com. I'm
• 34:25
  Speaker 1: Barry Ridhults. You're listening to Masters in Business on Bloomberg Radio.
• 34:28
  Speaker 1: Welcome to the podcast. This is Barry Ridhills, Professor Tetlock.
• 34:32
  Speaker 1: If I don't remember to say this later, thank you
• 34:34
  Speaker 1: so much for being so generous with your time. This
• 34:36
  Speaker 1: is really um been a fascinating conversation. I have a
• 34:40
  Speaker 1: lot of things to go over with you in the
• 34:44
  Speaker 1: last twenty minutes or so we have, but there are
• 34:47
  Speaker 1: a few questions that I'm just dying to ask you
• 34:50
  Speaker 1: because it's your lat previous book really was very influential
• 34:56
  Speaker 1: to me on expert political judgment. It was that book,
• 35:00
  Speaker 1: and it was a prior book called The Fortune Sellers
• 35:03
  Speaker 1: that really was more of a media criticism of this
• 35:07
  Speaker 1: parade of people who would come through the studios make
• 35:11
  Speaker 1: their outland just forecast, be completely wrong, never be held accountable,
• 35:17
  Speaker 1: and then they would get called back and the more outlandish, uh,
• 35:20
  Speaker 1: the better there were. There was an author who wrote
• 35:23
  Speaker 1: a book I'm trying to remember what year the book was.
• 35:26
  Speaker 1: It was called The Tao Jones T. A. O. Bennett Goodspeed,
• 35:30
  Speaker 1: and he called these folks the articulate incompetence plural, meaning
• 35:36
  Speaker 1: that they're very good salespeople. They can speak, but really
• 35:40
  Speaker 1: they have no expert knowledge. And if you've spent any
• 35:43
  Speaker 1: time in green rooms in various television studios, there's something
• 35:48
  Speaker 1: to that. So let me ask the basic question that
• 35:53
  Speaker 1: we kind of skirted around during the broadcast portion. Why
• 35:58
  Speaker 1: are we so enamored with forecasting and forecasters despite their
• 36:04
  Speaker 1: terrible track records. Well, I think because there's a lot
• 36:09
  Speaker 1: of motivated reasoning going on. As we noted earlier, there's
• 36:13
  Speaker 1: this tendency to use a lot of a do a
• 36:15
  Speaker 1: lot of a verbiage forecasting, to to paint a dramatic
• 36:18
  Speaker 1: scenario and then hold it together with some very weak
• 36:21
  Speaker 1: verbs like this might happen or could happen or the
• 36:24
  Speaker 1: distinct possibility of this happening. Um. So, there's this interesting
• 36:28
  Speaker 1: tendency that the pundits have of engaging our attention with
• 36:32
  Speaker 1: a vivid scenario disintegration of the Saudi regime or you know, uh,
• 36:37
  Speaker 1: Cino Japanese wars, something Tom Clancy issue on that kind
• 36:42
  Speaker 1: of scale, um and and and but to stitch it
• 36:44
  Speaker 1: all together with terms weasel weasel word terms that allow
• 36:48
  Speaker 1: them to retreat later on. And UM, we don't distinguish
• 36:53
  Speaker 1: very clearly in our own minds. We we don't think
• 36:55
  Speaker 1: we want to hold as they say, all of the
• 36:57
  Speaker 1: fault lies with the pundits. They couldn't do this unless
• 37:01
  Speaker 1: unless we were willing partners. UM. And I think that
• 37:04
  Speaker 1: you know here here I am talking to radio station
• 37:07
  Speaker 1: one of the most influential companies in the world, Bloomberg. UM.
• 37:11
  Speaker 1: Bloomberg is a major purchaser of expertise. UM. Bloomberg could
• 37:18
  Speaker 1: actually change the world to some degree if it implemented
• 37:22
  Speaker 1: systematic uh, if it implemented systems for tracking the accuracy
• 37:27
  Speaker 1: of many of the people who came through. If part
• 37:29
  Speaker 1: of the price for getting onto Bloomberg was that you
• 37:32
  Speaker 1: had to demonstrate that you were engaging in some kind
• 37:34
  Speaker 1: of rigorous scorekeeping, and Bloomberg could flash up some of
• 37:37
  Speaker 1: some batting average statistics. UM, as you as you appear,
• 37:42
  Speaker 1: Um you Bloomberg could increase the collective IQ of our society.
• 37:46
  Speaker 1: It could increase the collective IQ of the conversation. Um.
• 37:50
  Speaker 1: When most pundits stay away, Well, well that's the question
• 37:54
  Speaker 1: I mean. I I say Bloomberg because Bloomberg is so influential,
• 37:56
  Speaker 1: I think a lot of a lot of punets would say, well,
• 37:58
  Speaker 1: I'm not I'm not going to run away and high
• 37:59
  Speaker 1: from blue Burg. But the other media could do this
• 38:02
  Speaker 1: to the Wall Street Journal in New York Times. There
• 38:04
  Speaker 1: are lots of major media science that have the leverage
• 38:08
  Speaker 1: that could induce pundits to be much more intellectually honest.
• 38:11
  Speaker 1: They choose not to exercise that option. Um probably because
• 38:16
  Speaker 1: they don't perceive a great market demand for it. We
• 38:19
  Speaker 1: were talking during the break about my usage of a
• 38:24
  Speaker 1: little app called follow up then dot com. Whenever I
• 38:27
  Speaker 1: see an outregeous forecast, I just shoot an email to
• 38:31
  Speaker 1: a specific date, so Gold going to five thousand dollars,
• 38:35
  Speaker 1: and I send the forecast. I send the email out
• 38:38
  Speaker 1: to Oh, well, let's let's let's give them a year,
• 38:42
  Speaker 1: so we'll send a forecast out an email out March
• 38:45
  Speaker 1: one at follow up then dot com with the headline
• 38:49
  Speaker 1: and the web address of the article that made this forecast.
• 38:53
  Speaker 1: And then a year later, or if I write March one,
• 38:57
  Speaker 1: five years later, comes the email back that specifically gives
• 39:01
  Speaker 1: me that lank. Oh it's a reminder. Here's what this
• 39:04
  Speaker 1: person said a year ago. And occasionally I get to
• 39:08
  Speaker 1: do an article about, Hey, here's a wild forecast that
• 39:12
  Speaker 1: someone made and it's been completely wrong. So let's talk
• 39:16
  Speaker 1: about your book. You kind of call me out for
• 39:20
  Speaker 1: calling someone else out, and I'm curious as to your
• 39:24
  Speaker 1: perspective on this. So in two thousand and ten, when
• 39:28
  Speaker 1: the FED was in the midst of doing quantitative easing, uh,
• 39:32
  Speaker 1: there was a letter published and I believe online and
• 39:35
  Speaker 1: it ran in the Wall Street Journal in a number
• 39:37
  Speaker 1: of places warning that quantitative easing was going to cause
• 39:41
  Speaker 1: hyper inflation and collapse of the dollar and all these
• 39:44
  Speaker 1: terrible things. And so I figured, you've got to give
• 39:48
  Speaker 1: those people three years. So I set a reminder for
• 39:53
  Speaker 1: three years later. And three years later it popped up. Hey,
• 39:56
  Speaker 1: the dollars at multi year highs. There there is no
• 39:59
  Speaker 1: hyper inflation, there's there's deflation. These guys were wrong, and
• 40:03
  Speaker 1: so I called them out about it, and it went
• 40:06
  Speaker 1: totally viral, got picked up by a dozen different media outlets,
• 40:09
  Speaker 1: and a book called super Forecasting. Now at by today
• 40:13
  Speaker 1: we're six years forward, um, and we still have a
• 40:17
  Speaker 1: strong dollar and and no inflation. What is the issue
• 40:23
  Speaker 1: with an ambiguous forecast? With no specific I described forecasts
• 40:29
  Speaker 1: as an asset class, a price, and a specific date.
• 40:34
  Speaker 1: If you leave out the specific date, do you get
• 40:36
  Speaker 1: to say we're never wrong because there's just wait, you'll see.
• 40:41
  Speaker 1: Is is that a fair defense of that? It's definitely
• 40:44
  Speaker 1: not fair, but it it is the state of the
• 40:47
  Speaker 1: art at the moment. You know. There's an old communism
• 40:49
  Speaker 1: joke that that that's rather a proposed here um the
• 40:53
  Speaker 1: Soviet revolutionarily on Trotsky after he was thrown out of
• 40:57
  Speaker 1: the Soviet Union by Stalin, when it went around giving
• 41:00
  Speaker 1: talks to the left wing audiences around the world, and um,
• 41:03
  Speaker 1: one probably apocryphal story has it that he went once
• 41:06
  Speaker 1: when he was introduced to an audience of followers. Uh,
• 41:10
  Speaker 1: the speaker said it was was proclaiming Leon trotskya visionary
• 41:14
  Speaker 1: who could see far, far into the future. And you're saying,
• 41:17
  Speaker 1: and you know, comrades, the ultimate proof of the far
• 41:19
  Speaker 1: sightedness of of of of comrade Trotsky, not one of
• 41:22
  Speaker 1: his predictions has yet come true. That's how far sighted
• 41:26
  Speaker 1: he is. The the old joke about market forecasting is
• 41:30
  Speaker 1: you could give a price level or a date, but
• 41:32
  Speaker 1: never both at once. And that's just another way. So
• 41:36
  Speaker 1: so how long do we allow a forecast to persist
• 41:39
  Speaker 1: before we say, all right, at this point it's been
• 41:42
  Speaker 1: X number of years, you we're gonna have to put
• 41:45
  Speaker 1: you in the incorrect column. Well, there's there's no absolute rule,
• 41:52
  Speaker 1: because that's that's the way vague verbiage is vague. Verbiage
• 41:55
  Speaker 1: is vague because it's it's just it's it's always a slippery,
• 42:00
  Speaker 1: vague um thing that no nobody knows how to quantify
• 42:04
  Speaker 1: it and and keeps it, keeps the pundit safe. Um.
• 42:07
  Speaker 1: I mean, I have lots of forecasters from the earlier
• 42:09
  Speaker 1: work who predicted that Canada would disintegrate, or Nigeria would disintegrate,
• 42:13
  Speaker 1: or there are a lot of disintegration scenarios out there
• 42:15
  Speaker 1: that haven't happened yet. Um. And if you call them
• 42:18
  Speaker 1: on it, are they going to say, well, we're it's
• 42:21
  Speaker 1: it's only seen it could still happen, absolutely and and
• 42:25
  Speaker 1: wholeheartedly believe it. That's right. So how much of this
• 42:29
  Speaker 1: is just simply humans not being immune to human behavior?
• 42:34
  Speaker 1: You're a psychologist, Let's let's let's take that tact. Is
• 42:37
  Speaker 1: this just ordinary human behavior refusal to accept responsibility for error,
• 42:43
  Speaker 1: not wanting to admit being wrong, not wanting to do
• 42:46
  Speaker 1: anything that reduces their potential um status within the hierarchy.
• 42:52
  Speaker 1: Is that all this is well, we're moving into world
• 42:55
  Speaker 1: that's requiring us to make ever subtler distinctions among degrees
• 42:58
  Speaker 1: of uncertainty. This orts of distinction as we didn't have
• 43:00
  Speaker 1: to make in our evolutionary past um when we were
• 43:04
  Speaker 1: wandering the savannah plains of Africa. You know there were
• 43:06
  Speaker 1: are there is a lion or isn't a lions during
• 43:08
  Speaker 1: in the long grass and you're gonna you're gonna make
• 43:10
  Speaker 1: you have to make a judgment call pretty darn quickly.
• 43:12
  Speaker 1: And if you dontle very long, and maybe you're not
• 43:14
  Speaker 1: likely to pass your genes onto the next generation. You're
• 43:17
  Speaker 1: better off being wrong but jumping the gun then having
• 43:21
  Speaker 1: a higher, better track record. But if you're wrong once,
• 43:24
  Speaker 1: well then it's catastrophic in terms of progeny. And that
• 43:28
  Speaker 1: makes a lot of sense. That sort of thinking is
• 43:31
  Speaker 1: why we have a tendency to do all sorts of
• 43:33
  Speaker 1: things that just are inappropriate in investing but worked really
• 43:38
  Speaker 1: well way back when that's right and um, if you
• 43:44
  Speaker 1: can imagine a scenario, here's the here's the big problem
• 43:47
  Speaker 1: with tail risks and scenarios. Um, most of the time
• 43:52
  Speaker 1: people underrate them, but as soon as the scenario has
• 43:56
  Speaker 1: called to their attention, they overrated. Well, it's very very
• 44:00
  Speaker 1: caul for people to strike the right balance in dealing
• 44:02
  Speaker 1: with tailor risk scenario. So so that whole recency effect
• 44:06
  Speaker 1: thing is since nobody, very few people were forecasting the
• 44:09
  Speaker 1: sort of financial crisis we had in O eight oh nine,
• 44:13
  Speaker 1: and since then it's been nothing but catastrophic forecasts from
• 44:18
  Speaker 1: the recession never ended, We're going to turn into a depression.
• 44:22
  Speaker 1: Here comes another eight seven like stock crash, the parade
• 44:25
  Speaker 1: of horribles just have not now auto loans and the
• 44:30
  Speaker 1: new subprime it's going to be just like, uh, is
• 44:34
  Speaker 1: this just that that tel risk factor is so recent
• 44:37
  Speaker 1: in people's minds and they missed it coming, and so
• 44:39
  Speaker 1: now they're just like every general fights the last war.
• 44:43
  Speaker 1: These folks are still fighting the previous financial crisis and
• 44:46
  Speaker 1: the diplomats try to avoid the last war. Um, yeah,
• 44:49
  Speaker 1: I think that's right. I mean, you are you you
• 44:52
  Speaker 1: you go from Iraq to Syrian Libya or you that
• 44:55
  Speaker 1: you go from one error to another. And Iraq was
• 44:58
  Speaker 1: far enough after the Aetnam that it looked like all
• 45:01
  Speaker 1: it takes as a generation before those lessons are lost,
• 45:04
  Speaker 1: more or less yet more or less. All right, so
• 45:06
  Speaker 1: I only have you for another ten minutes, and um,
• 45:10
  Speaker 1: my last question before I jumped to my my standard questions.
• 45:14
  Speaker 1: Burton Malkiel said, when investors moved from stock to stock
• 45:18
  Speaker 1: or mutual fund to funds as if they were selecting
• 45:21
  Speaker 1: and discarding cards in a game of jin rummie, what
• 45:25
  Speaker 1: does this tell us about human's ability to participate in
• 45:30
  Speaker 1: uncertain equity markets? Well, this is another one of these
• 45:34
  Speaker 1: the ten commandments that we formulated from observing the super forecasters.
• 45:38
  Speaker 1: They're acutely aware of the principle of error balancing. If
• 45:42
  Speaker 1: you look at the research literature on human judgment, there
• 45:45
  Speaker 1: are two kinds of errors people can make in that
• 45:47
  Speaker 1: situation you're describing. One of them is the error of
• 45:50
  Speaker 1: excess of volatility, of um jumping every time there's a
• 45:54
  Speaker 1: little bit of news and exaggerating its diagnostic value visa
• 45:58
  Speaker 1: deep market trends. And the other big mistake you can
• 46:01
  Speaker 1: make is excessive rigidity and being so committed to a
• 46:05
  Speaker 1: particular preconception about where the future is going that you
• 46:07
  Speaker 1: just ignore the news altogether. And you don't you fit,
• 46:09
  Speaker 1: you fit, you failed to do any updating. So it's
• 46:12
  Speaker 1: it's it's kind of principal error balance things. But like
• 46:15
  Speaker 1: writing learning how to ride a bicycle. Um, I mean,
• 46:17
  Speaker 1: I could talk for hours about the principles of error
• 46:21
  Speaker 1: balancing and everybody is like, yeah, yeah, I kind of
• 46:23
  Speaker 1: get at Sure you can make one air, you can
• 46:24
  Speaker 1: make the other. But the only way it's really going
• 46:27
  Speaker 1: to sink into people's heads if they is if they
• 46:29
  Speaker 1: go to forecasting tournaments and they actually practice making judgments,
• 46:32
  Speaker 1: get on the bike and try to ride it. Uh.
• 46:34
  Speaker 1: And that's what going to forecasting tournaments all about. That
• 46:37
  Speaker 1: that's why we were continuing to run the gj open
• 46:40
  Speaker 1: dot com, which is a forecasting tournament where people can
• 46:42
  Speaker 1: indeed uh work to cultivate their skills. That sounds pretty
• 46:46
  Speaker 1: pretty fascinating. So that's that's really interesting for that. Actually,
• 46:52
  Speaker 1: we have a little more time, so I'm going to
• 46:54
  Speaker 1: keep bangalway on some of these questions. In the first
• 46:58
  Speaker 1: book on in the previous book on Expert Political Judgment,
• 47:03
  Speaker 1: you know, I look at these is really two sides
• 47:06
  Speaker 1: to the same coin. The first book talks about what
• 47:11
  Speaker 1: is essentially long term forecasts really a year and further out,
• 47:16
  Speaker 1: and they tend to be wrong. The book on super
• 47:20
  Speaker 1: forecasting is really looking at a year or less. So
• 47:23
  Speaker 1: are are they really saying two different things where we're
• 47:27
  Speaker 1: really looking at two different types of forecast, two different lengths.
• 47:31
  Speaker 1: I think that's a superb point. There are different time
• 47:36
  Speaker 1: periods in the different studies. UM. I'm much more optimistic
• 47:40
  Speaker 1: that we can improve forecasting using the right selection, training,
• 47:43
  Speaker 1: teaming tools in shorter time periods up to about a year.
• 47:49
  Speaker 1: I become progressively more pessimistic when you go out to
• 47:52
  Speaker 1: the longer reaches of three, five, ten years that were
• 47:54
  Speaker 1: included in Expert Political Judgment, and there I think it's
• 47:58
  Speaker 1: it's going to be very, very difficult to do much better, UM,
• 48:02
  Speaker 1: do much better than chance. How many times do you
• 48:04
  Speaker 1: have to shuffle a deck of cards, um, until it's
• 48:08
  Speaker 1: perfectly random? That's a good question. Well, I think the
• 48:12
  Speaker 1: UM I would guess five. The statistician and magician Percy
• 48:17
  Speaker 1: dot Com is Stanford. I think he estimated at seven. Well,
• 48:21
  Speaker 1: life is like shuffling the cards? How how many months,
• 48:24
  Speaker 1: how many years have to go by before so many
• 48:27
  Speaker 1: random contingencies accumulate that no no human being could conceivably
• 48:31
  Speaker 1: have anticipated anything that far out. And um O our
• 48:36
  Speaker 1: current best guests for the kinds of geopolitical geoeconomic questions
• 48:40
  Speaker 1: we're looking at, it is around a year or so.
• 48:42
  Speaker 1: Isn't that just the nature of society and a complex
• 48:48
  Speaker 1: system such as fill in the blank, stock markets, economy, geopolitics, elections,
• 48:55
  Speaker 1: anything along those lines. They're so sensitive to an all conditions,
• 49:01
  Speaker 1: they're nonlinear that you end up with a little change
• 49:04
  Speaker 1: here has out sized impact further down the road. Really,
• 49:08
  Speaker 1: what we're saying is the universe is pretty random beyond
• 49:11
  Speaker 1: twelve months. Uh, it's practically impossible to make any sort
• 49:17
  Speaker 1: of realistic forecast with any degree of specificity. Um about
• 49:23
  Speaker 1: certain things. I made a bet four years ago with
• 49:26
  Speaker 1: um I won't mention his name, but he's been a
• 49:28
  Speaker 1: guest on the show as to I was wondering who
• 49:31
  Speaker 1: was the GOP nominee likely to be? This is literally
• 49:36
  Speaker 1: three years ago after after the election, and he said, well,
• 49:42
  Speaker 1: what about the Democrats? And I said, well, it's easy,
• 49:44
  Speaker 1: that'll be Hillary, but who I have no idea who
• 49:46
  Speaker 1: the Republican is going to be. So we made a bet,
• 49:49
  Speaker 1: and so far looking pretty good. Um. But I don't
• 49:54
  Speaker 1: know if I got lucky, just got lucky and assumed
• 49:56
  Speaker 1: it was she was up next, she was next in line. Uh.
• 50:00
  Speaker 1: Under normal circumstances, when you're looking out to three or
• 50:03
  Speaker 1: four years, there are so many contingencies. Can anybody really
• 50:08
  Speaker 1: I got lucky. I'm not pretending I have any expertise
• 50:11
  Speaker 1: in politics or anything else, But can anybody consistently have
• 50:17
  Speaker 1: any sort of acumen thinking out more than twelve months?
• 50:23
  Speaker 1: It's it's really hard. I mean, there are there are
• 50:24
  Speaker 1: some categories of questions where great longer foresight is possible.
• 50:30
  Speaker 1: I think the Hillary thing was in the cards for
• 50:32
  Speaker 1: for for for quite a long time. Um. But for
• 50:35
  Speaker 1: most of these things, I mean, who mean to take
• 50:37
  Speaker 1: some extreme examples, I mean, in nineteen forty Dwight Eisenhower
• 50:41
  Speaker 1: was an anonymous Army colonel. In nineteen fifty two years
• 50:44
  Speaker 1: president United States in twelve years, right, uh in um Um.
• 50:49
  Speaker 1: Jimmy Carter was an anonymous peanut farmer in the nineteen
• 50:52
  Speaker 1: sixty four. In nineteen seventy six, he was being elected
• 50:56
  Speaker 1: president United States. Twelve years is a huge amount of
• 50:59
  Speaker 1: time in politics, So I don't think anybody really is
• 51:03
  Speaker 1: going to suppose there's very much possibility there UM, But
• 51:07
  Speaker 1: as you get closer and closer, it gets more and
• 51:09
  Speaker 1: more possible. That's not all that surprising. It's an analogy.
• 51:11
  Speaker 1: Will be like to Snell and I char when you
• 51:13
  Speaker 1: visit your optometrist and engage, it's easier and easier the
• 51:16
  Speaker 1: closer up you get for most things. UM. The trouble
• 51:19
  Speaker 1: is we're just not very well tuned to the parameters UM.
• 51:22
  Speaker 1: And if somebody can tell a really good story about
• 51:24
  Speaker 1: a relatively far off future about you know, the United
• 51:27
  Speaker 1: States is moving toward a techno utopia and which DDP
• 51:31
  Speaker 1: will will skyrocket as intelligent machines do amazing things for
• 51:35
  Speaker 1: us in the fourth and Dulsta Revolution. That that meant
• 51:38
  Speaker 1: may that scenario may indeed materialized by its UM, but
• 51:44
  Speaker 1: the likelihood of scenarios of that sort being accurate UM
• 51:49
  Speaker 1: is extremely low. I have a t shirt at home.
• 51:53
  Speaker 1: It says Where's my jet pack? I was promised the
• 51:56
  Speaker 1: jet pack by the year two thousand under forty four characters. Right.
• 52:01
  Speaker 1: That's right. When when you look back at future forecasts
• 52:06
  Speaker 1: from decades ago, and we now have enough for them
• 52:09
  Speaker 1: that we can look back years as to what people
• 52:14
  Speaker 1: were expecting from the future. What's fascinating is all the
• 52:18
  Speaker 1: amazing technology, technological developments, all the advantages of hardware, software, biotechnology,
• 52:24
  Speaker 1: medicine that we practically take for granted. They weren't the
• 52:29
  Speaker 1: things that people were forecasting. It was colonizing Mars and
• 52:33
  Speaker 1: other sort of hoverboards and other such things. Uh So,
• 52:38
  Speaker 1: even when you're thinking in terms of giant technological changes,
• 52:42
  Speaker 1: and of course there's a handful of people, you know,
• 52:45
  Speaker 1: Arthur C. Clarke is notorious for having forecast everything from
• 52:48
  Speaker 1: cell phones to satellites to to what have you. Um
• 52:53
  Speaker 1: what does this say about our ability to understand the few,
• 52:58
  Speaker 1: the present and extra appelate to the future. What it
• 53:02
  Speaker 1: suggests is we'd be better off if we were aware
• 53:05
  Speaker 1: of our limitations, achieved a certain baseline of appropriate humility,
• 53:12
  Speaker 1: and got in the habit of keeping score, and resisted
• 53:16
  Speaker 1: being sucked into clever scenarios and storytellers, and resisted being
• 53:22
  Speaker 1: seduced by credentials. If we could manage to do those things,
• 53:26
  Speaker 1: I think we would um proceed through life, making investments
• 53:30
  Speaker 1: and political decisions with better calibrated probabilities. And I think
• 53:34
  Speaker 1: we would be better office individuals and we'd be better
• 53:36
  Speaker 1: officers of society. That sounds that sounds tremendous. On a
• 53:40
  Speaker 1: related note to that, because because those those seven bullet
• 53:43
  Speaker 1: points are very significant. Let's let's talk about uncertainty, which
• 53:48
  Speaker 1: you is a is a uh concept that is dotted
• 53:53
  Speaker 1: throughout actually both books. Um, what is uncertainty and what
• 53:58
  Speaker 1: does it mean for individuals just trying to navigate their
• 54:02
  Speaker 1: way through the world. Do we understand uncertainty? Uh? Do
• 54:05
  Speaker 1: we misunderstand it? What? What exactly is it relative to
• 54:09
  Speaker 1: thinking about the future? Well, um, there are some types
• 54:16
  Speaker 1: of problems where the probabilities can be readily computed. We
• 54:22
  Speaker 1: can compute the probability of drawing an asis spades from
• 54:24
  Speaker 1: a randomly shuffled deck, um very accurately, uh, too many
• 54:29
  Speaker 1: decimal points if we want whatever one out of over
• 54:31
  Speaker 1: fifty two works out too. We can do that with
• 54:34
  Speaker 1: coin toss games and so forth. Um. So there are
• 54:38
  Speaker 1: some games in which the classic rules of statistics want
• 54:42
  Speaker 1: oh one very clearly apply and there are well defined probabilities.
• 54:46
  Speaker 1: In other words, we know what the range of outcomes are.
• 54:50
  Speaker 1: We just don't know what the specific outcome is going
• 54:52
  Speaker 1: to be. Got a well defined sampling universe. You've got clear, quick,
• 54:55
  Speaker 1: clear feedback about your about your predictions. Um, my of
• 55:00
  Speaker 1: the world, most of the world isn't like that. It's
• 55:02
  Speaker 1: not like it's definitely not like that. And and and
• 55:05
  Speaker 1: the question is what are the limits? How useful is
• 55:08
  Speaker 1: it to apply probabilistic forms of reasoning um outside their
• 55:12
  Speaker 1: traditional domains of application. Then, in a sense, is what
• 55:15
  Speaker 1: the U. S. Intelligence community really wanted to explore? I mean,
• 55:18
  Speaker 1: can we do better than say, distinct possibility, which, when
• 55:21
  Speaker 1: you look at it carefully, is such an elastic term.
• 55:23
  Speaker 1: It could mean anything from one percent to nine. So
• 55:26
  Speaker 1: let's talk a little bit about the intelligence community and
• 55:29
  Speaker 1: the Defense Department. Um, how did you get involved with
• 55:34
  Speaker 1: DARPA and the the competition, the forecasting competition? Right? Well, Um,
• 55:40
  Speaker 1: it was I r as DARA, the Intelligence Advanced Research
• 55:43
  Speaker 1: Projects Agency is post to DARPA, but it's it's it's
• 55:47
  Speaker 1: a cousin and and it and it models itself to
• 55:49
  Speaker 1: some degree. I think after dark by it it really
• 55:51
  Speaker 1: wants to do radical earth change, world changing forms of research.
• 55:57
  Speaker 1: And I think changing how we think about uncertainty would
• 55:59
  Speaker 1: would be would be pretty fundamental. Maybe not as fundamentals
• 56:02
  Speaker 1: inventing the Internet, but way up there. Um, it would
• 56:05
  Speaker 1: be a big deal, um to to to change how
• 56:08
  Speaker 1: we how we go about doing things. I think our
• 56:10
  Speaker 1: our democracy would would be transformed. I think the finance
• 56:13
  Speaker 1: industry would be transformed. It would not it would not
• 56:15
  Speaker 1: be not a small thing. These would not be small
• 56:19
  Speaker 1: things that how long have they been running this contest?
• 56:21
  Speaker 1: So they started. They approached my wife and May when
• 56:23
  Speaker 1: we were still on the faculty at University California, Berkeley
• 56:26
  Speaker 1: about six years ago, and we had a nice um
• 56:29
  Speaker 1: um a set of drinks over at the Clermont Hotel
• 56:32
  Speaker 1: in Berkeley, and um we um. We were just astonished
• 56:37
  Speaker 1: that the U. S Intelligence community was prepared to run
• 56:39
  Speaker 1: a series of forecasting tournaments. I mean, I predicted that
• 56:42
  Speaker 1: they would never want to do anything like that. So
• 56:43
  Speaker 1: it's kind of ironic, right that I I forecast that
• 56:46
  Speaker 1: forecasting tournaments would be impossible. And I know it's being
• 56:48
  Speaker 1: a bit of a HEDGEHOWK. What I what I said is, look,
• 56:50
  Speaker 1: government bureaucracies don't give a slingshot money to David right
• 56:55
  Speaker 1: Glass doesn't give sl slingshot money to David. Why would
• 56:58
  Speaker 1: be a massive influential government you're oocracy fifty billion dollars
• 57:01
  Speaker 1: or so, I wanted to spread millions of dollars around
• 57:04
  Speaker 1: to a bunch of small scale academic competition to see
• 57:06
  Speaker 1: whether or not they can do a better job of
• 57:08
  Speaker 1: assigning realistic probabilities to things of national security significance. And
• 57:12
  Speaker 1: this it didn't make any sense given the normal rules
• 57:15
  Speaker 1: of bureocratic behavior in Washington, d c UM and so
• 57:18
  Speaker 1: I was too Hedgehogy, I was wrong about that. Now
• 57:21
  Speaker 1: when they I'm delighted. I was wrong to say, at
• 57:24
  Speaker 1: least when when they came to you the prior book,
• 57:26
  Speaker 1: the Expert Political Judgment book, you had really run a
• 57:29
  Speaker 1: form of this. You would assembled a mast over eighty
• 57:34
  Speaker 1: two thousand separate forecasts from several thousand, was it or
• 57:38
  Speaker 1: several hundred political forecasters? It was it was a smaller number.
• 57:42
  Speaker 1: It was in the hundreds, but um the um. Yes.
• 57:46
  Speaker 1: In a sense, Expert Political Judgment was a small scale
• 57:49
  Speaker 1: dry run for what I RPA did on Expert Political
• 57:52
  Speaker 1: Judgment was run more on a shoestring budget, whereas UH
• 57:56
  Speaker 1: the r PA forecasting tournaments were run on on a
• 57:58
  Speaker 1: much more in a much more per fessional, large scale
• 58:00
  Speaker 1: basis with and you know, and one of the nice
• 58:03
  Speaker 1: things about the R project, and people worry about the
• 58:05
  Speaker 1: applicability of research and things like this, but this was
• 58:07
  Speaker 1: all independently monitored by the U S intelligence community. I mean,
• 58:10
  Speaker 1: these these forecasts were submitted at nine am Eastern time
• 58:13
  Speaker 1: every day on the day, every DA, every day. The
• 58:15
  Speaker 1: forecasting tournaments are running over four years UM, so there
• 58:19
  Speaker 1: is a very clear paper trail. So what what is
• 58:23
  Speaker 1: the state of the forecasting contests these days? Is it
• 58:26
  Speaker 1: something that they've put aside? What what is the takeaway
• 58:29
  Speaker 1: from all that? The takeaways are that it is possible
• 58:35
  Speaker 1: to make better probability estimates of events that many people
• 58:38
  Speaker 1: thought it would be impossible to estimate probabilistically. And it's
• 58:42
  Speaker 1: possible to do that by engaging in systematic talent spawning,
• 58:45
  Speaker 1: which you can only do if you're tracking score. And
• 58:47
  Speaker 1: it's also possible to do by designing good training modules,
• 58:51
  Speaker 1: by putting together teams that are open to dissent and
• 58:54
  Speaker 1: uh know how to do precision questioning of each other's assumptions,
• 58:58
  Speaker 1: and also by doing a little bit of algory with
• 59:00
  Speaker 1: mcmagic um. So do you think the result of that
• 59:05
  Speaker 1: contest has changed the way the US intelligence community recruits talent,
• 59:13
  Speaker 1: trains talent, and makes forecasts about future events. Well, you'd
• 59:16
  Speaker 1: have to ask the U S intelligence community about how
• 59:18
  Speaker 1: exactly things have changed. My understanding is that the National
• 59:22
  Speaker 1: Intelligence Council now does try to quantify it's probability estimates
• 59:27
  Speaker 1: rather than using just vague verbiage forecasting UM. It has
• 59:30
  Speaker 1: probability ranges. I think it tries to distinguish at least
• 59:33
  Speaker 1: seven degrees of uncertainty, which is a lot more than three.
• 59:36
  Speaker 1: H is more than five, which was which was the
• 59:38
  Speaker 1: preceding number. I think they may be underestimating themselves. I
• 59:41
  Speaker 1: think they could probably get up to ten or fifteen
• 59:43
  Speaker 1: if they if they wanted to. Uh. But I think
• 59:46
  Speaker 1: they're moving in the right direction. I think there's growing
• 59:48
  Speaker 1: interest in crowdsourcing forecast There's growing recognition that UM, the
• 59:53
  Speaker 1: average forecast derived from a group of forecasters, is often
• 59:56
  Speaker 1: more accurate than most of the individuals from whom the
• 59:59
  Speaker 1: average was derived. It sounds kind of magical, but it
• 1:00:02
  Speaker 1: makes it it is true. UM not always true, but
• 1:00:06
  Speaker 1: it's a good way to bet we We've I've been
• 1:00:09
  Speaker 1: critical of some of the prediction markets, not because the
• 1:00:13
  Speaker 1: theory underlying them is wrong. But very often they're narrow,
• 1:00:17
  Speaker 1: they're not diverse, they're not incentivized. All the various things
• 1:00:21
  Speaker 1: you need for a prediction market to work is often missing.
• 1:00:25
  Speaker 1: Um And sometimes the better as the participants are are
• 1:00:28
  Speaker 1: so similar to each other, it's hard to extrapolate that
• 1:00:32
  Speaker 1: out to other other factors. Um. Uh, These these sort
• 1:00:37
  Speaker 1: of contests and the various prediction markets. Can we describe
• 1:00:40
  Speaker 1: these as moneyball for the intelligence community? Is it just
• 1:00:45
  Speaker 1: quantifying data in a way that hasn't been done previously
• 1:00:49
  Speaker 1: to intelligence forecasts. I think that's a great way to
• 1:00:52
  Speaker 1: describe it. Just money ballfing the intelligence community. Um. I
• 1:00:56
  Speaker 1: think it's the movie The The Old World was a
• 1:00:58
  Speaker 1: world with baseball scouts. Uh, cl Clint Eastwood, Trusty baseball
• 1:01:03
  Speaker 1: scouts too. You know, we're gradually being displaced by these
• 1:01:07
  Speaker 1: number crunchers. Um. We're never going to do away with
• 1:01:10
  Speaker 1: people who have deep qualitative insights into the subject matter.
• 1:01:13
  Speaker 1: There are crucial source of inputs. But the question is
• 1:01:17
  Speaker 1: what roles should we be playing as the world changes?
• 1:01:20
  Speaker 1: And I think human judgment will always be playing a
• 1:01:23
  Speaker 1: critical role when we're dealing with human beings. Um Um.
• 1:01:28
  Speaker 1: But there are useful tools for combining human judgment, and
• 1:01:33
  Speaker 1: you can get more out of it than previously supposed.
• 1:01:36
  Speaker 1: That makes a lot of sense. But before we get
• 1:01:38
  Speaker 1: to our our favorite standard questions, anything from super forecasting,
• 1:01:42
  Speaker 1: I might have missed that you want to uh add
• 1:01:46
  Speaker 1: as as worth thinking about before we uh we get
• 1:01:50
  Speaker 1: into a little bit of your history. Well, the thing
• 1:01:52
  Speaker 1: that I most hope if I mean I'm getting older now,
• 1:01:55
  Speaker 1: I mean I've been doing this stuff for thirty plus years, UM.
• 1:01:58
  Speaker 1: And the thing that I most hope lasting legacy of
• 1:02:01
  Speaker 1: this work, and I hope it improves US foreign policy
• 1:02:05
  Speaker 1: and intelligence analysis, but I also hope it improves our democracy.
• 1:02:09
  Speaker 1: And I think in the in the closing chapter, we
• 1:02:11
  Speaker 1: talk about the debate between Paul Krugman and Nil Ferguson
• 1:02:15
  Speaker 1: on various issues, and how it more resembles a food
• 1:02:18
  Speaker 1: fight than it does a serious debate between extremely intelligent people,
• 1:02:22
  Speaker 1: which which both of them obviously are. UM. And the
• 1:02:25
  Speaker 1: question is, could we use forecasting tournaments? Could we structure
• 1:02:29
  Speaker 1: them in ways uh to facilitate more civilized debates on
• 1:02:32
  Speaker 1: issues that matter. So that's why I wrote a piece
• 1:02:35
  Speaker 1: in The New York Times several months ago with Peter
• 1:02:37
  Speaker 1: Skoblick on how we could do that with Iranian nuclear deal? Um,
• 1:02:41
  Speaker 1: and when a one way to proceed would be to say, Okay,
• 1:02:44
  Speaker 1: you've got hawks, you've got doves. You have different opinions
• 1:02:47
  Speaker 1: about what the long term consequences of signing this deal are.
• 1:02:50
  Speaker 1: We don't know for sure which historical trajectory were on.
• 1:02:53
  Speaker 1: Why don't the hawks generate five questions that they think
• 1:02:56
  Speaker 1: they have a comparative advantage in answering? Why don't the
• 1:02:58
  Speaker 1: doves generate five questions and they think they have a
• 1:03:01
  Speaker 1: comparative advantage in answering? And you know what, victory will
• 1:03:04
  Speaker 1: have a clear cut meaning here if the if the
• 1:03:06
  Speaker 1: doves can answer the dove questions better than the hawks,
• 1:03:09
  Speaker 1: and they can answer the hawk questions better, then the
• 1:03:11
  Speaker 1: doves win and vice versa for the hawks. Uh now, um,
• 1:03:14
  Speaker 1: anyone take you up on that? Well, we do have
• 1:03:16
  Speaker 1: a number of people who are participating in the tournament,
• 1:03:19
  Speaker 1: and um, one of the people, and g j open
• 1:03:22
  Speaker 1: dot com has written a memo on on where where
• 1:03:24
  Speaker 1: we are right now? The moderate seemed to be doing
• 1:03:26
  Speaker 1: the best at the moment, but you know that game
• 1:03:28
  Speaker 1: is far from over. I mean, this is just very
• 1:03:30
  Speaker 1: early stages of a long term process. Yeah, where you're
• 1:03:34
  Speaker 1: one of what a tenure treaty, it's quite a way
• 1:03:37
  Speaker 1: is to go. So so when you when you describe that,
• 1:03:40
  Speaker 1: and you mentioned debates, I immediately thought of the political
• 1:03:42
  Speaker 1: debates this year, which at least on the GEOP side,
• 1:03:46
  Speaker 1: have been not your usual policy debates. UM. And I'd
• 1:03:51
  Speaker 1: love to see some of the tenants from super forecasting
• 1:03:55
  Speaker 1: find its way to uh, the political parties and and
• 1:03:58
  Speaker 1: see if we can have a little more substantive discussion
• 1:04:01
  Speaker 1: about when this happens, here's what happens in the future,
• 1:04:04
  Speaker 1: and then hold these folks accountable. That really doesn't seem
• 1:04:07
  Speaker 1: to happen on whether with political experts or politicians. We
• 1:04:11
  Speaker 1: really don't hold their feet to the fire much do
• 1:04:13
  Speaker 1: How far into the future might it be when in
• 1:04:15
  Speaker 1: a in a presidential election, the presidential candidates take pride
• 1:04:19
  Speaker 1: in what their briar scores are. UM, I think it's
• 1:04:22
  Speaker 1: a long way off, judging by what's going on this year,
• 1:04:25
  Speaker 1: to say the least. So So let's talk about UM,
• 1:04:28
  Speaker 1: let's talk a little bit about you personally, rather than
• 1:04:31
  Speaker 1: than some of the books and the ideas that you've
• 1:04:34
  Speaker 1: you've put forth which which have been absolutely fascinating. Um,
• 1:04:39
  Speaker 1: So how did you find your way? You went, you
• 1:04:42
  Speaker 1: became a you went to Yale, you got your pH
• 1:04:44
  Speaker 1: d in psychology? How did you find your way into forecasting?
• 1:04:48
  Speaker 1: This really seems far afield from the traditional UM academic
• 1:04:55
  Speaker 1: realm of that. I was always a pretty strange psychologist.
• 1:04:59
  Speaker 1: I I had interest that it took me pretty deep
• 1:05:02
  Speaker 1: into social science, into political science in particular, into into
• 1:05:05
  Speaker 1: areas of business. But they always interested in organization as
• 1:05:09
  Speaker 1: I was interested in societies and cultures and large entities
• 1:05:12
  Speaker 1: that were not you know that obviously psychology matters there,
• 1:05:15
  Speaker 1: but it's it's a stretch for a psychologist. So in
• 1:05:18
  Speaker 1: my early work I did do a fair amount of
• 1:05:21
  Speaker 1: experimental UM work, but I was also also did a
• 1:05:24
  Speaker 1: lot of archival and naturalistic work. So it was a
• 1:05:26
  Speaker 1: kind of a natural progression for me. Um, who are
• 1:05:30
  Speaker 1: your early mentors? Uh? Well? Um uh. Peter Suitfeld was
• 1:05:35
  Speaker 1: my very first mentor in Canada. I was an undergraduate
• 1:05:38
  Speaker 1: University of British Columbia, and and he was wonderfully supportive
• 1:05:42
  Speaker 1: and of me, and and he believed in me, and
• 1:05:44
  Speaker 1: he he really told me that you know I would
• 1:05:48
  Speaker 1: probably have a pretty good time if I went to
• 1:05:51
  Speaker 1: graduate school at Yale and I took it on faith.
• 1:05:53
  Speaker 1: And I did that, and I met a number of
• 1:05:56
  Speaker 1: people at Yale who helped me. Um. The guy who
• 1:06:00
  Speaker 1: coined the term group think, Irving Janice, was one of
• 1:06:03
  Speaker 1: the people I worked with, and he was quite an
• 1:06:06
  Speaker 1: unusual psychologist. Also. UM. When I got to Berkeley, of course,
• 1:06:10
  Speaker 1: UM Daniel Koneman came along in a few years, and
• 1:06:13
  Speaker 1: he certainly had an influence on me. I already had
• 1:06:16
  Speaker 1: a PhD, and I was I was just recently tenured faculty.
• 1:06:19
  Speaker 1: But k Koneman is as a lot of very influential guy.
• 1:06:25
  Speaker 1: He's he's just a lot smarter than than most of us.
• 1:06:27
  Speaker 1: So it's a it's a good idea to listen very
• 1:06:30
  Speaker 1: carefully when he speaks. I really enjoyed, UM, thinking fast
• 1:06:34
  Speaker 1: and thinking slow, the metaphor for that entire two stage
• 1:06:41
  Speaker 1: way to look at how humans make decisions, either fast
• 1:06:45
  Speaker 1: and instinctual or longer and thoughtful, really just seems to
• 1:06:49
  Speaker 1: make a lot of sense. UM. What other books, uh,
• 1:06:53
  Speaker 1: have you really enjoyed? Whether books have been especially influential
• 1:06:58
  Speaker 1: to you. Another person who influenced me is just um
• 1:07:01
  Speaker 1: uptown here at Columbia University. Robert Jervis his book Perception, Misperception,
• 1:07:06
  Speaker 1: International Politics. It came out when I was in graduate
• 1:07:08
  Speaker 1: school and I could feel myself being tugged towards these topics.
• 1:07:12
  Speaker 1: It was. It's a brilliant analysis of mistakes that have
• 1:07:15
  Speaker 1: caused unnecessary wars, that perception and misperception in international politics.
• 1:07:21
  Speaker 1: By anything else stands out is as interesting or unusual
• 1:07:27
  Speaker 1: to you. Well, um, I mean life evolves in funny, quirky,
• 1:07:34
  Speaker 1: path dependent ways. I mean, you can you can look
• 1:07:36
  Speaker 1: back on your life and you can say, well, it
• 1:07:37
  Speaker 1: was kind of inevitable this happened or that happened. But
• 1:07:40
  Speaker 1: a lot of the things that that led to my
• 1:07:42
  Speaker 1: early forecasting tournament work where I think kind of quirky,
• 1:07:45
  Speaker 1: I mean, it was really kind of quirky. Thats. A
• 1:07:48
  Speaker 1: scholar as young as I was was appointed to a
• 1:07:50
  Speaker 1: National Research Council committee when I was just thirty or
• 1:07:53
  Speaker 1: thirty one year four. Uh, when I when I was
• 1:07:56
  Speaker 1: that young, um by far the most junior member a
• 1:07:59
  Speaker 1: committee like that, and had a lot of senior scientists
• 1:08:01
  Speaker 1: on it. But it gave me opportunities to meet a
• 1:08:03
  Speaker 1: lot of people, and it connected me to resources that
• 1:08:06
  Speaker 1: made it possible to do the early forecasting tournament work.
• 1:08:09
  Speaker 1: It also impressed on me the need to do it
• 1:08:11
  Speaker 1: because there there we were five liberals and conservatives, all
• 1:08:15
  Speaker 1: had very strong opinions about the Soviet Union and where
• 1:08:17
  Speaker 1: things were going. And the liberals thought that Reagan was
• 1:08:20
  Speaker 1: sending us to where the nuclear apocalypse, and the conservatives
• 1:08:23
  Speaker 1: thought that then the Soviet Union wasn't ere as an
• 1:08:27
  Speaker 1: evil empire would never change from within, essentially, as you
• 1:08:31
  Speaker 1: just had to keep up endless pressure and maybe it
• 1:08:32
  Speaker 1: would eventually crack. But you know, they didn't help a
• 1:08:35
  Speaker 1: lot any and they certainly didn't see garbage Of as
• 1:08:37
  Speaker 1: much of a change agent. Initially, um each side, neither
• 1:08:41
  Speaker 1: side really predicted gorbachalv and what Garbagechov did inside the
• 1:08:45
  Speaker 1: Soviet Union in the internal transformations that occurred. Both sides
• 1:08:49
  Speaker 1: could readily explain after the fact what happened. UM. So
• 1:08:53
  Speaker 1: it was this mismatch between virtually zero predictab ability and
• 1:08:56
  Speaker 1: virtually perfect expost explanatory ability that troubled me. And I thought, well,
• 1:09:01
  Speaker 1: you know, if if debates this important, like World War three,
• 1:09:05
  Speaker 1: are are being conducted, this shodily. You know, surely there's
• 1:09:08
  Speaker 1: a better way to do this um And that's what
• 1:09:11
  Speaker 1: led to the early work on expert political judgment. It
• 1:09:13
  Speaker 1: was a way to try, what can we do to
• 1:09:16
  Speaker 1: to keep score and and if we do keep score,
• 1:09:19
  Speaker 1: can we identify um better ways of making judgments. You
• 1:09:24
  Speaker 1: describe something that is an enormous pet peeve of mine.
• 1:09:27
  Speaker 1: In the markets, nobody knows what happens day to day.
• 1:09:30
  Speaker 1: There is zero predictive analysis, and then on any given day,
• 1:09:35
  Speaker 1: the market's up five hundred points, it's down five hundred points,
• 1:09:38
  Speaker 1: and ex post there is always a fantastic narrative explaining exactly,
• 1:09:44
  Speaker 1: here's why oil shot up and why the market rallied
• 1:09:48
  Speaker 1: three hundred points, or here's why this terrible thing happened
• 1:09:52
  Speaker 1: and the market dropped five hundred points. But nobody is saying,
• 1:09:56
  Speaker 1: if this happens tomorrow, then here's a result. It's always
• 1:09:59
  Speaker 1: an after the fact narrative that that seems to be
• 1:10:02
  Speaker 1: consistent across lots of different uh fields, not just politics,
• 1:10:07
  Speaker 1: but markets and economics. And after the fact we're fantastic storytellers.
• 1:10:12
  Speaker 1: Before the fact, we have no idea, And there there
• 1:10:15
  Speaker 1: are situations where we really want to continue doing that,
• 1:10:18
  Speaker 1: though it's not always bad. I mean, the National Transportation
• 1:10:21
  Speaker 1: Safety Board, for example, conducts these ex post postmortems on
• 1:10:25
  Speaker 1: plane plane crashes. One of the reasons why air travel
• 1:10:28
  Speaker 1: has become as safe as it is is because they're
• 1:10:30
  Speaker 1: so good at doing these postmortems. Obviously, they can't predict
• 1:10:33
  Speaker 1: which planes are going to go down, but they become
• 1:10:36
  Speaker 1: really pretty adept at identifying the critical factors that underlie
• 1:10:41
  Speaker 1: plane accidents, and as a result, the rules for pilots
• 1:10:45
  Speaker 1: and the design features of aircraft have changed in ways
• 1:10:49
  Speaker 1: that make us all safer. Safer so um. But they're
• 1:10:53
  Speaker 1: not just making up a story for the six o'clock news.
• 1:10:55
  Speaker 1: They're saying, hey, you know the whole shuttle investigation, the
• 1:10:59
  Speaker 1: O ring fair, that's right. Therefore, that's the most most
• 1:11:05
  Speaker 1: youth worlds. Most sectors don't have a black box that say, hey,
• 1:11:09
  Speaker 1: here's why the engine failed at two oh seven and fifteen.
• 1:11:14
  Speaker 1: That's right. So there are different types of postmortems, and
• 1:11:17
  Speaker 1: some of them are constrained by well defined bodies of
• 1:11:20
  Speaker 1: scientific knowledge and investigative procedure that reduced the serious risk
• 1:11:25
  Speaker 1: of capitalizing on chance, and others are just sort of
• 1:11:28
  Speaker 1: make it up as you go, and I think the
• 1:11:30
  Speaker 1: things we're talking about and make it up as you go.
• 1:11:32
  Speaker 1: But there are approaches to doing case studies and learning
• 1:11:36
  Speaker 1: from the past that are very disciplined and focused and
• 1:11:38
  Speaker 1: can make a safer and wealthier and happier. The The
• 1:11:41
  Speaker 1: recent book Um the Checklist talks about how much better
• 1:11:46
  Speaker 1: surgical procedures and outcomes have been since surgeons started using checklists,
• 1:11:51
  Speaker 1: including wash your hands, which very often was just assumed
• 1:11:56
  Speaker 1: um that it was done properly with a certain disinfectant
• 1:11:59
  Speaker 1: in a certain life of time. But we've we've apparently
• 1:12:02
  Speaker 1: dramatically reduced operating instruments left in abdomens and keeping count
• 1:12:09
  Speaker 1: of the number of sponges, and that's improved the subsequent outcome,
• 1:12:13
  Speaker 1: just as the National Transportation Safety Board has improved the
• 1:12:18
  Speaker 1: safety level of of travel. That's right. So the big
• 1:12:21
  Speaker 1: question for us is when is learning possible? When can
• 1:12:25
  Speaker 1: we learn to do certain categories of things better? And
• 1:12:28
  Speaker 1: when are we just spinning our wheels and deluding ourselves?
• 1:12:31
  Speaker 1: Are we spinning our wheels and deluding ourselves about financial markets?
• 1:12:34
  Speaker 1: So many political issues, but we're actually making real progress
• 1:12:37
  Speaker 1: in the domains of medicine, or airline safety or or whatnot.
• 1:12:41
  Speaker 1: And it's a it's a mixed picture. Um. And I
• 1:12:43
  Speaker 1: suppose what we're trying to do with this forecasting tournament
• 1:12:46
  Speaker 1: work is to bring some of the rigor that has
• 1:12:49
  Speaker 1: worked in these more scientific domains to bring it to
• 1:12:52
  Speaker 1: bear in in domains that they are more or less
• 1:12:55
  Speaker 1: like the wild West. Huh. Um. So let's you've been
• 1:12:59
  Speaker 1: doing this now for you said, thirty years. What's changed
• 1:13:03
  Speaker 1: in this industry more than anything? What is the significant
• 1:13:07
  Speaker 1: progress in in the forecasting and prediction industry? Um, during
• 1:13:12
  Speaker 1: the course of your career. That's a hard question. Um,
• 1:13:18
  Speaker 1: They're not all there there there, that's right, well there there.
• 1:13:21
  Speaker 1: I I think that our knowledge of the imperfections and
• 1:13:25
  Speaker 1: human judgment, thanks to a lot of the comment inspired
• 1:13:28
  Speaker 1: research programs, I think we've made discernible progress there. To
• 1:13:33
  Speaker 1: the least, I think we've I think some of the
• 1:13:36
  Speaker 1: statistical tools have improved in various ways. I think some
• 1:13:39
  Speaker 1: of the tools for running teams have even improved. I mean,
• 1:13:42
  Speaker 1: I think we can do There are versions of the
• 1:13:44
  Speaker 1: Delphi procedure, for example, which was developed a long time ago,
• 1:13:47
  Speaker 1: but has got the DELFI better, right, so, which is
• 1:13:51
  Speaker 1: what I remember. I was saying that a lot of
• 1:13:52
  Speaker 1: people thought it was kind of crazy to use teams.
• 1:13:54
  Speaker 1: You're better off having a lot of independent observers. But
• 1:13:57
  Speaker 1: there's a way to get the benefits of independence and
• 1:13:59
  Speaker 1: the benefits of creative interaction at the same time. And
• 1:14:02
  Speaker 1: one way to do that is by going getting everybody
• 1:14:05
  Speaker 1: to make their judgments anonymously. So you give your probability judgment,
• 1:14:08
  Speaker 1: your explanation, I give mine, and everybody around the table
• 1:14:11
  Speaker 1: gives theirs, and we circulate, and we we circulate that
• 1:14:14
  Speaker 1: and nobody knows who said what. So the high status
• 1:14:16
  Speaker 1: guy isn't swaying everything the way off. It often happens
• 1:14:20
  Speaker 1: in groups. UM, and everybody's expressing your judgments anonymously is don't,
• 1:14:24
  Speaker 1: so they're insulated from the group thing pressure. And you
• 1:14:27
  Speaker 1: can do that two or three times, and then the
• 1:14:29
  Speaker 1: question is how much better is the resulting group judgment
• 1:14:32
  Speaker 1: after you go through this process. Uh, Then it would
• 1:14:35
  Speaker 1: have been if you'd simply say, taken um an unweighted
• 1:14:38
  Speaker 1: average of each of the individual group group group judgments,
• 1:14:41
  Speaker 1: and the answer is this better? How much better? Uh?
• 1:14:45
  Speaker 1: I think met analysis suggests probably in the vicinity of
• 1:14:47
  Speaker 1: ten percent better is you know? That's real? Right? And
• 1:14:53
  Speaker 1: if if you're talking about avoiding a war or finding
• 1:14:56
  Speaker 1: a terrorist or anything along those lines, that's a real
• 1:15:00
  Speaker 1: worthwhile pursuit. It's nothing to sniff at, nothing to sniff at.
• 1:15:04
  Speaker 1: So now let me ask you for your forecast. What
• 1:15:07
  Speaker 1: are the next major changes? What are the next shifts
• 1:15:10
  Speaker 1: that are going to come in in the world of
• 1:15:13
  Speaker 1: forecasts and predictions? What do you see? Perhaps the better
• 1:15:15
  Speaker 1: way to say this to ask this is what do
• 1:15:19
  Speaker 1: you see as the influence of your work on on
• 1:15:22
  Speaker 1: the forecast and community? I see huge potential here, um
• 1:15:27
  Speaker 1: i R, which funded the first forecasting tournament, is going
• 1:15:30
  Speaker 1: to be funding to follow up forecasting tournaments, um and
• 1:15:34
  Speaker 1: I think many of your readers might be interested in these,
• 1:15:36
  Speaker 1: and there will be calls for volunteers to participate in
• 1:15:39
  Speaker 1: one form or another. One of them is focusing not
• 1:15:42
  Speaker 1: so much on the accuracy of your forecast as on
• 1:15:45
  Speaker 1: the probitive value of the explanations you generate. The probate
• 1:15:49
  Speaker 1: of value. Are you good at for or after the fact?
• 1:15:52
  Speaker 1: Are you good at explaining things? Um and? And again,
• 1:15:55
  Speaker 1: the extent to which we can crowdsource aspects of problem
• 1:15:59
  Speaker 1: solving and then eventually marrying that. I think the forecasting, UM,
• 1:16:03
  Speaker 1: I think that's a very ambitious project. It's it's it's
• 1:16:07
  Speaker 1: in the very early stages. It hasn't been launched yet, um,
• 1:16:11
  Speaker 1: but I'm optimistic that it will. UM. Um. So what
• 1:16:15
  Speaker 1: this is called the I r PA forecast things. It's
• 1:16:18
  Speaker 1: called create create c R E A T E. Uh.
• 1:16:22
  Speaker 1: It's it's it stands for the flex reasoning. That's right,
• 1:16:28
  Speaker 1: it's an acronym like that. All right, that sounds interesting.
• 1:16:31
  Speaker 1: I'll definitely I'll search for that and all link to
• 1:16:34
  Speaker 1: that and that. The other is another competition which is
• 1:16:36
  Speaker 1: called h FC, the Hybrid Forecasting Competition, which will be UM,
• 1:16:41
  Speaker 1: humans and machines and human machine combinations, uh, trying to
• 1:16:46
  Speaker 1: make predictions, which I am quite optimistic about. Um. So
• 1:16:51
  Speaker 1: it's Watson working with somebody, well, it would be that
• 1:16:53
  Speaker 1: would be one way of doing it. There are a
• 1:16:54
  Speaker 1: lot of there are a lot of possible machines, are
• 1:16:56
  Speaker 1: a lot of possible models that people could work with.
• 1:16:58
  Speaker 1: And the question is are you better off just using
• 1:17:00
  Speaker 1: the model or you better off just using the person,
• 1:17:03
  Speaker 1: or you're better when are you better off using the
• 1:17:04
  Speaker 1: combination of the model and the person, and the one
• 1:17:08
  Speaker 1: truth is we really don't know the answers to these
• 1:17:10
  Speaker 1: questions right now, and we're hoping to learn more. So
• 1:17:13
  Speaker 1: I think that's these are these are really important projects.
• 1:17:16
  Speaker 1: And I think the other big thing that that I'm
• 1:17:18
  Speaker 1: very focused on because it has relevance to improving the
• 1:17:21
  Speaker 1: debate in our debates in our society, is competitions to
• 1:17:25
  Speaker 1: generate better questions. It's not to generate better questions. It's
• 1:17:29
  Speaker 1: not just about forecasting. It's I mean, you can you
• 1:17:32
  Speaker 1: can forecast trivial pursuits, and you can become a great
• 1:17:34
  Speaker 1: forecaster in trivial pursuits. And really, the world is not
• 1:17:37
  Speaker 1: a better place for it? Or what? What? What? Your
• 1:17:40
  Speaker 1: world will be a better place when we join super
• 1:17:42
  Speaker 1: forecasting skills. Two questions on which big policy debates pivot.
• 1:17:47
  Speaker 1: So you say, if we knew the answer to this question,
• 1:17:50
  Speaker 1: would we have invaded Iraq? Or if we had known
• 1:17:52
  Speaker 1: the answer, if we had known the answer that if
• 1:17:54
  Speaker 1: we if we knew the answer this question, what would
• 1:17:55
  Speaker 1: would have changed how we what we do in Syria,
• 1:17:57
  Speaker 1: the Ukraine, or with respect to tax policy, with respect
• 1:18:00
  Speaker 1: to uh FED policy or whatnot uh so uh generating
• 1:18:05
  Speaker 1: probitive questions, generating high quality explanations, human machine competitions. I
• 1:18:12
  Speaker 1: think these are three really important areas for the future.
• 1:18:15
  Speaker 1: That's fascinating. The last two questions, Um, this is always
• 1:18:20
  Speaker 1: interesting and I'm trying to figure out the best way
• 1:18:21
  Speaker 1: to its phraser for you. Normally, I say, what advice
• 1:18:25
  Speaker 1: would you give to a millennial or someone just graduating
• 1:18:29
  Speaker 1: college who are going into your field? But I don't
• 1:18:31
  Speaker 1: know whether that's the field of psychology or the field
• 1:18:34
  Speaker 1: of analyzing and improving forecasts and predictions. But let me ask,
• 1:18:40
  Speaker 1: in an open ended fashion, what advice would you give
• 1:18:43
  Speaker 1: to someone just coming out of school starting their career
• 1:18:46
  Speaker 1: who wants to follow in your footsteps. Well, it's an
• 1:18:50
  Speaker 1: interesting point that I really don't have a field anymore.
• 1:18:53
  Speaker 1: I mean, the University of Pennsylvania when they hired me,
• 1:18:56
  Speaker 1: they didn't really know where to put me. So I'm
• 1:18:58
  Speaker 1: partly warden, partly psychology, partly a political science, and Nannenburg
• 1:19:02
  Speaker 1: um so a lot of different But it's really the
• 1:19:05
  Speaker 1: you're really studying this, let's let's just, for lack of
• 1:19:09
  Speaker 1: a better phrase, you're studying the science of decision making
• 1:19:14
  Speaker 1: that we're studying human judgment and the extent to which
• 1:19:17
  Speaker 1: human judgment can be improved using a variety of tools,
• 1:19:21
  Speaker 1: some of them drawn from psychology, some of them drawn
• 1:19:23
  Speaker 1: from statistics, some of them drawn from organization theory, a
• 1:19:25
  Speaker 1: lot of different tools. So someone who wanted to go
• 1:19:29
  Speaker 1: into that field, what advice would you give them? I
• 1:19:32
  Speaker 1: would say, Um, that there is no clear path to
• 1:19:38
  Speaker 1: where I am right now. Um that it's it's it's
• 1:19:42
  Speaker 1: not clear to me, Um, where you would go because
• 1:19:46
  Speaker 1: the work I'm doing is so weird and interdisciplinary it
• 1:19:48
  Speaker 1: doesn't fit into any of the existing university niches, Which
• 1:19:52
  Speaker 1: is kind of funny because most university niches have become
• 1:19:56
  Speaker 1: more and more specific and more and more narrowly focused,
• 1:20:00
  Speaker 1: And you're going in the opposite direction, pulling from three
• 1:20:02
  Speaker 1: distinct plus the whole quantitative side of it, three distinct
• 1:20:07
  Speaker 1: areas of practice with a heavy math overlay. Yeah. I
• 1:20:10
  Speaker 1: don't claim to be much more general than many of
• 1:20:13
  Speaker 1: the specialists my specialist colleagues. I think I'm just more
• 1:20:16
  Speaker 1: specialized in a weird way. I mean, I work is
• 1:20:20
  Speaker 1: very specialized and focused. It just draws on different components
• 1:20:24
  Speaker 1: of different disciplines in a very focused way. Um, so
• 1:20:28
  Speaker 1: that's intriguing. I think people who do in new discipline work,
• 1:20:31
  Speaker 1: you know, they're not Leonardo da Vinci. I mean, there
• 1:20:33
  Speaker 1: aren't any Leonardo DaVinci as far as I can tell
• 1:20:35
  Speaker 1: right now in the university world. Uh. Um, what we
• 1:20:38
  Speaker 1: what we do is we we we we and we
• 1:20:40
  Speaker 1: need to carve out a very specialized research programs that
• 1:20:43
  Speaker 1: deliver have tangible deliverables. Uh that we were really on
• 1:20:47
  Speaker 1: a tight accountability leash in this forecasting tournament. It was
• 1:20:51
  Speaker 1: you know, we had we were submitting forecast nine am
• 1:20:53
  Speaker 1: Eastern time every every day. It was. It was a
• 1:20:56
  Speaker 1: very rigorous process and we needed to have very focused group,
• 1:21:00
  Speaker 1: tangible deliverables and a focused process that that seems like
• 1:21:04
  Speaker 1: that's of great value to both business and government. Final question,
• 1:21:09
  Speaker 1: what is it that you know about forecasting today that
• 1:21:13
  Speaker 1: you wish you knew when you started down this road
• 1:21:15
  Speaker 1: thirty years ago? Well? What I wish I knew? What
• 1:21:19
  Speaker 1: I what? So the early work was mostly about cursing
• 1:21:22
  Speaker 1: the darkness. It was about cognitive bias and how we're
• 1:21:26
  Speaker 1: prisoners of our preconceptions, how we have a heart that
• 1:21:28
  Speaker 1: we're too quick to make up our minds were too
• 1:21:30
  Speaker 1: slow to change them. It was a rather dark purtrait
• 1:21:32
  Speaker 1: of human nature. UM. And there's some reason for being pessimistic,
• 1:21:36
  Speaker 1: given the way we think about UM politics and history
• 1:21:39
  Speaker 1: and economics for much of the time UM. The later
• 1:21:43
  Speaker 1: work has been more about lighting candles. It has a
• 1:21:45
  Speaker 1: more upbeat flavor that there are specific things you can
• 1:21:48
  Speaker 1: do to become more open minded, at least about relatively
• 1:21:51
  Speaker 1: near term futures. And if you can become more open
• 1:21:53
  Speaker 1: minded about relatively near term futures, maybe you can become
• 1:21:55
  Speaker 1: a little more open minded about medium and longer term futures.
• 1:21:58
  Speaker 1: Maybe you can be better able to see how alternative
• 1:22:01
  Speaker 1: perspectives might have some merit UM. And I think that
• 1:22:04
  Speaker 1: when you feel you're in competition with the other side,
• 1:22:08
  Speaker 1: and the other side might be getting to the truth
• 1:22:09
  Speaker 1: faster than you, that has a very salutary effect. I
• 1:22:13
  Speaker 1: think it will tend to make us a bit more
• 1:22:14
  Speaker 1: open minded. Thank you so much for being so generous
• 1:22:18
  Speaker 1: with your time, Professor Tetlock. Uh. We've been speaking with
• 1:22:21
  Speaker 1: Professor Philip Tetlock of the University of Pennsylvania UM both
• 1:22:26
  Speaker 1: Wharton and other schools, author of super Forecasting, The art
• 1:22:30
  Speaker 1: and science of prediction, as well as expert political judgment,
• 1:22:36
  Speaker 1: How Good is It? How Can We Know? And a
• 1:22:39
  Speaker 1: number of other books. If you've enjoyed this conversation, be
• 1:22:42
  Speaker 1: sure and look up an Inch or Down an Inch
• 1:22:44
  Speaker 1: on Apple iTunes and you'll see all of our other
• 1:22:47
  Speaker 1: eighty three or so um previous conversations. I want to
• 1:22:53
  Speaker 1: thank Mike bat Nick for doing the deep dive and
• 1:22:55
  Speaker 1: helping me on the research with this. I'm Barry Ridhltz.
• 1:23:00
  Speaker 1: You've been listening to Masters and Business on Bloomberg radioh