Latest / Statecraft / How to Predict the Future
Transcript
- 0:04SPEAKER_00: Hi,
- 0:05SPEAKER_00: I'm Santi Ruiz,
- 0:06SPEAKER_00: and you're listening to Statecraft,
- 0:08SPEAKER_00: an interview series about how policymakers get things done.
- 0:11SPEAKER_00: Today's episode was originally recorded and published in November 2023,
- 0:15SPEAKER_00: but we never published the audio until now.
- 0:18SPEAKER_00: Jason Matheny, our interviewee today, is kind of DC royalty.
- 0:22He spent time in academia,
- 0:24Oxford,
- 0:24Princeton,
- 0:25Johns Hopkins Applied Physics Laboratory,
- 0:28But from 2015 to 2018,
- 0:30he ran IARPA,
- 0:31the Intelligence Advanced Research Projects Activity.
- 0:36If you're familiar with the ARPA model or DARPA,
- 0:39you've got a good idea of what IARPA is.
- 0:41SPEAKER_00: Basically, they're the R&D lab for the intelligence community.
- 0:45SPEAKER_00: Since then,
- 0:46SPEAKER_00: he spent time in the Biden White House,
- 0:48SPEAKER_00: and he now runs the RAND Corporation,
- 0:50SPEAKER_00: a major think tank,
- 0:51SPEAKER_00: really one of the original think tanks.
- 0:53A couple of years ago,
- 0:54Jason and I talked about how to predict the future.
- 0:57SPEAKER_00: Jason's been really interested in prediction markets for a while,
- 1:00SPEAKER_00: in predictive accuracy and Tetlock's work on human judgment.
- 1:04And a lot of his time at IARPA was spent not just on procuring
- 1:07cool new gadgets of
- 1:08SPEAKER_00: the James Bond style,
- 1:10but trying to build tools so that intelligence analysts and
- 1:13operatives could make
- 1:14SPEAKER_00: better calls.
- 1:16SPEAKER_00: As he says in the interview,
- 1:17SPEAKER_00: the wars in Iraq and Afghanistan were multi-trillion dollar decisions.
- 1:21If you could improve the accuracy of forecasting individual
- 1:26strategies by just a
- 1:28SPEAKER_00: percentage point,
- 1:29SPEAKER_00: that would be worth tens of billions of dollars.
- 1:31SPEAKER_01: Thanks for doing these.
- 1:32I think it's such a great service to society to try to record
- 1:38things that have
- 1:38worked and things that haven't worked in different kinds of
- 1:42policy effort.
- 1:43SPEAKER_00: I hope you're right.
- 1:43SPEAKER_00: I think you're right.
- 1:44SPEAKER_00: And it's a lot of fun, too.
- 1:45SPEAKER_00: So...
- 1:46SPEAKER_00: I'll get into kind of my formal question asking in a moment,
- 1:49SPEAKER_00: but just for context,
- 1:51SPEAKER_00: I would love to talk about the ARPA model generally,
- 1:54SPEAKER_00: about what makes made iARPA different,
- 1:58SPEAKER_00: what's similar,
- 1:59SPEAKER_00: different,
- 2:00SPEAKER_00: unique.
- 2:01SPEAKER_00: And I'd love to talk about forecasting work specifically at iARPA,
- 2:04because I know that was something that mattered a lot to you in
- 2:07your time there.
- 2:08SPEAKER_00: In your view, what makes the ARPA model unique and unique?
- 2:13Obviously,
- 2:13it's been replicated a couple of times in a couple of different
- 2:15contexts.
- 2:16And you've spoken about and thought a lot about which parts can
- 2:19be replicated and
- 2:20SPEAKER_00: which parts need to be adjusted to different contexts.
- 2:22SPEAKER_00: Yeah.
- 2:24So the basic ARPA model is having entrepreneurial program
- 2:32managers who are hired
- 2:34SPEAKER_01: and given a budget and a lot of autonomy to pick research problems and
- 2:41SPEAKER_01: that are considered high risk,
- 2:42SPEAKER_01: high reward,
- 2:45select researchers to fund either based on their individual
- 2:49judgment or the
- 2:50SPEAKER_01: judgment of a small selection panel,
- 2:53and then have the researchers compete in parallel against a
- 2:57common set of metrics,
- 2:59usually with some disciplined approach to ending funding
- 3:03annually for teams that
- 3:04SPEAKER_01: aren't succeeding.
- 3:07So I think maybe the three sort of core elements then are the
- 3:12program managers,
- 3:14SPEAKER_01: the research tournament,
- 3:17SPEAKER_01: and then the defunding of research that isn't succeeding.
- 3:22I think all three are pretty unusual and distinguish the ARPA
- 3:29model from the
- 3:29SPEAKER_01: conventional model of funding science and technology efforts.
- 3:35Would the ARPA model work without really kind of stellar
- 3:40program managers?
- 3:41SPEAKER_00: You said entrepreneurial as one of your kind of descriptors.
- 3:43SPEAKER_00: What else makes a good program manager?
- 3:47SPEAKER_01: Yeah.
- 3:47SPEAKER_01: So,
- 3:47SPEAKER_01: I mean,
- 3:48SPEAKER_01: just to go a little bit over how program managers are recruited,
- 3:52SPEAKER_01: selected,
- 3:54SPEAKER_01: they're recruited by office directors,
- 3:57SPEAKER_01: typically on the basis of technical knowledge and
- 4:00SPEAKER_01: an entrepreneurial mindset,
- 4:02SPEAKER_01: creativity,
- 4:03SPEAKER_01: risk tolerance,
- 4:05predicted ability to be able to succeed in pitching a program
- 4:12and executing that
- 4:14SPEAKER_01: program.
- 4:16The program manager candidates make a program pitch based on
- 4:20the Heilmeier questions.
- 4:23SPEAKER_01: They go through multiple rounds of review,
- 4:24SPEAKER_01: first with the office director,
- 4:27SPEAKER_01: who sort of serves as a gatekeeper.
- 4:30SPEAKER_01: There are other characteristics to program managers that are important.
- 4:33SPEAKER_01: Typically, they need to get a security clearance.
- 4:36SPEAKER_00: Is that true across all of the ARPAs?
- 4:39SPEAKER_00: I know it's true for iARPA.
- 4:42SPEAKER_01: Yeah, it's true for iARPA.
- 4:43SPEAKER_01: I think all of the DARPA PMs, I believe, have security clearances.
- 4:48SPEAKER_01: I don't know if it's true of ARPA Energy or of ARPA Health.
- 4:54SPEAKER_01: But I believe it's certainly true of everybody at IRF.
- 4:57SPEAKER_01: I believe it's true of DARPA as well.
- 5:00SPEAKER_01: I don't think there are exceptions.
- 5:02SPEAKER_01: Now,
- 5:02SPEAKER_01: the level of security clearance that could vary,
- 5:04SPEAKER_01: but that does mean that certain kinds of folks aren't qualified.
- 5:08SPEAKER_01: In some ways,
- 5:09SPEAKER_01: that might even disadvantage research,
- 5:11SPEAKER_01: you know,
- 5:12SPEAKER_01: like non-US citizens,
- 5:13SPEAKER_01: which is 96% of the world is disqualified.
- 5:19SPEAKER_01: The program managers are given very little training.
- 5:24SPEAKER_01: They're also given very little management.
- 5:26SPEAKER_01: The main performance measure is whether you get your program approved.
- 5:32And that process is sort of like a cross between a dissertation
- 5:38defense and a pitch
- 5:40SPEAKER_01: to a venture capitalist.
- 5:43SPEAKER_01: You know, you structure your pitch in answers to the Hallmark questions.
- 5:49But the goal is ultimately to persuade this group of people
- 5:54that you have an idea
- 5:56SPEAKER_01: that could be transformative if successful.
- 5:58SPEAKER_00: Is there such a thing as too risky or too difficult a project to qualify?
- 6:02SPEAKER_00: Or is it just measured on impact if successful?
- 6:06SPEAKER_01: Yeah, great question.
- 6:07SPEAKER_01: Some projects are too risky.
- 6:10That is,
- 6:10if the sort of premise is,
- 6:12hey,
- 6:13if successful,
- 6:13this would be transformative.
- 6:15SPEAKER_01: But in order to be successful,
- 6:17SPEAKER_01: I have to violate the following laws of physics,
- 6:20or I have to solve one particular problem that Nobel laureates
- 6:25haven't managed to
- 6:26SPEAKER_01: solve in the last 50 years.
- 6:29SPEAKER_01: Then I think that's going to be deemed too risky.
- 6:32SPEAKER_01: Yeah.
- 6:33There's probably some Goldilocks zone of appropriate
- 6:38probabilities there.
- 6:40SPEAKER_01: It's probably under 50% and over 5%, something like that.
- 6:48What's interesting is that there are then problems that aren't
- 6:52risky enough.
- 6:54SPEAKER_01: And that's seen as being not ARPA hard.
- 6:58SPEAKER_01: Even if those problems, if solved, could be transformative.
- 7:02SPEAKER_01: And,
- 7:02SPEAKER_01: you know,
- 7:04SPEAKER_01: it's often said that the goal of the ARPA should be high-risk,
- 7:08SPEAKER_01: high-reward research.
- 7:10Really just from a policy point of view,
- 7:12what you would love is low risk,
- 7:15high reward research.
- 7:17SPEAKER_01: I mean,
- 7:17you shouldn't disqualify a research project just because it
- 7:20turns out that the
- 7:22SPEAKER_01: project would be easier to solve.
- 7:24SPEAKER_01: Ultimately,
- 7:25you should be sort of optimizing based on the cost benefit of a
- 7:29particular research
- 7:30SPEAKER_01: investment.
- 7:32SPEAKER_01: And that does mean then that the ARPAs do not tend to go after
- 7:37SPEAKER_01: transformative but easy problems,
- 7:41even if the reason that they haven't been solved is because
- 7:43they're neglected or
- 7:45SPEAKER_01: they have just been kind of ignored.
- 7:48Part of the rationale for the ARPAs using their funding as
- 7:51opposed to other
- 7:53SPEAKER_01: agencies is that the risk profile should be high risk,
- 7:58SPEAKER_00: high reward.
- 8:00SPEAKER_00: And just to flesh that out,
- 8:01is that because ARPAs are aimed to fill a hole in the funding
- 8:06space that wouldn't
- 8:07SPEAKER_00: be filled otherwise?
- 8:09SPEAKER_01: That's right.
- 8:10So the sense it is across the federal enterprise or the sort of
- 8:15portfolio of
- 8:17SPEAKER_01: federal agencies that are funding research and development,
- 8:21SPEAKER_01: that the niche for the ARPAs is to focus on higher risk research.
- 8:29SPEAKER_01: And that would be its comparative advantage,
- 8:32that the model of having entrepreneurial program managers with
- 8:38relatively little
- 8:40SPEAKER_01: oversight is to give them the ability to go after high-risk research,
- 8:45SPEAKER_01: that you don't need the ARPA model,
- 8:46SPEAKER_01: maybe if you're going after lower-risk research.
- 8:50Can you give me an example of research that fits in the target
- 8:54area for an ARPA and
- 8:55SPEAKER_00: then research that would be really transformative but doesn't quite fit?
- 9:00Stealth technology,
- 9:01for example,
- 9:02would be in the high risk,
- 9:03high reward space.
- 9:06SPEAKER_01: And that was historic DARPA investment.
- 9:08SPEAKER_01: It was also a U.S.
- 9:09SPEAKER_01: Air Force investment.
- 9:12And the riskiness of it was that we had models for how you
- 9:17could significantly
- 9:19SPEAKER_01: reduce the radar cross-section of an aircraft.
- 9:27But those models produced aircraft designs that we couldn't
- 9:32figure out how to fly.
- 9:34SPEAKER_01: And so that was technically risky.
- 9:38But the impact on military operations,
- 9:41if successful,
- 9:41was going to be transformative.
- 9:43It would mean that you could have aircraft that were
- 9:47penetrating your defense zones.
- 9:49SPEAKER_01: And I think that's a pretty classic case of an ARPA hard problem.
- 9:55SPEAKER_01: high risk,
- 9:56SPEAKER_01: high reward,
- 9:57one that benefited from having program managers who were
- 10:01constructive contrarians.
- 10:03SPEAKER_01: I mean,
- 10:03SPEAKER_01: you know,
- 10:04SPEAKER_01: despite being told,
- 10:05SPEAKER_01: oh,
- 10:05it just won't be aerodynamically feasible to design an aircraft
- 10:10that has such a
- 10:11SPEAKER_01: small cross section.
- 10:14SPEAKER_01: An example of a project that isn't ARPA hard.
- 10:17SPEAKER_01: So one would be, or sorry, isn't a good fit for an ARPA.
- 10:22SPEAKER_01: One would be something that doesn't seem risky enough.
- 10:27SPEAKER_01: For instance,
- 10:28SPEAKER_01: you know,
- 10:29producing a specific app for a specific device that solves a
- 10:35problem that seems
- 10:36SPEAKER_01: like the commercial sector is going to solve in a year anyway.
- 10:39You know,
- 10:39like at this point in time,
- 10:42like a better camera for a cell phone.
- 10:45SPEAKER_01: Yeah.
- 10:47SPEAKER_01: One,
- 10:47it's not clear that it's super risky because advances are
- 10:50happening in these
- 10:51SPEAKER_01: commercial cameras all the time.
- 10:52SPEAKER_01: Second, it's not going to be transformative for permissions.
- 10:56And the counterfactual of the world in which an ARPA doesn't
- 11:00invest in that is that
- 11:01SPEAKER_01: ultimately those advances get made anyway.
- 11:06SPEAKER_01: What might be another example?
- 11:08SPEAKER_01: In the other direction...
- 11:10SPEAKER_01: you know, faster than light travel.
- 11:12SPEAKER_01: Like, you know, maybe it's possible.
- 11:14SPEAKER_01: Maybe there are tachyons that are achieving,
- 11:16SPEAKER_01: you know,
- 11:16SPEAKER_01: faster than light speeds,
- 11:18but figuring out how to really leverage that is probably beyond
- 11:22the risk tolerance
- 11:24SPEAKER_01: of an ARPA program.
- 11:26SPEAKER_00: And I want to get into kind of the bureaucratic history of ARPA,
- 11:28SPEAKER_00: but first you just kind of mentioned counterfactuals and the amount of
- 11:32counterfactual thinking that goes into deciding what qualifies
- 11:36as a good ARPA
- 11:36SPEAKER_00: project.
- 11:38SPEAKER_00: Do the ARPAs have a framework for thinking through counterfactuals?
- 11:43SPEAKER_00: So I love the ARPA model.
- 11:45I mean,
- 11:45I think it's a great alternative to a lot of traditional
- 11:50science and technology funding.
- 11:52SPEAKER_01: There's a lot of variation in this model.
- 11:54SPEAKER_01: And there's also, I think, some blind spots.
- 11:58SPEAKER_01: The Heilmerer questions don't, for instance, have a question about
- 12:03SPEAKER_01: the counterfactual impact or would this sort of work get done otherwise?
- 12:11There's a question about what difference will it make if you're
- 12:14successful,
- 12:15SPEAKER_01: which I think partially gets at this.
- 12:18SPEAKER_01: But the ARPAs tend not to do a real rigorous assessment of,
- 12:22SPEAKER_01: okay,
- 12:23here are all the other funding streams going to solve this
- 12:26particular problem.
- 12:28Here's what we understand about the likelihood of success or
- 12:30failure of each of
- 12:31SPEAKER_01: those alternative efforts.
- 12:34SPEAKER_01: To what degree will this problem be obviated by some other development?
- 12:39SPEAKER_01: We tend also not to think as much about strategic move and counter move,
- 12:46and particularly when you're working in an environment that's
- 12:48competitive,
- 12:49SPEAKER_01: like national security,
- 12:50SPEAKER_01: where you have different actors that are competing against one another in
- 12:55SPEAKER_01: technology.
- 12:56SPEAKER_01: We probably should be spending more time to be thinking about,
- 13:00SPEAKER_01: well,
- 13:00SPEAKER_01: if we develop this technology X,
- 13:03SPEAKER_01: our competitor is likely to develop technology Y.
- 13:06SPEAKER_01: Do we have a reason to believe that technology X will be asymmetrically
- 13:12SPEAKER_01: advantageous to us?
- 13:14Or will it simply be one that a competitor could copy or could
- 13:20actually gain
- 13:21SPEAKER_01: leverage out of by exploiting some vulnerability in technology X?
- 13:26So I think there is a kind of strategic analysis that isn't
- 13:30inherent in the DARPA
- 13:32SPEAKER_01: Heilmeier questions or the ARPA Heilmeier questions.
- 13:35And there have been some efforts over the years to kind of
- 13:38append the original
- 13:40SPEAKER_01: Heilmeier questions.
- 13:42SPEAKER_01: At IARPA,
- 13:43SPEAKER_01: I would ask some questions about this kind of competitive aspect,
- 13:47SPEAKER_01: also the way in which we need to think about
- 13:50SPEAKER_01: baking in security and safety into the technologies we create.
- 13:55SPEAKER_01: So for example,
- 13:56SPEAKER_01: are there ways of making a technology less prone to reverse engineering,
- 14:02SPEAKER_01: to theft,
- 14:04SPEAKER_01: to misuse?
- 14:05SPEAKER_01: Because many of the things that the US government funds get stolen.
- 14:12SPEAKER_01: Sure.
- 14:12SPEAKER_01: Even when we classify them.
- 14:14And if you assign some probability to the theft of some
- 14:17classified piece of
- 14:18SPEAKER_01: research,
- 14:19and I think it probably is prudent to assign at least a 10%
- 14:22probability to some
- 14:24exquisite technology that we're developing in a classified
- 14:28environment to that
- 14:29SPEAKER_01: being stolen,
- 14:31SPEAKER_01: then you should be thinking in advance,
- 14:33SPEAKER_01: well,
- 14:33will I be better off or worse off having developed this
- 14:36technology if it's stolen?
- 14:37SPEAKER_01: Sure.
- 14:39SPEAKER_01: And you should plan ahead for that.
- 14:40SPEAKER_01: And you should think,
- 14:41SPEAKER_01: is there some way of baking into the technology itself,
- 14:44SPEAKER_01: some protection against the possibility that it's stolen,
- 14:48SPEAKER_01: the possibility that it's stolen and used against me,
- 14:50SPEAKER_01: the possibility that it's stolen and used against me,
- 14:53SPEAKER_01: not only by a state actor,
- 14:54SPEAKER_01: but by a non-state actor.
- 14:56And so I think that kind of strategic analysis is something
- 14:59that we probably need
- 15:00SPEAKER_01: to add on to the way in which the Heilmeier questions are used.
- 15:05SPEAKER_00: Yeah.
- 15:07Real quick for our listeners,
- 15:08will you just outline what the high market questions are?
- 15:12SPEAKER_01: Yeah, there's a few different versions of these.
- 15:15SPEAKER_01: What are you trying to do?
- 15:17SPEAKER_01: How is it done today?
- 15:18SPEAKER_01: And what are the limits of current practice?
- 15:20SPEAKER_01: What is new in your approach?
- 15:22SPEAKER_01: And why do you think it will be successful?
- 15:24SPEAKER_01: Who cares?
- 15:25SPEAKER_01: What difference will it make?
- 15:26SPEAKER_01: What are the risks?
- 15:28SPEAKER_01: How much will it cost?
- 15:29SPEAKER_01: How long will it take?
- 15:31SPEAKER_01: And those questions are deceptively simple.
- 15:34I think in answering those questions about a program that one
- 15:38is trying to propose
- 15:40SPEAKER_01: to an ARPA leadership,
- 15:44SPEAKER_01: the...
- 15:46SPEAKER_01: Oh,
- 15:46and actually there's the last question of what are the midterm
- 15:49and final exams to
- 15:51SPEAKER_01: check for success,
- 15:51SPEAKER_01: which is an important one.
- 15:52It's just,
- 15:53you know,
- 15:53how do you evaluate whether you're succeeding or whether you're
- 15:55failing?
- 15:57SPEAKER_01: But anyway, I think the questions help to guide one's approach.
- 16:02SPEAKER_01: to designing a program,
- 16:04it doesn't help as much in thinking about which programs are
- 16:10worth pursuing at all.
- 16:12That is like,
- 16:13are we picking the problems that are going to matter the most?
- 16:16Because typically you're answering these questions for two or
- 16:20three concepts.
- 16:21You're not thinking about of all the possible problems that I
- 16:24could work on,
- 16:25SPEAKER_01: how do I ensure that I'm working on the most important one?
- 16:30SPEAKER_01: This is a sort of different set of questions.
- 16:32SPEAKER_01: There's a few different formulations of that.
- 16:34One is Richard Hamming at Bell Labs used to go around as the
- 16:39Bell Labs lunchroom
- 16:41SPEAKER_01: and invite himself to people's tables,
- 16:44SPEAKER_01: probably in a sort of way that was irritating to most staff.
- 16:47But he would ask them,
- 16:49what are the most important problems in your field and why?
- 16:52SPEAKER_01: And then he would ask them what they were working on.
- 16:54And he would typically find that people were not working on the
- 16:57problems that they
- 16:58SPEAKER_01: just said were the most important ones.
- 16:59And then he would ask them why they weren't working on the
- 17:01problems that they were
- 17:02SPEAKER_01: most important.
- 17:04SPEAKER_01: And one,
- 17:05SPEAKER_01: it was surprising,
- 17:05SPEAKER_01: I think,
- 17:06how often people just didn't think about ensuring that they
- 17:11were working on the
- 17:11SPEAKER_01: most important problems.
- 17:13SPEAKER_01: And then second,
- 17:14SPEAKER_01: the various barriers to working on those problems,
- 17:16either that they didn't think that they would get support for
- 17:19it or they weren't
- 17:19SPEAKER_01: sure that they would be successful.
- 17:21So there is a sort of set of questions that needs to proceed to
- 17:25Heidelberg
- 17:25SPEAKER_01: questions,
- 17:26SPEAKER_01: which is,
- 17:27are you really sure this is the most important problem that you
- 17:30should be spending
- 17:31SPEAKER_01: your time on?
- 17:32SPEAKER_01: Are there more important problems?
- 17:34SPEAKER_01: Because there are all sorts of technical problems to solve.
- 17:39SPEAKER_01: And the opportunity cost for your time is really high.
- 17:43Because if you take on a program at an ARPA,
- 17:47that's probably at least a quarter of your time.
- 17:49As a program manager,
- 17:51you can really concentrate on running probably three or four
- 17:55new programs.
- 17:56Beyond that,
- 17:57you're really delegating the management of the programs to the
- 17:59SETA.
- 18:01SPEAKER_01: The SETA, sorry, is?
- 18:04SPEAKER_01: Oh, the Scientific and Engineering Technical Assistance.
- 18:07SPEAKER_01: So those are the on-site contractors who help in running the programs.
- 18:13SPEAKER_00: So in your time leading IARPA,
- 18:16SPEAKER_00: you made some adjustments to the Heilmeier questions you guys were using.
- 18:20I'd love to hear kind of both about those changes or additions
- 18:24and what your
- 18:26SPEAKER_00: process was for making that bureaucratic change internally.
- 18:31SPEAKER_01: We continued to use the hallmark questions,
- 18:34SPEAKER_01: but we also thought there are important questions to add that get up how
- 18:39SPEAKER_01: competitors are going to respond to a technology being developed.
- 18:44SPEAKER_01: Since we should be thinking about move and counter move and how different
- 18:47technology investments could either create or not create an
- 18:51asymmetric advantage,
- 18:53SPEAKER_01: we should be thinking then about what happens after
- 18:56SPEAKER_01: The technology is developed and is introduced to the world.
- 18:59SPEAKER_01: So one of the questions is,
- 19:00what's your estimate about how long it would take a major
- 19:03nation competitor to
- 19:04SPEAKER_01: weaponize this technology after they learn about it?
- 19:07What's your estimate for a non-state terrorist group with
- 19:10resources like those of
- 19:11SPEAKER_01: Al-Qaeda in the first decade of the century?
- 19:14Second,
- 19:15if the technology is leaked,
- 19:16stolen,
- 19:17or copied,
- 19:17would we regret having developed it?
- 19:20What if any first mover advantage is likely to endure after a
- 19:23competitor follows?
- 19:25SPEAKER_01: Third, how could the program be misinterpreted by foreign intelligence?
- 19:29SPEAKER_01: Do you have any suggestions for reducing that risk?
- 19:32Fourth,
- 19:32can we develop defensive capabilities before or alongside
- 19:36offensive ones?
- 19:37Fifth,
- 19:38can the technology be made less prone to theft replication and
- 19:41mass production?
- 19:42SPEAKER_01: What design features could create barriers to entry?
- 19:45SPEAKER_01: And sixth, what red team activities could help answer these questions?
- 19:49SPEAKER_01: Whose red team opinions would you particularly respect?
- 19:53SPEAKER_01: On that last point,
- 19:54I do think that red teaming is a really important check on the
- 20:01strategic value of a
- 20:03SPEAKER_01: particular technology.
- 20:05SPEAKER_01: And it's not done frequently enough.
- 20:08I think we typically start research programs in the ARPAs
- 20:15without having somebody
- 20:17SPEAKER_01: who's assigned a kind of red team role of thinking,
- 20:21SPEAKER_01: okay,
- 20:21SPEAKER_01: imagine we're successful.
- 20:22What are you going to do in response to this technology that
- 20:25we've just created?
- 20:27SPEAKER_01: How are you going to counter it?
- 20:28SPEAKER_01: If you manage to steal it, how are you going to use it?
- 20:31SPEAKER_01: That's just not something that we routinely do.
- 20:34SPEAKER_00: Why not?
- 20:35SPEAKER_00: Why is that not a common...
- 20:37mode of analysis we don't do enough red teaming in general it's
- 20:43interesting being
- 20:43at rand because rand has a long history of doing word gaming
- 20:46you know since the 40s
- 20:48have sometimes it's just really awkward because you're trying
- 20:52to beat the stuff
- 20:54that you're investing a lot of money and time and building and
- 20:58you're not highly
- 20:58SPEAKER_01: motivated to see the ways that it breaks and
- 21:01So I think some kinds of institutional and even personal
- 21:05incentives not to explore
- 21:07SPEAKER_01: the ways in which one's own investment is potentially vulnerable.
- 21:12SPEAKER_01: But we know from red teams and pen tests and,
- 21:17SPEAKER_01: say,
- 21:18SPEAKER_01: the cyber domain that it is really important to do that kind of analysis.
- 21:23In fact,
- 21:23it's maybe one of our best protections against different kinds
- 21:27of vulnerabilities.
- 21:28It's our best way of identifying them early and then being able
- 21:31to create countermeasures.
- 21:34SPEAKER_01: And we don't do that enough,
- 21:35SPEAKER_01: though,
- 21:36SPEAKER_01: when we're developing technologies,
- 21:38SPEAKER_01: I think partly for the institutional disincentive,
- 21:43SPEAKER_01: partly also because it takes time and investment.
- 21:46And that's trading off against other time or investment that is
- 21:50seen as kind of
- 21:51SPEAKER_01: more within the job jar of a research and development agency.
- 21:57SPEAKER_01: So,
- 21:57SPEAKER_01: you know,
- 21:58SPEAKER_01: the ARPAs could say,
- 21:59SPEAKER_01: hey,
- 21:59SPEAKER_01: look,
- 21:59SPEAKER_01: that's not our job is to be figuring out how to break our own technology.
- 22:03But it's not clear that there would be any agency better
- 22:06qualified to figure out
- 22:08SPEAKER_01: how to break the technology than the ARPAs themselves.
- 22:12They just need to find folks who are really going to be sincere
- 22:15about trying to break it.
- 22:16SPEAKER_01: Aren't friends with the program manager,
- 22:18are sufficiently kind of separated as an honest broker so that
- 22:23they can be a
- 22:24SPEAKER_01: trusted third party.
- 22:25SPEAKER_00: How do you institute that kind of...
- 22:29SPEAKER_00: honest broker red teams bureaucratically?
- 22:31SPEAKER_00: Because I'm guessing if I sit at a lunch table with you every day,
- 22:34SPEAKER_00: I may not be as good of a red teamer as somebody who,
- 22:37SPEAKER_00: you know,
- 22:38SPEAKER_00: has a neutral or even adversarial relationship with you.
- 22:42SPEAKER_01: Yeah,
- 22:43SPEAKER_01: most of the ARPA programs will have testing and evaluation teams,
- 22:48SPEAKER_01: but their role is to make sure in a very honest,
- 22:52credible way that the technology that's being developed is
- 22:56actually working and
- 22:58SPEAKER_01: that the research is on track.
- 23:00SPEAKER_01: Their role is not, hey, let me figure out how to defeat this technology.
- 23:05SPEAKER_01: Now,
- 23:06SPEAKER_01: the testing evaluation role is typically played by an FFRDC,
- 23:11SPEAKER_01: a federally funded research and development center,
- 23:13SPEAKER_01: or a UARC,
- 23:14SPEAKER_01: a university affiliated research center,
- 23:16which have these kind of privileged relationships with
- 23:19government in that they're
- 23:21SPEAKER_01: able to leverage private sector expertise and
- 23:24SPEAKER_01: But they have reliable enough funding that they don't have to worry about
- 23:29SPEAKER_01: irritating government,
- 23:31SPEAKER_01: a particular government program manager,
- 23:33SPEAKER_01: that they can just be trusted to tell the truth.
- 23:35And they're also just kind of like temperamentally selected to
- 23:40be places that just
- 23:41SPEAKER_01: will tell the truth no matter what,
- 23:43SPEAKER_01: even if it's unpopular.
- 23:44SPEAKER_01: Sure.
- 23:45SPEAKER_01: And, you know, RAND is an example of a place like that.
- 23:47SPEAKER_01: Ida and Center for Naval Analysis and others are,
- 23:51SPEAKER_01: I think they also have that kind of shared commitment.
- 23:53Like they're just nerd magnets and they attract nerds who are
- 23:59like honest to a fault.
- 24:01SPEAKER_01: And so that's really, really good.
- 24:03The ARPAs then put the FFRDCs to work in evaluating whether the
- 24:09program is accessible.
- 24:11SPEAKER_01: They are typically not put to work on this other side,
- 24:14which is now pretend that you're our enemy and are figuring out
- 24:20how to counter this
- 24:21SPEAKER_01: technology or figure out once you've stolen it,
- 24:25SPEAKER_01: how you're going to put it to use.
- 24:28So I think the institutions already exist potentially to carry
- 24:32out that work.
- 24:32SPEAKER_01: They just aren't typically asked to do it.
- 24:36SPEAKER_01: Again,
- 24:36SPEAKER_01: in the cyber realm,
- 24:37SPEAKER_01: using that as an example,
- 24:39SPEAKER_01: we have organizations that focus on red teaming and pen tests.
- 24:43SPEAKER_01: And then there are even competitions,
- 24:45SPEAKER_01: like DEFCON,
- 24:47SPEAKER_01: in which that kind of red team activity is celebrated.
- 24:54And we don't do that as systematically in national security or
- 24:59even other parts of
- 25:02SPEAKER_01: policy where you have kind of a permanent red team activity.
- 25:05SPEAKER_01: I mean,
- 25:06SPEAKER_01: to give another example,
- 25:08SPEAKER_01: in our foreign policy,
- 25:10there's no permanent red team at the National Security Council
- 25:14that tries to think,
- 25:16SPEAKER_01: how is China going to react to this policy and is deeply immersed in
- 25:22SPEAKER_01: China thought and policy,
- 25:24and whose sort of full-time job then is to play the role of
- 25:29China in anticipating
- 25:32SPEAKER_01: how China will respond to a particular US action.
- 25:35SPEAKER_01: And that might actually be an incredibly valuable function for a cell,
- 25:41SPEAKER_01: a red team cell,
- 25:43SPEAKER_01: to exist in a place like the National Security Council.
- 25:47SPEAKER_00: It's funny you mentioned this kind of
- 25:50institutional or bureaucratic weakness at creating versions of
- 25:55ourselves to try to
- 25:55SPEAKER_00: think like adversaries.
- 25:59SPEAKER_00: The name of this newsletter,
- 26:00SPEAKER_00: Statecraft,
- 26:02I was staring at my bookshelf and looking at a book by Angela
- 26:04Cotavilla informing
- 26:05SPEAKER_00: Statecraft.
- 26:06He's a longtime intelligence community guy with the Senate
- 26:11Intelligence Committee
- 26:12SPEAKER_00: staff for a while.
- 26:13SPEAKER_00: And his big critique of Cold War era
- 26:16american intelligence community was it was too bureaucratic in
- 26:20its thinking and
- 26:21unable to put itself in the shoes of the kgb and other
- 26:25adversaries and maybe to
- 26:26zoom out a little bit iarpa has like a kind of a particular
- 26:29relationship with the
- 26:30intelligence community i'm curious you can talk about what that
- 26:32relationship looks
- 26:33like how the how iarpa and this range of other federal entities
- 26:38interact yeah so
- 26:42SPEAKER_01: And by the way,
- 26:42SPEAKER_01: on the topic of red teams,
- 26:43there's a great book by Micah Zenko on just kind of the value
- 26:48of red teams and the
- 26:49SPEAKER_01: history of red team activities that I recommend.
- 26:54So IARPA is similar to DARPA in that it has the same overall
- 27:01kind of model of
- 27:02SPEAKER_01: program managers,
- 27:04SPEAKER_01: programs that are run like research tournaments,
- 27:08SPEAKER_01: and a rigorous process for down selection,
- 27:14SPEAKER_01: for stopping funding for things that aren't working.
- 27:17The main difference compared to DARPA is that the primary end
- 27:22user of IARPA's
- 27:24SPEAKER_01: research is the intelligence community,
- 27:27SPEAKER_01: which is a collection of,
- 27:29SPEAKER_01: I think now,
- 27:3018 agents that collect and analyze and interpret and protect
- 27:36information for U.S.
- 27:39SPEAKER_01: decision makers.
- 27:40SPEAKER_01: Yeah.
- 27:42And the goal then of IRPA is to be funding advanced research
- 27:49that ultimately
- 27:50SPEAKER_01: improves decision advantage in one of those,
- 27:53SPEAKER_01: you know,
- 27:53SPEAKER_01: in collecting information,
- 27:54SPEAKER_01: analyzing information,
- 27:56SPEAKER_01: protecting information.
- 28:00SPEAKER_01: The model is quite similar to DARPA.
- 28:05SPEAKER_01: It's a much smaller group of consumers.
- 28:08I mean,
- 28:08the intelligence community is maybe one-tenth the size of the
- 28:13Defense Department.
- 28:14SPEAKER_01: And because things don't need to be mass-produced,
- 28:18SPEAKER_01: I mean,
- 28:18SPEAKER_01: because you don't need as many numbers of
- 28:23of pieces of equipment for the intelligence community compared
- 28:28to the defense
- 28:28SPEAKER_01: department,
- 28:30SPEAKER_01: the problems that are worked on can be sort of one-offs.
- 28:33SPEAKER_01: I mean,
- 28:34it can be more exquisite in that there might be just a single
- 28:39device that you need
- 28:40SPEAKER_01: to be able to build,
- 28:43or there might be a device that you're only going to need to
- 28:46make five of.
- 28:48SPEAKER_01: And that means,
- 28:49SPEAKER_01: I think then,
- 28:50SPEAKER_01: and because there's often a greater premium on secrecy,
- 28:54SPEAKER_01: on both sort of covert and clandestine uses of technologies,
- 29:02there's a lot more thinking then about the various kinds of
- 29:06security around
- 29:07SPEAKER_01: technology efforts.
- 29:09SPEAKER_01: And in some cases, how do you
- 29:12SPEAKER_01: How do you even prevent it being an area of research that's recognized?
- 29:19SPEAKER_01: So those are some distinguishing features.
- 29:21SPEAKER_01: The level of security often needs to be higher.
- 29:25SPEAKER_01: The technology base is different.
- 29:29SPEAKER_01: For DARPA, there's a lot of established defense contractors.
- 29:32SPEAKER_01: The intelligence community will often have
- 29:36SPEAKER_01: contractors that are smaller and less well-known for security reasons.
- 29:43SPEAKER_01: What else is different?
- 29:45SPEAKER_01: All the program managers require a TSSCI clearance.
- 29:48SPEAKER_01: That's not true of DARPA,
- 29:50where most of the program managers might have secret or
- 29:53collateral top secret.
- 29:56SPEAKER_00: Let me cut out here and ask about...
- 29:59So some of the kind of consequences of that smaller purchasing
- 30:02base of the
- 30:03intelligence community versus defense and the lower
- 30:07manufacturing compared to most
- 30:09SPEAKER_00: DARPA projects.
- 30:11SPEAKER_00: Does that limit what IARPA can usefully work on or does it constrain it?
- 30:16SPEAKER_01: Well,
- 30:16I think in some ways it can open it up because if you don't
- 30:19have to think about
- 30:20SPEAKER_01: something that needs to be mass produced,
- 30:22SPEAKER_01: then the technical space of options could offer more degrees of freedom.
- 30:27I think also because a lot of the intelligence community's
- 30:31problems come down to problems that
- 30:35SPEAKER_01: of human judgment.
- 30:36There's also more room for investing in research and in human
- 30:44judgment and cognitive psychology
- 30:49SPEAKER_01: That,
- 30:49SPEAKER_01: I think,
- 30:49has historically been an area of significant investment by
- 30:55IARPA in a way that is
- 30:57SPEAKER_01: distinguished from the other archives.
- 30:59The problem of collection is typically a problem of collecting
- 31:02electrons or photons.
- 31:05SPEAKER_01: And so those end up being physics and electrical engineering problems.
- 31:11SPEAKER_01: There's also chemistry problems, especially in the
- 31:15SPEAKER_01: world today where we're increasingly concerned about misuses of biology.
- 31:19So figuring out ways of analyzing chemical signatures of
- 31:25biological activity or
- 31:27SPEAKER_01: biological signatures in some cases of biological activity.
- 31:31SPEAKER_01: is something that's increasingly important.
- 31:33But intelligence collection usually involves a lot of physics,
- 31:37a lot of electrical engineering.
- 31:38The analysis side certainly does involve things like statistics
- 31:44and computer
- 31:44SPEAKER_01: science,
- 31:45SPEAKER_01: machine learning research.
- 31:47SPEAKER_01: But ultimately, decision-making is predominantly human.
- 31:53And the human psychology of cognitive biases and heuristics
- 31:59then is really important.
- 32:02SPEAKER_01: And that has been an area of research at IRPA.
- 32:06SPEAKER_01: That I think has been pretty unusual compared to the other ARPAs.
- 32:11SPEAKER_01: And then there's a loop of information protection.
- 32:14SPEAKER_01: And there it's kind of back to the physics and engineering problems.
- 32:19And that the way in which you collect information is typically
- 32:23by breaking other
- 32:23SPEAKER_01: people's efforts at protecting it.
- 32:26SPEAKER_01: So on the protection side,
- 32:28it's sort of a cat and mouse game of trying to figure out how
- 32:31you would counter all
- 32:32SPEAKER_01: of the clever things that you're doing to collect information.
- 32:36My impression of your time at IARPA is that you really
- 32:40championed the kind of focus
- 32:42SPEAKER_00: on human judgment and tools to augment human forecasting.
- 32:49Was that an interest of yours before you,
- 32:53I mean,
- 32:53before you kind of entered the IARPA space?
- 32:56SPEAKER_01: I was first
- 32:58interested in research on human judgment after reading the book
- 33:03Expert Political
- 33:04SPEAKER_01: Judgment by Phil Tetlock,
- 33:07SPEAKER_01: which is,
- 33:07SPEAKER_01: you know,
- 33:08I think one of the more ambitious pieces of research on the
- 33:13accuracy of human
- 33:14SPEAKER_01: judgment about policymaking.
- 33:17SPEAKER_01: And, you know, over, you know,
- 33:2020-year period evaluating the accuracy of forecasts from a
- 33:25bunch of different
- 33:26participants about world events and just finding that the
- 33:29accuracy wasn't very
- 33:30SPEAKER_01: good.
- 33:30SPEAKER_01: Many cases, you know, wasn't substantially better than random chance.
- 33:36And it seemed like an important problem because the decisions
- 33:42that humans make
- 33:46SPEAKER_01: can be extraordinarily costly.
- 33:50And improving the accuracy of those judgments then could be
- 33:54worth an expectation.
- 33:56SPEAKER_01: You know, billions of dollars per percentage point improvement.
- 34:00SPEAKER_01: And I mean,
- 34:02SPEAKER_01: if you think about a decision like the wars in Iraq and Afghanistan,
- 34:06SPEAKER_01: I mean,
- 34:06SPEAKER_01: those were multi-trillion dollar decisions.
- 34:10SPEAKER_01: And improving the accuracy,
- 34:12either in our forecasts about what the likelihood of success
- 34:15would be on different
- 34:15SPEAKER_01: time horizons,
- 34:17or even being able to forecast the success of individual
- 34:22strategies being used in
- 34:24SPEAKER_01: those two countries.
- 34:26SPEAKER_01: If you can improve the accuracy by just a percentage point,
- 34:30SPEAKER_01: that would be worth tens of billions of dollars.
- 34:34And yet society does not invest tens of billions of dollars in
- 34:41figuring out how to
- 34:42SPEAKER_01: improve the accuracy of human judgment.
- 34:46SPEAKER_01: That seems really odd,
- 34:47SPEAKER_01: just that as a society,
- 34:49SPEAKER_01: we have underinvested in something that's so consequential.
- 34:55SPEAKER_01: and probably amenable to improvement.
- 34:59SPEAKER_01: As the work at IARPA demonstrated,
- 35:01you really can substantially improve human judgment through a
- 35:06few different
- 35:06SPEAKER_01: interventions that are pretty robust,
- 35:10SPEAKER_01: that over different cohorts of people,
- 35:13SPEAKER_01: over different periods of time,
- 35:15SPEAKER_01: over different kinds of analytic questions,
- 35:19SPEAKER_01: accuracy.
- 35:20SPEAKER_01: So,
- 35:20SPEAKER_01: you know,
- 35:21part of the reason for working on that set of problems at IARPA
- 35:24was just because it
- 35:25seemed neglected relative to what the societal benefit could be
- 35:31for the United
- 35:32SPEAKER_01: States,
- 35:32SPEAKER_01: for decision-making and national security and foreign policy.
- 35:38Did you encounter kind of pushback to making human judgment
- 35:43projects the level of
- 35:45SPEAKER_00: priority that they were in your time at IARPA?
- 35:48SPEAKER_00: Were there different camps that thought that was not...
- 35:50the most valuable use of resources i was lucky that i had great
- 35:56bosses who saw this
- 36:00as important i don't think that they were completely convinced
- 36:03that the you know
- 36:04that the methods were necessarily going to work but recognized
- 36:10the the value of
- 36:13SPEAKER_01: methods that would work and
- 36:17SPEAKER_01: were willing to take the risk.
- 36:19SPEAKER_01: And we were lucky not just in the leadership at IRPA,
- 36:25but also the leadership at the Office of the Director of
- 36:29National Intelligence.
- 36:30SPEAKER_01: So,
- 36:30SPEAKER_01: you know,
- 36:31SPEAKER_01: having Jim Clapper as DNI and Robert Cardillo as Deputy DNI and Stephanie
- 36:39SPEAKER_01: O'Sullivan as Principal Deputy DNI.
- 36:42SPEAKER_01: I mean, the three of them were also
- 36:45SPEAKER_01: supportive and protective of this kind of research.
- 36:51SPEAKER_01: And I think there was also an approach that we took,
- 36:53SPEAKER_01: which was not really to picking which horse was going to win in advance,
- 36:58but instead setting up a repeatable process in which you can
- 37:03evaluate methods and
- 37:05SPEAKER_01: really creating...
- 37:07an experimental test bed in which you could evaluate a bunch of
- 37:12different analytic
- 37:13SPEAKER_01: methods for their accuracy.
- 37:14SPEAKER_01: And some of them could be based on human judgment.
- 37:17SPEAKER_01: Some of them can be based on machine learning.
- 37:20SPEAKER_01: Some of them could be based on some combination.
- 37:23And then you can run them all in a forecasting tournament to
- 37:27see whether they're
- 37:27SPEAKER_01: making accurate forecasts or not about real world events.
- 37:32The goal really was to create an observatory in which different
- 37:36kinds of methods
- 37:37SPEAKER_01: could be compared to one another and tested for accuracy.
- 37:40SPEAKER_01: I was lucky that I had top cover for my bosses to do that type of work.
- 37:46And we were lucky that we could get world-class researchers who
- 37:50worked on this problem.
- 37:53In at least one of the tournaments that you guys ran,
- 37:57and I think likely in more contexts,
- 38:01Crowdsourced forecasts seem to significantly outperform
- 38:05traditional kind of
- 38:06SPEAKER_00: statistical forecasts or experts.
- 38:08SPEAKER_00: You seem to find a lot of alpha in crowdsourcing.
- 38:12SPEAKER_00: What's the explanation for that?
- 38:14SPEAKER_00: To what extent can that be leveraged by the intelligence community?
- 38:17SPEAKER_00: How useful is that to know that?
- 38:20So I think the math intuition here is that if you think of a
- 38:24human judgment as
- 38:26SPEAKER_01: including truth plus random error plus systematic bias,
- 38:35SPEAKER_01: when you take a bunch of human judgments and take the average judgment,
- 38:40SPEAKER_01: you're tending to cancel out the random error.
- 38:44Because if it's truly random,
- 38:46it's just as likely to be an overestimate as an underestimate.
- 38:49And just averaging over a large number of observations is going
- 38:52to cancel that out.
- 38:54SPEAKER_01: And if those judgments that you're averaging across...
- 38:59SPEAKER_01: are diverse,
- 39:01SPEAKER_01: they're based on different sources of information,
- 39:04SPEAKER_01: they're based on different mental models,
- 39:05SPEAKER_01: that will tend to cancel out the systematic bias.
- 39:08And so your estimate from taking that average is going to get
- 39:14closer and closer to the truth.
- 39:17This sort of math reason for taking the average of a large
- 39:23number of crowdsource judgments
- 39:25It goes all the way back to Francis Dalton and his experiments
- 39:30of getting crowds to
- 39:32SPEAKER_01: estimate the weight of an ox.
- 39:34SPEAKER_01: And it's been supported by...
- 39:37Lots of research since then and the work that we did at IARPA
- 39:41in the ACE program
- 39:43and in the forecasting tournaments that followed that supported
- 39:47this notion of
- 39:48SPEAKER_01: averaging large numbers of judgments.
- 39:51SPEAKER_01: There is a way of improving on that,
- 39:53which is finding subsets of people who are consistently more
- 39:59accurate than others.
- 40:01SPEAKER_01: Superforecasters.
- 40:02SPEAKER_01: Superforecasters.
- 40:04SPEAKER_01: And then taking the average of their judgments.
- 40:07SPEAKER_01: It's still,
- 40:08SPEAKER_01: though,
- 40:09taking an average of relatively large numbers of judgments as
- 40:14opposed to simply
- 40:16SPEAKER_01: taking the average of the superforecaster,
- 40:19SPEAKER_01: the super-duper forecaster,
- 40:21SPEAKER_01: who's consistently number one.
- 40:25SPEAKER_01: None of the methods of picking a single...
- 40:29SPEAKER_01: individual performed well,
- 40:31even if it was cherry picking the single best individual for a
- 40:36particular field.
- 40:38SPEAKER_01: That's a really important insight into analytic accuracy.
- 40:43SPEAKER_01: And it's one that not many institutions have adopted.
- 40:48And that's peculiar because it's a research finding that's been
- 40:53pretty robust
- 40:53SPEAKER_01: across different subjects.
- 40:57And because there's lots of institutions on the planet that you
- 40:59would think are
- 41:00SPEAKER_01: highly motivated to increase their accuracy,
- 41:04SPEAKER_01: why don't we see these crowdsourcing mechanisms more widely used?
- 41:08And why don't we see institutions making more regular use of
- 41:12research tournaments
- 41:13SPEAKER_01: internally to find their own super forecasters?
- 41:17SPEAKER_01: I don't have good answers to that question.
- 41:19It's something that Robin Hanson has,
- 41:23I think,
- 41:23a pretty cynical hypothesis about,
- 41:25which is that
- 41:27The leaders of most institutions aren't primarily motivated by
- 41:33increasing accuracy,
- 41:34but are more motivated by protecting their own special status
- 41:39or those of senior
- 41:41SPEAKER_01: leaders.
- 41:41SPEAKER_01: And crowdsourcing is inherently disruptive to leadership.
- 41:47This was Cotevilla's view about other failures of the
- 41:51intelligence community.
- 41:52It's hard to resist bureaucratic capture and to prevent these
- 41:55fundamental human challenges.
- 41:58SPEAKER_01: Yeah, I think you're right.
- 42:00And you do see other examples of this in institutions that
- 42:04collectively have
- 42:06SPEAKER_01: reasons to be anxious about either threats to their credibility or their
- 42:11SPEAKER_01: legitimacy.
- 42:13SPEAKER_01: Having your homework graded is uncomfortable.
- 42:16Almost everybody would prefer to grade their own homework and
- 42:19determine how it's
- 42:20SPEAKER_01: going to be graded.
- 42:21SPEAKER_01: These kinds of research tournaments have the homework of
- 42:27senior leaders graded in that the senior leaders are typically
- 42:30around the table in
- 42:33SPEAKER_01: a bog set process where they're
- 42:37SPEAKER_01: deliberating and reaching some sort of conclusion.
- 42:40And most of what we know from research and cognitive psychology
- 42:45and human judgment
- 42:47over the last 50 years suggests that that is not the best way
- 42:50to be making
- 42:51SPEAKER_01: judgments.
- 42:52SPEAKER_01: It might be one of the worst ways of making judgments.
- 42:55SPEAKER_01: And yet that is the norm in most institutions.
- 42:58SPEAKER_01: It's this unstructured group deliberation.
- 43:02SPEAKER_00: Has IARPA had...
- 43:04SPEAKER_00: Maybe this is a sensitive question,
- 43:05but has IARPA had success at pushing the intelligence community
- 43:09to do less of the
- 43:10bogsat bunch of guys sitting around the table judgment-making
- 43:15and more along these
- 43:17SPEAKER_00: lines,
- 43:17SPEAKER_00: leveraging crowds and averages and statistical insights?
- 43:23SPEAKER_01: For a while, it did.
- 43:25SPEAKER_01: For about nine or ten years, there was...
- 43:31SPEAKER_01: a crowdsourced analysis effort that was,
- 43:36SPEAKER_01: you know,
- 43:37SPEAKER_01: started at IARPA and then it became a kind of grassroots activity.
- 43:44SPEAKER_01: I mean,
- 43:44there were thousands of intelligence analysts across all of
- 43:47those agencies that
- 43:49were using crowdsourced tools to be making forecasts or
- 43:54analytic judgments.
- 43:56SPEAKER_00: When was this?
- 43:58SPEAKER_01: From 2019.
- 44:02SPEAKER_01: But to my knowledge, there isn't an activity like that right now.
- 44:08SPEAKER_01: There are activities like that in other intelligence communities.
- 44:10SPEAKER_01: For example,
- 44:11SPEAKER_01: the UK intelligence community has this effort called Cosmic Bazaar,
- 44:16SPEAKER_01: which was prompted by the IARPA effort and I think is still going strong,
- 44:23SPEAKER_01: includes crowdsourced judgments.
- 44:27So I think it's the sort of activity that unless you've got top
- 44:31cover and unless
- 44:33SPEAKER_01: you have a budget that's really dedicated to this and protected for it,
- 44:38it's going to fade away because managers aren't necessarily
- 44:41highly motivated to
- 44:43SPEAKER_01: keep this kind of activity going.
- 44:45SPEAKER_01: And analysts, working level analysts, love it.
- 44:50SPEAKER_01: And I mean,
- 44:50those were our strongest proponents for this kind of activity
- 44:54were the folks who
- 44:54SPEAKER_01: are working on a hard analytic question.
- 44:56And they wanted to see what other analysts across the
- 45:00intelligence community
- 45:01SPEAKER_01: thought about it and wanted to understand the reasoning.
- 45:03I wanted to see whether the probability assigned to conflict in
- 45:08Kashmir was going
- 45:09SPEAKER_01: up or was going down over the next month.
- 45:11and if if some analysts thought it was going up what what
- 45:14evidence were they
- 45:15looking at so you could turn it into an intelligence warning
- 45:18tool and a kind of
- 45:20like heart rate monitor for the analytic community to be
- 45:25understanding what was
- 45:26driving the anxiety of intel analysts um but that kind of tool
- 45:35it is it is one that
- 45:36can be contrarian right because you're especially for you're
- 45:40rewarding with
- 45:41bragging rights you know through like a leaderboard folks who
- 45:45are are accurate
- 45:46about things where the majority is wrong that is an important
- 45:50element of these
- 45:51kinds of crowdsourcing tools is you have to give bragging
- 45:56rights for folks who are
- 45:59SPEAKER_01: correct about something that is unpopular right
- 46:03SPEAKER_01: Because otherwise,
- 46:04SPEAKER_01: if it's just a popularity index,
- 46:07SPEAKER_01: then it's not going to be as useful analytically.
- 46:10SPEAKER_00: Sure.
- 46:11You've talked a little bit about the bureaucratic disincentives
- 46:16to doing this kind
- 46:16SPEAKER_00: of work or to,
- 46:18SPEAKER_00: you know,
- 46:19SPEAKER_00: rewarding low status correct decisions.
- 46:24I've got a quote from an interview you did a few years ago
- 46:27where you talk about
- 46:29kind of the same thing as applied to IR by that it's hard to
- 46:32create an organization
- 46:32SPEAKER_00: that rewards high risk activities and to sustain it in particular.
- 46:38How do you go about sustaining an organization that is an
- 46:41institution like this and
- 46:43stopping it from becoming another kind of bureaucratically
- 46:47captured institution?
- 46:49SPEAKER_01: I mean, a lot of it is just hiring great people.
- 46:52SPEAKER_01: Almost every enterprise,
- 46:54SPEAKER_01: like 90% of the variance is determined by the people who are hired,
- 47:00SPEAKER_01: the people who retain,
- 47:02SPEAKER_01: the people who are promoted.
- 47:04SPEAKER_01: And I think the ARPAs...
- 47:07work primarily because of great hiring and having unusual
- 47:13hiring authorities that
- 47:14SPEAKER_01: allows them to optimize hiring for people who are going to be very sharp,
- 47:19SPEAKER_01: creative,
- 47:20constructive contrarians and think differently about problems
- 47:25where there may
- 47:26already be a lot of inertia inside of a bureaucracy that you're
- 47:30going to have to
- 47:31SPEAKER_01: work against.
- 47:33It helps to have Congress on your side because Congress is
- 47:37ultimately appropriating
- 47:39SPEAKER_01: funds for these agencies.
- 47:41SPEAKER_01: And a track record really helps.
- 47:44SPEAKER_01: DARPA,
- 47:45SPEAKER_01: having played such an important role in stealth and GPS and the internet,
- 47:51SPEAKER_01: is going to have great congressional support.
- 47:56SPEAKER_01: So having some important wins matters.
- 47:59SPEAKER_01: And IARPA got some important wins,
- 48:03SPEAKER_01: particularly in the classified domain,
- 48:05SPEAKER_01: early on,
- 48:06SPEAKER_01: which helped.
- 48:07The other important things,
- 48:08I think,
- 48:09are being able to clearly communicate information.
- 48:13SPEAKER_01: to leaders why it's important to have a cell of misfits.
- 48:20SPEAKER_01: And I think a lot of leaders actually do get this.
- 48:23My favorite bosses all have gotten this,
- 48:26is that you want a group of misfits
- 48:29who you're sort of protecting and allowing them to operate in a
- 48:34way that is,
- 48:35SPEAKER_01: is different from other parts of an organization.
- 48:39SPEAKER_01: I mean, it should obviously all be legal.
- 48:42SPEAKER_01: It should all be ethical, but it could still be quirky.
- 48:45You know,
- 48:45it could be one where,
- 48:46you know,
- 48:47people are wearing shorts and flip flops and,
- 48:50um,
- 48:52where they don't have to worry as much about the kind of
- 48:57cultural conventions that
- 48:59SPEAKER_01: are common in other parts of an organization.
- 49:02SPEAKER_01: Because scientists and engineers tend to be pretty quirky.
- 49:06SPEAKER_01: They tend to have a stronger kind of libertarian bent,
- 49:11and you want a place where eccentric scientists and engineers
- 49:17can feel comfortable
- 49:19SPEAKER_01: and can work effectively.
- 49:22SPEAKER_01: We need nerds and we need institutions that attract nerds.
- 49:28Is the fact that IARPA is working on often secret or covert or
- 49:35non-public projects
- 49:37SPEAKER_00: that help attract the right kind of talent?
- 49:40Is it a barrier to people who want their name to be on public
- 49:44research papers?
- 49:46SPEAKER_00: How does that fact change hiring dynamics?
- 49:50There is a kind of selection effect in that the people for whom
- 49:58the most important
- 49:59SPEAKER_01: thing is getting their name on journal articles are not going to be
- 50:05SPEAKER_01: attracted to.
- 50:08So the types of people who might come to IARPA are folks who
- 50:14were academics and
- 50:15were really disappointed by the rat race for tenure or the
- 50:22kinds of incentive
- 50:24SPEAKER_01: systems within academic journals that
- 50:30prevented publication of works that was actually really
- 50:33important because of the
- 50:35SPEAKER_01: kind of publication incentive system.
- 50:39And to want to solve problems that are really important to the
- 50:43country and the
- 50:44SPEAKER_01: world,
- 50:46SPEAKER_01: for which there might not be much academic reward.
- 50:49For folks who are coming in from industry,
- 50:52people who didn't want to just develop the next
- 50:57SPEAKER_01: you know,
- 50:57SPEAKER_01: form of ad targeting,
- 51:00SPEAKER_01: you know,
- 51:01SPEAKER_01: like click through rates by,
- 51:04SPEAKER_01: but instead wanted to work on something that,
- 51:07SPEAKER_01: you know,
- 51:07SPEAKER_01: could save the world.
- 51:09SPEAKER_01: And so I think that tends to be a selection effect.
- 51:13The need for secrecy sometimes means then that you're also
- 51:17picking people who might
- 51:19SPEAKER_01: already not care as much about having a public profile or,
- 51:25I think actually a lot of those,
- 51:27that can be a quite positive selection effect.
- 51:29SPEAKER_01: I mean,
- 51:29that can tend to be finding people who are really
- 51:33mission-focused are pretty
- 51:36SPEAKER_01: humble,
- 51:37SPEAKER_01: both sort of by temperament,
- 51:39SPEAKER_01: but also kind of epistemically humble and self-critical.
- 51:44SPEAKER_01: So I was really amazed by the quality of people at IARPA.
- 51:50SPEAKER_01: I think it has a really strong self-selection effect.
- 51:54SPEAKER_00: When does an IARPA project that's working,
- 51:56SPEAKER_00: I mean,
- 51:57SPEAKER_00: what's the,
- 51:59some IARPA projects happen in public and some of them
- 52:03presumably are not for public
- 52:06SPEAKER_00: consumption.
- 52:07SPEAKER_00: Given IARPA's focus on tools not being used by adversaries or stolen,
- 52:13SPEAKER_00: how do you balance working in public and covert or secret projects?
- 52:21SPEAKER_01: Yeah.
- 52:23SPEAKER_01: IERPA developed a process that we called research technology protection,
- 52:29SPEAKER_01: which is before any program started,
- 52:31SPEAKER_01: we would do a cost-benefit analysis of making the program secret or not.
- 52:40SPEAKER_01: And
- 52:42It included questions like,
- 52:45you know,
- 52:45where are the researchers to solve this problem?
- 52:48SPEAKER_01: Are they people who are in academia,
- 52:51SPEAKER_01: in which case most of them don't have security clearances,
- 52:53SPEAKER_01: so we wouldn't be able to access them if the program is secret?
- 52:59SPEAKER_01: Does it require a mix of academics and, you know, clear contractors?
- 53:05SPEAKER_01: Does it require crowdsourcing itself to solve?
- 53:09SPEAKER_01: I mean,
- 53:09is it the sort of problem that you want to pose as a prize
- 53:11challenge rather than
- 53:13SPEAKER_01: one that,
- 53:13SPEAKER_01: you know,
- 53:14SPEAKER_01: give it to like three teams,
- 53:15SPEAKER_01: you give it to the world to solve?
- 53:18SPEAKER_01: You want to figure that out.
- 53:21in advance and decide whether the costs of secrecy exceed the
- 53:25benefits or the
- 53:27SPEAKER_01: benefits of secrecy exceed the costs.
- 53:31SPEAKER_01: And it really depends.
- 53:33SPEAKER_01: I mean,
- 53:33SPEAKER_01: there's some problems that I would have thought of as,
- 53:36SPEAKER_01: oh,
- 53:36SPEAKER_01: yeah,
- 53:36SPEAKER_01: this is much better to keep secret.
- 53:39SPEAKER_01: But then you actually do the math on this and you think,
- 53:43SPEAKER_01: actually,
- 53:43SPEAKER_01: the benefits of secrecy in this case aren't so significant.
- 53:48And we wouldn't be able to solve this problem if we tried to
- 53:51keep it secret.
- 53:53SPEAKER_01: And then there are others where the opposite is true.
- 53:55SPEAKER_01: Yeah.
- 53:56I do think that the questions that we added to the Heilmeier
- 54:00Catechism are ones
- 54:03SPEAKER_01: that bear on this question of...
- 54:06SPEAKER_01: There are certain kinds of research programs that by making them known,
- 54:12SPEAKER_01: you create a security dilemma in which,
- 54:16SPEAKER_01: one,
- 54:16you prompt other foreign intelligence services into thinking
- 54:20that you're doing
- 54:22something very dangerous and reckless and that you're planning
- 54:25to use some new
- 54:26SPEAKER_01: offensive weapon.
- 54:28SPEAKER_01: And you need to be really then thoughtful about...
- 54:32what sort of information can be misinterpreted by others if
- 54:37they're aware of an investment.
- 54:39SPEAKER_01: I think we also tend to overestimate our ability to keep secrets.
- 54:44SPEAKER_01: Why is that?
- 54:45I think in part because when you're operating in a classified
- 54:50environment,
- 54:51SPEAKER_01: you're so aware of the number of protections that you're going through.
- 54:55SPEAKER_01: The
- 54:57SPEAKER_01: The guards,
- 54:58SPEAKER_01: guns,
- 54:58SPEAKER_01: and gates,
- 54:59SPEAKER_01: you know,
- 54:59SPEAKER_01: at the front,
- 55:01SPEAKER_01: the special rooms with,
- 55:02SPEAKER_01: you know,
- 55:03SPEAKER_01: air gapping and,
- 55:04SPEAKER_01: you know,
- 55:05SPEAKER_01: no electronic devices except those that have been,
- 55:09SPEAKER_01: you know,
- 55:09SPEAKER_01: specially created for this room.
- 55:11SPEAKER_01: And so because you live in that environment, you assign people.
- 55:18SPEAKER_01: a high level of confidence to it,
- 55:20in part because it creates so many inconveniences for somebody
- 55:24personally to be
- 55:26SPEAKER_01: going into and out of that environment every day.
- 55:28SPEAKER_01: They just sort of think, well, this is really tough.
- 55:32But there's so many historical instances of our protections
- 55:36failing in one way or another.
- 55:37SPEAKER_01: And we should be more realistic about
- 55:43SPEAKER_01: the base rate of things getting stolen.
- 55:46SPEAKER_01: And if we're realistic about that,
- 55:48then I think that has implications for the kind of research
- 55:54that we pursue and how
- 55:56SPEAKER_01: we pursue it.
- 55:57SPEAKER_01: For example, if you develop a technology
- 56:00that would actually create an asymmetric disadvantage if used
- 56:05against you.
- 56:06Cyber weapons in general are a case of technology where the
- 56:10United States might be
- 56:11more vulnerable than some of our competitors because we have
- 56:14like a bigger attack
- 56:15SPEAKER_01: surface.
- 56:17SPEAKER_01: We're like a more open society.
- 56:19And so you should be especially thoughtful before developing
- 56:22cyber weapons of how
- 56:23those tools can be used against us because we might actually be
- 56:26more vulnerable in
- 56:27SPEAKER_01: some ways than our competitors.
- 56:31SPEAKER_01: The same might be true for advances in biology.
- 56:34So the United States has a strong taboo against the use of
- 56:38biological weapons.
- 56:39We have a legal set of commitments against developing or using
- 56:45biological weapons.
- 56:46We had a unilateral moratorium on biological weapons even
- 56:51before there was an
- 56:52SPEAKER_01: international treaty on it.
- 56:54SPEAKER_01: And not every country...
- 56:58SPEAKER_01: believes they're taboo.
- 56:59SPEAKER_01: And some countries have active offensive biological weapons programs.
- 57:04That's another one where the technologies that we're developing
- 57:07in synthetic
- 57:09SPEAKER_01: biology,
- 57:09SPEAKER_01: for example,
- 57:10SPEAKER_01: can create risks that are asymmetric for us.
- 57:14And we have a very open society,
- 57:16one that tends to be counter authoritarian in many ways.
- 57:20And so if there were a biological attack against the United
- 57:23States that required
- 57:25very rapid vaccination or antiviral distribution to the entire
- 57:30population.
- 57:31SPEAKER_01: And we saw what we went through with COVID,
- 57:33which was low rates of vaccine acceptance across many parts of
- 57:37the country.
- 57:38We're just asymmetrically vulnerable to infectious disease in
- 57:44the United States.
- 57:46And so that's another one where we should just be extremely
- 57:49cautious in the kinds of
- 57:52SPEAKER_01: technologies that we develop that can be weaponized by others.
- 57:56SPEAKER_01: So I think cyber and bio are two examples.
- 57:58In some ways,
- 57:59they draw from the same set of vulnerabilities and open
- 58:02democratic societies.
- 58:05SPEAKER_01: which is that we tend to have larger attack surfaces.
- 58:09SPEAKER_01: Our degree of openness and interaction means that infectious agents,
- 58:14SPEAKER_01: whether biological or digital,
- 58:17SPEAKER_01: are ones that get transmitted quickly in our societies.
- 58:21SPEAKER_01: And we're decentralized and counter-authoritarian in ways that make the
- 58:27SPEAKER_01: distribution of countermeasures quite difficult and inefficient.
- 58:37SPEAKER_00: Can you give me the inverse,
- 58:39an example of the inverse of the kinds of things that we might
- 58:43be especially
- 58:43interested in developing because we're asymmetrically protected
- 58:46from them or
- 58:47SPEAKER_00: adversaries are especially vulnerable?
- 58:50SPEAKER_01: Yeah, encryption or other privacy enhancing technologies.
- 58:54SPEAKER_01: You know,
- 58:56SPEAKER_01: democracies have an asymmetric advantage,
- 59:00SPEAKER_01: I think,
- 59:01SPEAKER_01: in developing and using those kinds of technologies.
- 59:05Technologies that inherently involve some amount of dissent or
- 59:11loss of control.
- 59:13SPEAKER_01: Large language models, I think, would be an example of that.
- 59:17The regulation around language models in China has been driven
- 59:24by an anxiety within
- 59:27the CCP of models spitting out anti-party rhetoric or providing
- 59:34historical
- 59:34SPEAKER_01: references that are not approved.
- 59:37SPEAKER_01: There are others.
- 59:38SPEAKER_01: Immigration,
- 59:39SPEAKER_01: I think,
- 59:39is a driver of immense innovation that democratic societies
- 59:46tend to have an
- 59:47SPEAKER_01: asymmetric advantage in leveraging.
- 59:50First,
- 59:50because STEM immigrants overwhelmingly prefer to live in
- 59:54democracies.
- 59:56SPEAKER_01: And second,
- 59:57SPEAKER_01: because democracies tend to be more receptive to receiving immigrants and
- 1:00:03SPEAKER_01: integrating immigrants.
- 1:00:06Another might be international standards and that the process
- 1:00:10for creating them is
- 1:00:13one in which the democratic process of those standards bodies
- 1:00:17is one that's more
- 1:00:17SPEAKER_01: compatible with how democracies think about the development of standards
- 1:00:21SPEAKER_01: domestically.
- 1:00:22And then there's a few areas of asymmetric advantage that
- 1:00:27probably aren't because
- 1:00:29SPEAKER_01: of some inherent advantage of democracies,
- 1:00:31SPEAKER_01: but are more because of historical inertia.
- 1:00:36SPEAKER_01: So,
- 1:00:36SPEAKER_01: for example,
- 1:00:37the moats around semiconductor manufacturing is due to a series
- 1:00:45of historical
- 1:00:46events that the United States had a lead in semiconductors
- 1:00:52going all the way back.
- 1:00:54SPEAKER_01: to the 50s and the design firms, the tool making firms grew up here.
- 1:01:02SPEAKER_01: The manufacturing ended up moving to other democratic states,
- 1:01:08SPEAKER_01: but still ones that were close allies and partners of the United States.
- 1:01:13SPEAKER_01: So I think,
- 1:01:13SPEAKER_01: you know,
- 1:01:14there are a variety of technologies that probably have some
- 1:01:17asymmetric advantage
- 1:01:19SPEAKER_01: for democracies.
- 1:01:19I think this,
- 1:01:20by the way,
- 1:01:20is like a really important and understudied question.
- 1:01:23SPEAKER_01: You know,
- 1:01:24SPEAKER_01: the Internet used to be one where folks would say like,
- 1:01:27SPEAKER_01: oh,
- 1:01:27SPEAKER_01: yeah,
- 1:01:27SPEAKER_01: that would be an asymmetric advantage for democracy.
- 1:01:30SPEAKER_01: Not necessarily.
- 1:01:32And it makes sense to be thinking about this as a bigger
- 1:01:36research enterprise for
- 1:01:38SPEAKER_01: technology strategy for policymakers.
- 1:01:42SPEAKER_01: is to think about what are the technologies that,
- 1:01:44SPEAKER_01: if we develop them,
- 1:01:45SPEAKER_01: will,
- 1:01:46SPEAKER_01: at the margins,
- 1:01:47SPEAKER_01: have a greater advantage to democracies versus autocracies.
- 1:01:52SPEAKER_01: Yeah, I'd love to see more work on that.
- 1:01:55SPEAKER_00: I would love to ask about your time at CSET.
- 1:01:57SPEAKER_00: Obviously, you founded this think tank, and...
- 1:02:02One thing we've been talking about recently internally at IFP
- 1:02:05is which kinds of
- 1:02:07information that we produce are most helpful for policymakers
- 1:02:10making decisions.
- 1:02:11SPEAKER_00: One thing a team member noticed was that we don't often create PDFs,
- 1:02:15but the national security apparatus loves PDFs and shares them
- 1:02:19all the time.
- 1:02:20SPEAKER_00: We are not in the habit of making them.
- 1:02:22SPEAKER_00: I'm curious,
- 1:02:22SPEAKER_00: maybe at whatever level of specificity you want to talk about it,
- 1:02:25SPEAKER_00: but how do you influence...
- 1:02:28SPEAKER_00: the national security apparatus?
- 1:02:29SPEAKER_00: How do national security folks make decisions?
- 1:02:31SPEAKER_00: What's the bogsat story for how these things get made?
- 1:02:34SPEAKER_00: And how do you play in that process most effectively?
- 1:02:38SPEAKER_01: First, I love your questions.
- 1:02:40SPEAKER_01: All of your questions have been awesome.
- 1:02:41SPEAKER_01: So I'm just, I'm so grateful that you're doing this work.
- 1:02:46SPEAKER_01: I mean,
- 1:02:46SPEAKER_01: I wish,
- 1:02:46SPEAKER_01: I mean,
- 1:02:48SPEAKER_01: just 20 years ago,
- 1:02:50I wish that your interviews had been available because they
- 1:02:52would have been just so
- 1:02:53SPEAKER_01: helpful to me.
- 1:02:54I was giving a talk recently in which I was sort of giving
- 1:02:57advice about policy careers.
- 1:03:00SPEAKER_01: And I think
- 1:03:04The thing that I found most surprising about the policy world
- 1:03:09is just how much is
- 1:03:11SPEAKER_01: based on interpersonal trust.
- 1:03:15SPEAKER_01: Because there aren't figures of merit that are
- 1:03:21SPEAKER_01: that are easily measurable in policy.
- 1:03:25You know,
- 1:03:25for many policies,
- 1:03:26it's going to be years before you see whether they were
- 1:03:29successful.
- 1:03:30SPEAKER_01: And it's a case of a group effort where the effect of
- 1:03:38SPEAKER_01: An individual is difficult to isolate.
- 1:03:41SPEAKER_01: So,
- 1:03:42so much of selection and advancement for personnel in the
- 1:03:48policy world is just
- 1:03:49SPEAKER_01: based on interpersonal trust.
- 1:03:53And it's really important then to invest in trust and to try
- 1:03:59not to degrade it.
- 1:04:01And I think there are a few things that people who want to have
- 1:04:05an impact in policy
- 1:04:08and improve the world should probably assign more weight to
- 1:04:12than they would
- 1:04:12SPEAKER_01: intuitively.
- 1:04:14SPEAKER_01: And one of them is to be kind,
- 1:04:17SPEAKER_01: because if you're in a stressful policy environment,
- 1:04:22your first instinct might not be to spend time being kind and
- 1:04:28making sure that your
- 1:04:29SPEAKER_01: colleagues are doing okay,
- 1:04:30SPEAKER_01: making sure that they're taking care of themselves,
- 1:04:33SPEAKER_01: that they're sleeping,
- 1:04:34SPEAKER_01: that they're eating,
- 1:04:35SPEAKER_01: making sure you bring dark chocolate to meetings and
- 1:04:40SPEAKER_01: But one,
- 1:04:40SPEAKER_01: you should,
- 1:04:41because these people almost certainly could be much wealthier
- 1:04:47doing something else.
- 1:04:48SPEAKER_01: They could have more free time doing something else.
- 1:04:50So there's already a selection effect where the people doing
- 1:04:53this job tend to be
- 1:04:55SPEAKER_01: people who have a level of altruism that's unusual.
- 1:04:58And just finding every opportunity to express appreciation for
- 1:05:02that is really important.
- 1:05:04But also,
- 1:05:06people really like being with kind people more than they do
- 1:05:11unkind people.
- 1:05:12And there are some kinds of institutions where being a
- 1:05:16brilliant jerk is tolerated.
- 1:05:20I think industry and academia tend to have a greater tolerance
- 1:05:26for brilliant jerks,
- 1:05:28SPEAKER_01: in part because you can measure their contribution and say,
- 1:05:33SPEAKER_01: well,
- 1:05:34SPEAKER_01: John's a jerk,
- 1:05:35SPEAKER_01: but we can tell that he's made this plus player.
- 1:05:39SPEAKER_01: Yeah, plus player, exactly.
- 1:05:42SPEAKER_01: He's a 100x programmer or whatever.
- 1:05:45SPEAKER_01: you can't really do that in the policy world.
- 1:05:48SPEAKER_01: And so nobody's willing,
- 1:05:50or very rarely are they willing to make the sacrifice to put up
- 1:05:56with somebody who's
- 1:05:58SPEAKER_01: a jerk,
- 1:05:58SPEAKER_01: even if they appeared to be brilliant.
- 1:06:02I know there are certainly counterexamples to this of people in
- 1:06:05the policy world
- 1:06:06SPEAKER_01: being known for being
- 1:06:08SPEAKER_01: abrasive and toxic,
- 1:06:12but I think it's actually very unusual compared to industry or
- 1:06:16even academia.
- 1:06:18I've just found much less evidence for that style of
- 1:06:23personality succeeding in the
- 1:06:26SPEAKER_01: policy world compared to those other parts of society.
- 1:06:30SPEAKER_01: In general,
- 1:06:31my experience was that the policy world is much more like West
- 1:06:35Wing or Madam
- 1:06:36SPEAKER_01: Secretary than it is like House of Cards.
- 1:06:39SPEAKER_01: It tends to be much more pro-social and compassionate.
- 1:06:47SPEAKER_01: So that's,
- 1:06:47SPEAKER_01: I think,
- 1:06:48SPEAKER_01: one thing that at CSET,
- 1:06:50we definitely were thoughtful about hiring people who are not
- 1:06:55only brilliant and
- 1:06:57SPEAKER_01: hardworking,
- 1:06:58but were also kind and compassionate and the sort of people
- 1:07:02that other folks are
- 1:07:03SPEAKER_01: going to enjoy working with.
- 1:07:05A second feature of the way we hired at CSET was to pick people
- 1:07:11who really were
- 1:07:13Theologically committed to finding facts and following them and
- 1:07:21were really
- 1:07:21tenacious about getting evidence for or against a particular
- 1:07:27policy and not having
- 1:07:28a strong default position on the policy question before they
- 1:07:35started working on it.
- 1:07:37And a third feature I think was finding people who believed
- 1:07:42that analysis would be
- 1:07:43SPEAKER_01: improved by spending some amount of time in government.
- 1:07:46SPEAKER_01: So folks who were interested in being researchers,
- 1:07:49SPEAKER_01: but were also interested in,
- 1:07:51SPEAKER_01: you know,
- 1:07:51SPEAKER_01: spending a year or two in government after a year or two of analysis,
- 1:07:55SPEAKER_01: and then coming back that in fact,
- 1:07:58SPEAKER_01: the experience of spending time in government would make them even better
- 1:08:01SPEAKER_01: researchers.
- 1:08:03SPEAKER_01: So I don't know.
- 1:08:04Those were some of the characteristics that we looked for when
- 1:08:06we were hiring.
- 1:08:08SPEAKER_00: And in terms of the product,
- 1:08:12the materials and the things that you're creating for the
- 1:08:16national security
- 1:08:17SPEAKER_00: apparatus,
- 1:08:18what kinds of any tips and tricks there for what the best
- 1:08:23outputs for a think tank
- 1:08:24SPEAKER_00: are if you want to be effective?
- 1:08:27SPEAKER_01: Yeah,
- 1:08:27it can vary because the policy audience is so broad and some of
- 1:08:33them want a
- 1:08:35SPEAKER_01: one-pager and you're not going to get them to read more than that.
- 1:08:41Others are going to be willing to really absorb a 200-page
- 1:08:46report or their staff is
- 1:08:48SPEAKER_01: going to be willing to
- 1:08:50SPEAKER_01: absorb a 200-page report.
- 1:08:53SPEAKER_01: And so knowing your audience is, I think, the important first step.
- 1:08:58SPEAKER_01: Yeah, let me think.
- 1:08:59SPEAKER_01: So first, picking the important problems.
- 1:09:02SPEAKER_01: And in this doc that I'll send you, I...
- 1:09:06SPEAKER_01: I have at least a few heuristics that I try to follow.
- 1:09:09One is the Hamming question about what is the most important
- 1:09:13problem in your field.
- 1:09:14SPEAKER_01: I think in general,
- 1:09:16people underestimate their own potential to make contributions
- 1:09:21to the most
- 1:09:21SPEAKER_01: important problems.
- 1:09:22They probably overestimated how many people are already working
- 1:09:25on the most important problems.
- 1:09:28SPEAKER_01: And they underestimate the impact that they can have on the margins
- 1:09:33Because there's so many incredibly important problems that are
- 1:09:36just really neglected.
- 1:09:38SPEAKER_01: I mean,
- 1:09:38SPEAKER_01: like,
- 1:09:38SPEAKER_01: for example,
- 1:09:38SPEAKER_01: this question about asymmetric advantages for democracies from different
- 1:09:43SPEAKER_01: technologies.
- 1:09:44SPEAKER_01: I don't think there's been,
- 1:09:44at least I haven't seen a really deep strategic analysis of
- 1:09:48that question.
- 1:09:50SPEAKER_01: And there's something I call the mummy G conjecture,
- 1:09:52which is named after my mother-in-law because I was helping her
- 1:09:57check luggage once.
- 1:09:59And she had this like giant suitcase and I asked her who's
- 1:10:01going to pick this thing up.
- 1:10:02And she was like,
- 1:10:03if you don't know whose responsibility it is,
- 1:10:05then it's probably yours.
- 1:10:07SPEAKER_01: Yeah.
- 1:10:08I think there's a lot of problems like that where people just
- 1:10:10assume that there's
- 1:10:11SPEAKER_01: somebody else working on them.
- 1:10:13SPEAKER_01: If you can't figure it out after like a few days worth of homework,
- 1:10:18SPEAKER_01: then it is probably a neglected problem.
- 1:10:20SPEAKER_01: And so it's probably up to us to solve it.
- 1:10:23And then there's just a lot of important problems that for
- 1:10:26different institutional
- 1:10:28SPEAKER_01: reasons are neglected.
- 1:10:30SPEAKER_01: And in fact, problems that are neglected
- 1:10:34SPEAKER_01: That is a kind of Bayesian evidence that the problem is likely to be more
- 1:10:40SPEAKER_01: cost-effective to solve than problems that are popular.
- 1:10:43SPEAKER_01: Because if a problem is popular,
- 1:10:44SPEAKER_01: it has lots of people trying to solve it,
- 1:10:46SPEAKER_01: and it hasn't already been solved,
- 1:10:47SPEAKER_01: then it's probably really hard.
- 1:10:49SPEAKER_01: And your impact on the margins is likely to be low.
- 1:10:52SPEAKER_01: So anyway,
- 1:10:53I think the problem selection really needs to be step one for
- 1:10:56figuring out what's
- 1:10:57SPEAKER_01: the product.
- 1:10:59SPEAKER_01: Step two is,
- 1:11:01SPEAKER_01: and this really should precede the choice of methods and everything else,
- 1:11:05is figure out who is the decision maker for solving this
- 1:11:09problem that your analysis
- 1:11:10SPEAKER_01: will ultimately need to reach.
- 1:11:12SPEAKER_01: And how do they receive information in a way that's useful?
- 1:11:15SPEAKER_01: And that includes not just the style of product or its length, but
- 1:11:24Or it's,
- 1:11:25you know,
- 1:11:25where it's contents,
- 1:11:26like,
- 1:11:26is it just a graph rather than text?
- 1:11:29SPEAKER_01: Or is it like a dashboard?
- 1:11:32SPEAKER_01: But also it's timeline.
- 1:11:34So,
- 1:11:35you know,
- 1:11:35a project that takes two years when you know a decision maker
- 1:11:39needs to reach...
- 1:11:41SPEAKER_01: closure on an executive order in three months.
- 1:11:44SPEAKER_01: I mean, that project isn't going to be useful if delivered late.
- 1:11:49SPEAKER_01: So the choice of audience needs to inform content and quality and style.
- 1:11:57SPEAKER_01: It can also influence the level of certainty or rigor.
- 1:12:02I mean,
- 1:12:02there's kind of diminishing marginal utility in certainty or
- 1:12:07polish in
- 1:12:08SPEAKER_01: And as long as you appropriately caveat your conclusions by saying,
- 1:12:14SPEAKER_01: you know,
- 1:12:14SPEAKER_01: I only have like 70% confidence in this.
- 1:12:17SPEAKER_01: With another six months, I could reach 80% confidence.
- 1:12:21SPEAKER_01: I think,
- 1:12:22SPEAKER_01: you know,
- 1:12:22decision makers in general prefer to receive information
- 1:12:26earlier with a trade-off
- 1:12:28SPEAKER_01: and certainty as long as they understand the trade-off.
- 1:12:33SPEAKER_01: In terms of content,
- 1:12:35I do think there is a generational change that we were
- 1:12:37conscious of at CSET.
- 1:12:50SPEAKER_01: In some ways,
- 1:12:50the model that we had been operating under is the RAND model of
- 1:12:56producing reports.
- 1:12:58SPEAKER_01: And we hired from RAND.
- 1:13:00SPEAKER_01: Our head of analysis was from RAND.
- 1:13:03SPEAKER_01: The style of the reports looks like RAND.
- 1:13:07SPEAKER_01: I think it's the same type font.
- 1:13:09SPEAKER_01: Sure.
- 1:13:09SPEAKER_01: But there is a generational change going on,
- 1:13:12SPEAKER_01: which is that more policymakers grew up with,
- 1:13:16SPEAKER_01: with phones and expect apps,
- 1:13:18SPEAKER_01: they expect dashboards.
- 1:13:20They don't necessarily want to report as much as they want an
- 1:13:22annotated map or a,
- 1:13:24a
- 1:13:28a toggle that they can sort of look through a bunch of
- 1:13:31scenarios by changing what
- 1:13:33SPEAKER_01: an assumption is and understand its dynamic effects on an outcome.
- 1:13:38I think that is going to change the way that we produce
- 1:13:41analysis for policymakers.
- 1:13:43I think CSET was pretty early and seeing that shift and
- 1:13:48starting to respond to it
- 1:13:49SPEAKER_01: by,
- 1:13:49SPEAKER_01: you know,
- 1:13:50SPEAKER_01: creating more dashboards,
- 1:13:51creating more of these kind of like tech observatories where
- 1:13:54you could actually dig
- 1:13:55SPEAKER_01: into the data and run scenario analysis.
- 1:13:58And I think the policy analysis world will probably need to be
- 1:14:02doing more of that
- 1:14:02SPEAKER_00: over the years.
- 1:14:04SPEAKER_00: Two last questions for you on forecasting specifically.
- 1:14:07One is I just want to go back to your point about differential
- 1:14:12technologies,
- 1:14:13about which technologies may be asymmetrically helpful for
- 1:14:17democracies and which
- 1:14:19SPEAKER_00: may not.
- 1:14:20SPEAKER_00: How confident are you about this mode of analysis?
- 1:14:23SPEAKER_00: You mentioned the
- 1:14:25SPEAKER_00: case of the internet,
- 1:14:26SPEAKER_00: which everybody I think assumed was a boon to democracy and now is a more
- 1:14:32SPEAKER_00: complicated question.
- 1:14:35SPEAKER_00: Given that kind of case track record,
- 1:14:38how do we think about making these kinds of predictions when
- 1:14:41the possible effects
- 1:14:43SPEAKER_00: 20 years down the line are very hard to know?
- 1:14:45SPEAKER_01: Yeah, I mean, I think we should treat it skeptically.
- 1:14:48I mean,
- 1:14:48we should make sure that we're really thinking from both sides
- 1:14:50of the game board.
- 1:14:52SPEAKER_01: And I don't know if anybody,
- 1:14:57SPEAKER_01: you know,
- 1:14:57SPEAKER_01: at DARPA or elsewhere,
- 1:14:58SPEAKER_01: when we were thinking about the Internet,
- 1:15:03SPEAKER_01: sort of thought,
- 1:15:04SPEAKER_01: OK,
- 1:15:04SPEAKER_01: suppose I'm an authoritarian state.
- 1:15:06SPEAKER_01: How am I going to use the Internet?
- 1:15:09Suppose I'm a revanchist state that wants to disrupt some
- 1:15:14democratic power.
- 1:15:15SPEAKER_01: How am I going to launch disinformation attacks?
- 1:15:18SPEAKER_01: I don't know that anybody was actually given that analytic task.
- 1:15:22SPEAKER_01: So I think this goes again to our discussion of red teaming.
- 1:15:28I think for a question like what technologies provide an
- 1:15:33asymmetric advantage for
- 1:15:35SPEAKER_01: democracies,
- 1:15:37you should have somebody who's playing the role of the autocrat
- 1:15:42who thinks about
- 1:15:43how they will use that technology in order to break you or in
- 1:15:47order to advance
- 1:15:48SPEAKER_01: their own set of objectives and
- 1:15:51And that goes for the differential technology development for
- 1:15:55safety and security too.
- 1:15:56I mean,
- 1:15:56we have some bets on which kinds of technologies are defense
- 1:16:00dominant.
- 1:16:01And we should have red teams that are really thinking through
- 1:16:07whether that defense
- 1:16:09SPEAKER_01: dominance is real.
- 1:16:12SPEAKER_00: But I guess to push on that,
- 1:16:15it's hard for me to imagine what would have been different
- 1:16:18about the development of
- 1:16:18SPEAKER_00: the internet if you'd had...
- 1:16:20SPEAKER_00: even if you'd kind of known in advance,
- 1:16:23authoritarians will use this to spread disinformation and it
- 1:16:26will be firewalled in
- 1:16:28SPEAKER_00: autocratic regimes.
- 1:16:30SPEAKER_00: I guess,
- 1:16:30what's the positive case for how that kind of red teaming would
- 1:16:34shape that
- 1:16:35SPEAKER_00: technological flow?
- 1:16:37SPEAKER_01: Yeah,
- 1:16:38I think there have been a few papers written about if we were
- 1:16:42designing the
- 1:16:42SPEAKER_01: internet from scratch again,
- 1:16:44SPEAKER_01: like here's how we probably could have done better.
- 1:16:48SPEAKER_01: For example, thinking about like
- 1:16:56SPEAKER_01: Having packets that include headers that are not completely vulnerable to
- 1:17:01SPEAKER_01: anonymization,
- 1:17:04SPEAKER_01: to obfuscation,
- 1:17:05SPEAKER_01: to misattribution.
- 1:17:07SPEAKER_01: Having some way of assigning credibility to some nodes versus others.
- 1:17:16Having also a legal framework that would precede the
- 1:17:21development of opportunities
- 1:17:23for regulatory capture by large entities so that you could have
- 1:17:29certain kinds of
- 1:17:30SPEAKER_01: defenses against disinformation that...
- 1:17:36that would have preceded the case where you've got a handful of
- 1:17:41companies that are
- 1:17:42managing large information platforms that are no longer
- 1:17:45interested in preventing
- 1:17:47disinformation as long as that disinformation is generating
- 1:17:50advertising revenue.
- 1:17:53SPEAKER_01: I think we really could have done a better job.
- 1:17:55SPEAKER_01: I don't think we would have gotten it perfect.
- 1:17:59SPEAKER_01: And I certainly don't think that there's an area where
- 1:18:03You know,
- 1:18:03DARPA or NSF or the other federal funders would have said,
- 1:18:08eh,
- 1:18:08pass.
- 1:18:09SPEAKER_01: We're not going to develop this.
- 1:18:12SPEAKER_01: This looks too potentially toxic.
- 1:18:16But I do think that we could have baked in certain kinds of
- 1:18:20defenses against disinformation.
- 1:18:21SPEAKER_01: Yeah.
- 1:18:23in the way that we had designed the internet and its early
- 1:18:27regulatory structure.
- 1:18:30The same thing I think is true of some technologies that we
- 1:18:33have now that are distributed.
- 1:18:37SPEAKER_01: We should have been more thoughtful about DNA synthesis,
- 1:18:40SPEAKER_01: that if you're creating DNA synthesizers,
- 1:18:44SPEAKER_01: you might
- 1:18:45SPEAKER_01: You know,
- 1:18:45SPEAKER_01: each one of which is,
- 1:18:47SPEAKER_01: has more destructive potential than,
- 1:18:50SPEAKER_01: you know,
- 1:18:51SPEAKER_01: a uranium enrichment facility.
- 1:18:54you probably want to figure out some way of creating some baked
- 1:18:59in level of
- 1:19:00SPEAKER_01: security and safety.
- 1:19:02SPEAKER_01: And we didn't do that.
- 1:19:03And now there's commercially available DNA synthesizers that
- 1:19:07somebody could use to
- 1:19:08SPEAKER_01: recreate smallpox or something worse for under $100,000 at this point.
- 1:19:15SPEAKER_01: And I think we really didn't even ask the question at the time that DNA
- 1:19:21SPEAKER_01: synthesizers were being developed.
- 1:19:24Is there any way of putting some form of security or safety
- 1:19:31into this technology from the start?
- 1:19:33SPEAKER_01: Yeah.
- 1:19:34SPEAKER_01: And I think for things like AI now,
- 1:19:38we're having this discussion because it still feels like early
- 1:19:41days in the kinds of
- 1:19:42SPEAKER_01: tools that we're making to ask,
- 1:19:45SPEAKER_01: well,
- 1:19:46SPEAKER_01: can you build alignment into models when they're being built?
- 1:19:50SPEAKER_01: Can you do red teaming much earlier on?
- 1:19:53SPEAKER_01: And now they're, I mean, I think this is actually a positive thing.
- 1:19:56example of red teaming is that you're seeing most of the major
- 1:19:59AI labs doing red
- 1:20:02SPEAKER_01: teaming,
- 1:20:02SPEAKER_01: like sincerely,
- 1:20:03SPEAKER_01: I mean,
- 1:20:04SPEAKER_01: really trying to figure out how the tools could be misused.
- 1:20:08SPEAKER_01: So I think we're going to see out of that a lot of baked in security.
- 1:20:12SPEAKER_01: I mean,
- 1:20:12SPEAKER_01: reinforcement learning with human feedback,
- 1:20:15the way in which that's being applied is an example of trying
- 1:20:17to bake in guardrails
- 1:20:19SPEAKER_01: from the start rather than kind of retrofit them.
- 1:20:23SPEAKER_00: That's a really helpful answer for me.
- 1:20:25Will you send me any of those papers that you mentioned about
- 1:20:29differential
- 1:20:30SPEAKER_00: development of the internet?
- 1:20:33SPEAKER_00: Yeah.
- 1:20:33SPEAKER_00: I'll follow up in an email.
- 1:20:34SPEAKER_00: But last question for you before I let you go.
- 1:20:38If you had full autonomy and authority over embedding
- 1:20:43forecasting work in the US
- 1:20:44SPEAKER_00: government,
- 1:20:45SPEAKER_00: if you got to supercharge your IR stuff,
- 1:20:48SPEAKER_00: what would be top of list?
- 1:20:50In an ideal world,
- 1:20:51how is the US government using forecasting to make decisions?
- 1:20:56I think ensuring that a place like the National Security
- 1:21:00Council has its key
- 1:21:04questions on a prediction market or some other sort of
- 1:21:09forecasting platform would
- 1:21:11SPEAKER_01: be the thing I would focus on.
- 1:21:14Having worked there and seeing the time pressure that folks are
- 1:21:19operating under,
- 1:21:21SPEAKER_01: And you don't have time to do research.
- 1:21:25SPEAKER_01: You don't have time to read very much.
- 1:21:29You're operating under sleep deprivation and hunger and with
- 1:21:35some of the most
- 1:21:36SPEAKER_01: consequential decisions that the country faces.
- 1:21:41And you just don't have nearly as much time to read and collect
- 1:21:48information.
- 1:21:49SPEAKER_01: So finding an efficient way of
- 1:21:52SPEAKER_01: of crowdsourcing forecasts that are conditional.
- 1:21:57SPEAKER_01: If I pursue option for action A, this is likely to be the consequence.
- 1:22:02SPEAKER_01: If I pursue action B, this is likely to be the consequence.
- 1:22:07SPEAKER_01: I would have loved having access to a tool like that
- 1:22:11It's also the case that the intelligence that reaches the NSC
- 1:22:17has been sort of
- 1:22:18SPEAKER_01: whittled down and you don't have access to a broad view.
- 1:22:23You don't have time to ask for that coordinated analytic
- 1:22:27product like a national
- 1:22:28SPEAKER_01: intelligence estimate because that takes months.
- 1:22:32But what you want is really some kind of EKG for the analytic
- 1:22:38community about a
- 1:22:39SPEAKER_01: particular problem.
- 1:22:40SPEAKER_01: at any given moment.
- 1:22:42And it's also incredibly valuable in a place like the NSD to be
- 1:22:51given some
- 1:22:53SPEAKER_01: indication when you might be making a catastrophic mistake.
- 1:22:57SPEAKER_01: And it's very hard for
- 1:23:01SPEAKER_01: intelligence briefers,
- 1:23:03SPEAKER_01: you know,
- 1:23:03to be collecting information from enough analysts to be able to
- 1:23:09indicate whether
- 1:23:10SPEAKER_01: you could be making a catastrophic mistake.
- 1:23:13SPEAKER_01: So anyway,
- 1:23:14that's probably the place where I think having access to more
- 1:23:20accurate forecasts,
- 1:23:21SPEAKER_01: more timely forecasts,
- 1:23:23SPEAKER_01: and ones where there's strong incentives for
- 1:23:28SPEAKER_01: telling the truth,
- 1:23:30SPEAKER_01: even when it's going to be extremely inconvenient,
- 1:23:35SPEAKER_01: that's the place where it would be most valuable.
- 1:23:37SPEAKER_00: Jason, thank you for giving us so much of your time.
- 1:23:41SPEAKER_00: I know we're way over.
- 1:23:43SPEAKER_01: Thanks so much.
- 1:23:44SPEAKER_01: And really, thanks again for doing this.
- 1:23:46SPEAKER_01: It's really, really valuable to the world.
- 1:23:50SPEAKER_01: And getting folks to think about institutional design,
- 1:23:53SPEAKER_01: incentive mechanisms,
- 1:23:55SPEAKER_01: you know,
- 1:23:55SPEAKER_01: heuristics that folks can use in order to make better policy.
- 1:23:59SPEAKER_01: I think all that's really valuable.
- 1:24:01Yeah,
- 1:24:02it's been a pleasure to do these kinds of interviews,
- 1:24:04including this one.
- 1:24:04SPEAKER_00: So thanks for the opportunity, Jason.
- 1:24:07SPEAKER_01: Of course.
- 1:24:07SPEAKER_01: Take care.