Latest / 80,000 Hours Podcast / #204 – Nate Silver on making sense of SBF, and his biggest critiques of effective altruism

Transcript

#204 – Nate Silver on making sense of SBF, and his biggest critiques of effective altruism · 80,000 Hours Podcast

• Nate Silver: People think of tilt as what happens, which it often does, when you’re on a losing streak or take a bad beat. And therefore, you can have different reactions: you can either try to chase your losses, or just as often people become way too tentative and risk averse. But winners’ tilt can be just as bad, right? If you have a couple of bets in a row that pay off, especially if they’re contrarian bets, it’s one of the things I think Elon Musk’s issues is, or Peter Thiel, for example. If you make a couple of contrarian bets and they pay off, it’s really satisfying to get a financial purse; it’s really satisfying to prove people wrong — and if you get both at once, I mean, that’s like some drug cocktail, seriously. That has profound effects on you. And then if you do that like twice, have a couple of bets that pay off, it’s very hard to outlive that in some ways. And if it goes wrong, then you’re kind of chasing the high that you had before. It’s hard to get off that roller coaster, I think. The kind of instant, gamified feedback — especially through Twitter in particular, which I think seems to drive certain people, maybe including the founder or the owner of Twitter, slightly crazy — I think that kind of is an accelerant. Rob Wiblin: Hey everyone, Rob Wiblin here. Today I speak with election forecaster and author Nate Silver about: His theory of Sam Bankman-Fried. The culture of effective altruism and rationality. How Nate would do effective altruism better, or differently. Whether EA is incompatible with game theory. How similar Sam Altman and Sam Bankman-Fried really are. Whether it’s selfish to slow down AI progress. The ridiculous 13 Keys to the White House. Whether prediction markets are now overrated. And whether venture capitalists talk a big talk about risk while pushing all the risk off onto entrepreneurs. The conversation orbits Nate’s recent book, On the Edge, where he lays out an elite culture clash he thinks is key for understanding our times: There’s “the Village,” which he says is “basically the liberal establishment. Harvard and the New York Times; academia, media and government.” And then there’s what he calls “the River,” which he defines as “very analytical but also highly competitive.” There’s many different streams to the River — including poker and sports betting, quant finance, tech entrepreneurs and crypto, and effective altruism and rationality. Common cultural traits include comfort with challenging authority, decoupling, contrarianism, using explicit models, a higher risk tolerance — and in my mind, the most important of all: the use of expected value calculations to decide what risks to take. Nate thinks the River has been gaining influence at the expense of the Village, which has had a mix of good and bad effects — mostly good, in his and my view — and brought the River and Village into open conflict. We mostly stick to topics he hasn’t already addressed elsewhere, so if you want to hear that story you’ll need to grab the book, On the Edge. All right, without further ado, here’s Nate Silver. Today I’m speaking with Nate Silver. Nate, as I imagine most listeners will know, is the creator of the FiveThirtyEight website and election forecasting system — which I imagine many people like me have doom-refreshed over the many years since it’s been running. I think I first started looking at it back in 2007 during the Obama-McCain election. You’ve brought me sanity and anxiety in equal measure over the years. But you’ve since sold FiveThirtyEight and you now publish your election model through the Silver Bulletin on Substack that people can check out. But we’re here chatting today because you’ve written this book On the Edge: The Art of Risking Everything — which talks about, on the one hand, gambling, sports gambling, poker, that kind of thing; and then in the second half, you turn to effective altruism, rationalism, existential risk, AI, that sort of thing. And I guess Sam Bankman-Fried, who you spoke with at some length after his downfall. So we’re going to talk about a bunch of those themes over the course of the conversation. Thanks so much for coming on the show. Nate Silver: Of course. I’m always happy to be on a show I actually listen to. Rob Wiblin: Let’s talk about effective altruism a bit. That’s the thing I want to spend the most time on, because I think you haven’t been asked about that so much in other interviews. You did interviews with quite a lot of people involved in the rationality community and the effective altruism community. And I think you try to do us justice — try to be fair and point out the good things — but you also don’t pull your punches, and you point to a lot of different ways in which people have criticised it, maybe legitimately. I was very interested to hear a bit more about how, when you’re building anything, you have to make a lot of difficult decisions, a lot of difficult tradeoffs — Are you going to be more political or less political? Are you going to be more lavish or more austere? — and usually there’s competing considerations on either side. And someone who has a strong preference either way, you’re going to get criticised, probably from both directions, if you’re striking a balance. So I was kind of curious to hear your overall takes on where we could be better in practice. Not just what are possible weaknesses, but where would you direct things differently? And maybe where would you have directed things differently if you were helping to set things up back in 2011? Nate Silver: The kind of funny thing is that the book is about a certain type of person who is very analytical and nerdy and good at things like decoupling — where you’re removing context from something — where they have an inclination to quantify things, even things that are hard to quantify. Poker players have this, sports bettors have this, people in venture capital have this to some degree — it’s a slightly different skill set, but close enough — and the EAs have this. But in every other field, you’re kind of competing by some standard where you get feedback, I suppose, and you have this incentive to be accurate — which is often a financial or career-adjacent incentive. I guess the irony of the book is that you might think the EAs are really unselfish, which I think they are. I think they literally are, in most cases, altruistic. And I think their hearts are in the right place, and I think they’re very intelligent people. And just the fact that — whether it’s because of EA or kind of EA-adjacent — you have all these multibillionaires now donating a lot of their net worth, and at least making some efforts to see that that money is donated well… I don’t know if Bill Gates would call himself an EA, but he clearly espouses some of the same ideas and is very rigorous about it. That is doing a huge amount of good for the world. But the irony is, not having as much skin in the game, I think sometimes EAs don’t learn as much about the limitations of models, so to speak, and may also be — as in the case of SBF — a little bit too trusting. In the poker world, we learn to have a healthy amount of distrust, I suppose. Rob Wiblin: Yeah. On the trust point, I think you quote someone in the book that, in 2022, “effective altruism felt like the world’s biggest trust bubble.” Do you feel like that bubble has popped to an appropriate degree? I think we’ve become less trusting. Have we become the right amount less trusting? Nate Silver: Yeah, maybe it’s about the right [amount]. This was Oliver Habryka, I think, who is kind of EA. Everyone says they’re EA- or rationalist-adjacent. It’s still a pretty small world, if you go to the Manifest conference, for example — which I probably would say it’s more rationalist than EA. But it’s not a huge number of people. You can trace most of the intellectual history of these movements by profiling 10 or 12 people prominently, which is not tiny, but not super large. Yeah, I think it probably errs a little bit on the side of over-trusting, and maybe could use more outside view within the movement. But really, some of it’s more critique of utilitarianism, especially the Peter Singer, maybe more strict, “impartial” form of utilitarianism. I’m more sympathetic to rule utilitarianism, for example. That’s part of it. Part of it is I think you have to update significantly on SBF. This is where Scott Alexander and other people have said, “Well, who could have known that he would be a once-in-a-generation fraud?” But if you get into risk assessment, and you’re not assessing risks to your internal movement… And it wasn’t like he was running some secret sex ring on the side, right? He was kind of, in his core activities, being untrustworthy in ways that did give off a lot of signs. I mean, he told Tyler Cowen, the economist/podcast extraordinaire, that if he could press a button to make the world 2x plus some epsilon as good, with a 50/50 chance of blowing the world up, then he would press that button repeatedly. I don’t think I want that guy to be the major funder — along with Dustin Moskovitz — but I wouldn’t want him to be a major funding source of EA. And the way he founded FTX and former people at Alameda were not that happy. So I think, given that it’s a small world and that he was a big part of the small world, you have to update significantly for now based on SBF. Rob Wiblin: Yeah. The quote where he said he’ll destroy the world with 50% chance if he could more than double it: you say in the book that a lot of people with a background in philosophy say stuff like that. But I heard that and I was like, “This is philosopher talk. No one would actually do that kind of thing.” And I think in the great majority of cases where people say stuff that sounds a bit crazy like that, that is the case: that they’re just in a philosophy seminar room, basically. I didn’t actually think that even Sam meant it. How do you tell whether people actually are crazy when they’re willing to indulge in thought experiments of that kind? Nate Silver: I talked to Sam, and you’re right. I mean, there are definitely people in this space that will be kind of trollish or provocative, or it’s a thought experiment and that maybe is kind of left unstated. Even the Nick Bostrom paperclip scenario is kind of sometimes taken too literally. It’s a thought experiment that’s kind of cheeky and funny… I mean, the result wouldn’t be funny, obviously. Look, I think Sam was relatively serious about this. He was very consistent about saying this. I talked to Sam about five or six different times — before, during, and after the bankruptcy — and he was pretty consistent about saying that if you don’t take enough risk to literally destroy yourself, then you’re not maximising your expected value enough. I don’t think I’ll get into this Kelly criterion thing, which is a sports betting formula, but he is willing… Basically, the “rational” thing to do is to maximise your expected return conditional on not going broke, on having very low risk of ruin. We can call Elon Musk a risk taker for buying Twitter/X at a price of $50 billion, but unless there are shareholder lawsuits that get out of hand, there’s no risk of ruin, existential risk to Elon Musk from if it’s a poor financial purchase of Twitter. Whereas Sam really did think that if you are not maximising your chances of becoming the world’s first trillionaire and/or the first autistic US president, then you’re not taking enough risk. So I think there’s some degree of pathology there. Rob Wiblin: I never believed that Sam really, really meant it to that degree. And the reason was that obviously he talked about how he cared a lot about existential risk, about AI, about these kinds of relatively niche causes. Obviously, if you have a billion dollars, you can fund many of the things you care about within that area. If you have $10 billion, now you’re going to be really struggling to find anything else to fund. There’s hugely declining returns on dealing with these issues. Why would you double down and take a risk of ruining yourself and ruining everyone around you in order to get a trillion dollars, when it’s not even clear what you would spend that money on? It made no sense. So I just assumed that he was just mouthing off, saying, “We should take a bit more risk than people usually do.” What was going on? Was he kind of an idiot about this on some level? I don’t get it. Nate Silver: I mean, he made several bad risks, right? The decisions that he made. For example, during his trial, about whether to testify or not. The source I talked to after — a named source in the book, a crypto attorney — was like, “The government had him dead to rights. Caroline Ellison’s testimony is extremely persuasive. Sam is caught lying multiple times, and also contradicting things that he told me, for what it’s worth. He’s not a very sympathetic defendant. All that he’ll do by going on trial will be pissing off the judge, who’s a no-nonsense judge, and the jury, and giving himself a longer sentence.” Which is exactly what he did. He probably cost himself, pending appeal, an additional 10 or 15 years in prison by insisting on taking the witness stand. I talked to him in Palo Alto four or five months before the trial, and said, “What if they offered you a two-year plea deal? Two years, slap on the wrist. After that, you probably can’t do crypto stuff again, but two years and then you can get some new money and do a new startup.” And he was like, “I’d have to think about it.” Rob Wiblin: You shouldn’t think about it. Nate Silver: I think he’s not very good at assessing risk, or has some type of destructive streak. I think too — and this is me being a little bit more speculative — Sam is somebody who says that he has Asperger’s, or is on the spectrum, and yet is kind of thrust into an environment where he is socialising a lot and is kind of a big public figure. There’s some poker players that are examples of this, who also self-diagnose as having autism or Asperger’s, and they kind of play characters, in essence. “If I can adopt a persona or a character…” It’s almost like how a large language model would do it, right? It’s like the LLM can adopt the persona of an Irishman who’s had too many pints of Guinness or something like that. And that’s almost easier for it than having its own independent persona. And I feel like sometimes with people who are on the spectrum, they do that, and then forget that it’s a persona and come to own that. And it becomes kind of a schtick that they lose sight of, or… I don’t know. I mean, I talked to Sam more than a lot of people, but you’d have to ask his parents or something. Rob Wiblin: So the autism thing, that could be one factor. But I think there’s this other archetype, which is someone who is really smart, and maybe in some senses they have good judgement to start with. But then they become very successful. They have a bunch of wins unexpectedly. Maybe it’s a combination of skill and luck. They get on Twitter, and they’re posting on Twitter all the time, and it just seems like their judgement degrades, and they start taking kind of wild swings at things. Nate Silver: It’s very hard, because I’ve been a couple of times where I’m on the… Not to Sam’s degree, but enough that there were times after the 2012 election where you would go out in New York or something, and probably more than half the time I go get a coffee or something, then you get recognised in public. I don’t particularly like that, for what it’s worth, but I’m aware of, as your fame ebbs and flows, how much more sycophantic people become, how many more opportunities you have, how being kind of a celebrity has a weird pull — where you kind of become detached, or you can become detached from the real person, because in some sense the idea of celebrity is an object that exists outside of you. That’s what celebrity is: this idea of Sam Bankman-Fried or Nate Silver or… I don’t know who else a celebrity is. I don’t know if Eliezer Yudkowski is a celebrity, but he’s kind of memeable. And that’s a very weird thing — maybe especially if you’re having some neurodivergence or neurodiversity, I suppose. And yeah, it’s not surprising. And also, some of the things too that we talk about in the book, in the context mostly of poker, but also traders on Wall Street: you also have chemical reactions when you’re on a winning streak, right? You have powerful endorphin releases. You probably have more testosterone and things like that. So the fact that people seem to be unable to avoid these parables that seem so predictable — about being on a winning streak and hubris and getting in over their skis — it’s quite literally almost chemical. Rob Wiblin: I find this quite disturbing, because I think you can just observe that many of the most powerful people, the most influential people in society at any point in time, are kind of at the peak of their careers after a long streak of unlikely successes that have brought them to where they are now. And they’re kind of on tilt in this way, that they just feel kind of godlike at that moment, or they feel that things can’t go wrong, that their judgement is so good. And they start taking bigger and bigger risks, and they end up affecting all the people around them with this kind of distorted judgement. Nate Silver: Yeah. People think of tilt in poker as — tilt is playing emotionally in a way that is costing you expected value, basically. People think of tilt as what happens, which it often does, when you’re on a losing streak or take a bad beat. And therefore you can have different reactions: you can either try to chase your losses, or just as often, people become way too tentative and risk averse, and you have to be aggressive in most forms of poker. But winners’ tilt can be just as bad. Where you can tell yourself in advance, if there are 10,000 entrants and the World Series of Poker, if you’re the best player in the world, your chances of winning the World Series of Poker are probably one in 1,000. So you have a 10x return, which is actually very good — but there’s still, overwhelmingly, luck. And if you have a couple of bets in a row that pay off, especially if they’re contrarian bets — one of Elon Musk’s issues, or like Peter Thiel, for example — if you make a couple of contrarian bets and they pay off, it’s really satisfying to get a financial purse. It’s really satisfying to prove people wrong. And if you get both at once, I mean, that’s like — Rob Wiblin: That’s a hell of a drug. Nate Silver: It’s like some drug cocktail, seriously, that has profound effects on you. Then you do that like twice, have a couple of bets that pay off, it’s very hard to outlive that in some ways. And if it goes wrong, then you’re kind of chasing the high that you had before. So it’s hard to get off that rollercoaster, I think. Rob Wiblin: Yeah, I don’t really know how to fix it, but I think it’s a deep, systemic issue. I guess it probably has been the case in every society at all points in time that it’s created disasters. Nate Silver: Yeah. But I think we also have focal points in today’s society where the fact that you have this instant feedback, where things are quantified on social media. Or if you have a book out, you can refresh the Amazon page and see what your ranking is, or see what that new review is: a two-star review from that person in Des Moines, Iowa. Screw them. And things like that. So the kind of instant, gamified feedback — especially through Twitter in particular, which I think seems to drive certain people, maybe including the founder or the owner of Twitter, slightly crazy — I think that is an accelerant. Rob Wiblin: Let’s talk about expected value for a minute. I feel like doing expected value calculations or thinking about expected value — assigning probabilities, assigning benefits and costs to things, and then weighing them all up — I think that’s maybe the most distinctive trait of this River tendency that you talk about in the book. It’s almost the thing that defines it primarily. I think it’s also the thing that is perhaps most distinctive about Sam Bankman-Fried. This has shown up in a lot of books, where he just seemed to think in expected value terms all the time. I guess Michael Lewis really emphasises this. Even Michael Lewis, as a financial journalist, found this to be a little bit extreme. What should we take from that? Should expected value take a big status hit because Sam Bankman-Fried was so into it? Nate Silver: Again, I think he was actually a pretty bad EV maximiser. One thing you learn from poker and the game theory of poker is that you’re very often indifferent between one or more solutions. And then often, in equilibrium, the EV differences are quite small. People agonise over, “Which restaurant should I go to?” or, “How do I plan my flight itinerary?” And probably, in situations where things are competitive, there’s not a whole heck of a lot of difference. So I think spending less time on those problems and picking at random can be helpful sometimes. I think people sometimes have a bias against things that can be quantified versus things that can’t, but they still value quite a bit. That was an issue in COVID, for example: we saw on the news tickers, BBC or CNN, the number of COVID cases or deaths in a certain area going up. There’s no ticker for disutility from being unhappy because of lack of social engagement or future expected value loss from loss of educational years or things like that. I think avoiding those biases and defaulting more toward “common sense”; I think many things that we think of as being irrational are probably rational on some slightly tweaked version of rationality. And often, I think, as you go through life, you’re actually… It seems pretty common that this behaviour that we thought was irrational, actually, now that we have better data, it was really quite smart. So I think kind of trusting markets a little bit more, and realising that cases where people are behaving persistently irrationally are not rare, but maybe rarer than you might assume when you’re greener to the ears — just getting into being an EV maximiser, I suppose. So respecting revealed preference and market wisdom a little bit more. Rob Wiblin: I was trying to think about what takeaway do I make from SBF being so obsessed with EV and it seeming to lead him astray? One thing you could say is that expected value is the wrong way, philosophically, theoretically, to weigh up costs and benefits and probabilities. But I don’t think I want to go down that road. You could say it’s good in theory, but it’s bad in practice — because humans are so bad at doing it, and so maybe we just shouldn’t use it at all. That seems too extreme. So I think instead you should say that that should be one factor in your decision making, but it needs to be paired with common-sense checks and heuristics, and what do other people think. You don’t just take your expected value estimates literally, which I guess people have been saying kind of always. An alternative angle would be that it’s actually none of those things. The issue with Sam Bankman-Fried was that he had the wrong values: that in the expected value calculation should have been, “…and then I’m stealing the money, basically.” And that should be massively penalised, because you should disvalue it. And he failed to do that because his values were bad, or at least they’re not our values, so of course that’s going to lead him to do things that we would evaluate negatively. Nate Silver: I think there’s also a real question about whether he had an urge to be self-destructive. Rob Wiblin: Talk about that a bit more, because I haven’t heard that so much. Nate Silver: Tara Mac Aulay, who was his original cofounder at Alameda, and was one of many people who quit Alameda before Caroline Ellison took over — before FTX — because she thought Sam was behaving… You can read it in the book; I don’t want to misquote her. We’re in the UK where they have stricter libel laws. But she thought SBF was a liability in various ways. And she said that Sam would confide in her that prison didn’t seem so bad, because he has anhedonia, self-diagnosed, which means a lack of ability to process or feel pleasure. So if you don’t enjoy whatever people enjoy — going hiking with your friends or drinking and eating or having sex, or whatever other people do to enjoy themselves, or playing sports or going to a sports match and watching sports — if you don’t get joy from any of those things, then… Rob Wiblin: Maybe you just feel very detached from your decisions. Nate Silver: You feel very detached from your decisions, and there’s not that much consequence to… I’m not sure how you go, but if you’re promising that, in heaven, you’re going to have all this access to all these things — there’ll be all these virgins in heaven, kind of — and you’re not experiencing the pleasures of the flesh on Earth or something… I think he’s an unusual person. I’d put it like that. And even among gamblers, gamblers kind of celebrate the term “degen,” short for “degenerate gambler.” That’s kind of taken as a term of affection, right? “Oh, I won the tournament. Don’t worry. I didn’t degen it up at the club.” Or “I degened it up at the blackjack tables, gave some of it back.” And it seems like a badge of courage and kind of honour in gamblingness. But I mean, the line between self-destruction and rational risk taking is often, I think, quite thin. Rob Wiblin: Something that you address very directly in the book is how similar Sam Bankman-Fried and Sam Altman, the CEO of OpenAI, are. Obviously one of them was a CEO, one of them is a CEO. They’re both called Sam. What are some other similarities or parallels that people could draw between them? Nate Silver: I’d say Sam A is familiar with the expected value framework. When I talked to Sam A was in mid-2022 — which turned out to be a good time to talk to him, because this is when they are aware of how much progress that GPT-3.5 and potentially GPT-4 are making. So sitting on this very cool secret almost, but also, he’s not quite as much in the spotlight — so he’s a little bit more unguarded. And the explicit invocation of expected value thinking, where he’ll say, “Yeah, actually, there is a chance that AI could go very badly, and if it’s misaligned, it could even severely harm, destroy, catastrophic effects for civilisation. But I think it’s going to end global poverty and extend human lifespans and push us to the next frontier, so therefore it’s worth the risk.” Now, I don’t think he’d flip the coin for 50/50. One source said he might flip the coin for 90/10. And he has said things that prove he’s not a strict utilitarian. One thing he said, I think in an interview in The New Yorker, is, “Yeah, I value my friends and family way more than random strangers.” I don’t know if he was aware of how much of an EA trope this was, but he’s like, “Yeah, I would kill 10 strangers to defend my family” or something. Which actually made me more trusting of Sam A, I think. But the fact is that it’s maybe the anonymous collective that kind of really runs the AI labs anyway. It’s the kind of silent votes of people who are the engineers who build these models and could leave. We saw what was maybe somewhat incorrectly billed as the “EA coup” against Sam Altman. I think the reporting on that’s more banal, actually: that it actually was what it sounded like, which sounded very boring, but like a lack of transparency and internal struggles and things like that. Although I don’t know. I was going to say you’re going to get the game theory equilibrium in AI, which I think is true — although, because you have a finite number of players, then it’s a little bit less obvious. I think probably individual actions matter more from someone like Sam Altman than in a more open competition. Rob Wiblin: So it sounded like one distinction that you were drawing is just that you think Sam A is making more high-EV bets; and that SBF just had bad judgement, and he made bad bets, he did things that were negative EV. Whereas Sam Altman has good judgement, and so he’s taking more positive EV bets, and that’s the difference. Of course, that’s not really a deep structural difference as much. I suppose it does really matter, but the situation would otherwise potentially be analogous in the willingness to take risks with other people’s wellbeing. But actually, when I thought about this, I realised I think there are some quite important structural differences. One is that SBF was deceiving people into thinking that their money was safe, and that he wasn’t gambling with their money, with their assets, and with their wellbeing. He was pretending that he wasn’t. Whereas Sam Altman, to his credit, has been completely honest that AI could kill us all, and that it’s extremely risky. And that what OpenAI is doing should maybe be of concern to other people, because it’s going to affect them a lot, and it could be very negative. That seems very different, because that means that other people could intervene. People could try to regulate OpenAI if they disagreed with the risks that it was taking. Whereas it was kind of impossible to intervene if you don’t realise that something like this is going on. Did you agree that’s an important distinction? Nate Silver: Yeah. You know, people especially who are not familiar with AI, or maybe on the left in the US, are like, “Well, everything he’s saying is kind of to his advantage.” I think that slightly misunderstands how this tribe of nerds behaves. I think for the most part, they actually are pretty truth seeking. Rob Wiblin: Yeah, I just think he said that because he thought it was true. And I don’t know, maybe he’s just a little bit looser of a person, and so he was just willing to say the thing that he believed, even if it wasn’t in his selfish interest. Nate Silver: Yeah, he says things on Twitter that are a little too candid for a CEO of a company as large as OpenAI. But that makes me trust him more in some ways, right? Rob Wiblin: Yeah. I guess the other important disanalogy that I thought of was: SBF’s depositors, whose money he was gambling with secretly, didn’t stand to gain if he made good investments. He gained all of the upside, and they took all of the risk if it went below zero. On the other hand, with Sam Altman, it’s not really like that at all. It’s true that Sam Altman personally stands to gain enormously and disproportionately if OpenAI is successful: both financially and in terms of reputation, he would be one of the great figures of history if things go well. On the other hand, if AI goes well, everyone else is going to benefit enormously. We all have enormous equity in this technology going well, so the upside and downside risk is more balanced. We all have skin in the game to a similar degree. Nate Silver: I think one of the things we haven’t mentioned yet about SBF was that he assigned no diminishing marginal returns to additional wealth, because he claimed he was going to give it all away and/or because he had such crazy ambitions — like I guess buying the presidency — that he cared about. The first $100 billion is equally good to the second $100 billion, right? Rob Wiblin: Which is crazy on any view. But setting that aside… Nate Silver: Yeah. Whereas objectively, Sam Altman is going to have all his material needs fulfilled for the rest of his life, even if OpenAI gets shut down by the government. Well, I don’t know what kind of liability he might have. Maybe I take that back. But once you get to a level where you have high seven figures to eight figures in the bank, and you’re readily employable and you have annuitised income streams, then yeah, you 10x your money and you might increase your personal utility by 1.05x or something. And maybe you increase by 100x and you can buy a share in a basketball team or things. One thing about working on this book is I spent more time with really, really rich people than I had ever before. Their lives aren’t really that much better. They’re a little bit better. Some cooler things — they have really cool homes — but they also have more burdens and more people they have to pay. Rob Wiblin: More people hassling them. Nate Silver: More people hassling them. It’s not actually that much of an increase in utility, necessarily. I think one thing that people from “outside the River,” as I call it, misunderstand is that people are actually not that monetarily driven. They’re driven a lot by competition, by wanting to prove people wrong — and the finances are a way to keep score, and/or that making lots of money happens to be 0.9 correlated with kind of winning this competition, even though it’s not quite the same thing. Rob Wiblin: So you talk about various virtues and vices that effective altruism might have, or various weaknesses it might have. I’m curious to know, all things considered, in your view are there other groups doing philanthropy better than Open Phil, or competing with Open Phil in terms of their impact per dollar? Or are there other groups that give better advice, or similarly good advice to 80,000 Hours on how you can do good with your life or your career? Nate Silver: Not that I’m… Yeah, I think it’s probably a pretty big gap. Although, again, I think the Gates Foundation. I went to some dinner that Bill Gates hosted. It was not on the record so I can’t share specific comments, but enough to say that they are quite rigorous about what they do. And he is very knowledgeable about the different programmes and how effective they are, relatively speaking. But yeah, I think this is a pretty huge win. Even if you assume that maybe there’s some mean reversion versus what the true value of a malaria net is versus the estimated value, and maybe you’re making slightly favourable assumptions, the delta between malaria nets in Africa and giving to the Harvard endowment has to be like 100,000x or something. In fact, I think actually even the Harvard endowment is probably disutilitarian: I think it probably actually could be bad for society. So yeah, I think there are a lot of low-hanging fruit and easy wins. But one property of being in the process of being a gambler or doing adjacent things for a long time is that I think people don’t quite realise the curve where you only have so much low-hanging fruit, and then it gets harder and harder, and/or there are more competitors in the market, and then you kind of quickly get to the surface where we’re going for smaller and smaller advantages — and then small errors in your models might mean that you’re into negative expected value territory. Rob Wiblin: Do you think that might be going on with Open Phil to some extent? Nate Silver: I think on the charity side of things, they’re probably still doing a lot of good. I don’t know, maybe at some point you need… I don’t know, I don’t do corporate or nonprofit governance stuff. Maybe you actually want things that are a bit more siloed. Maybe you want x-risk people and you want philanthropy people and you don’t necessarily want them in the same meta structure. You want two different organisations. Maybe. I don’t know what other issues there are, if they’re red-teaming for future types of risks and stuff like that. But yeah, maybe you need more separation. For one thing, if this movement broadly grows, then I think again of the Manifold conference in Berkeley, California, which they’ve had now a couple of times, where you’re getting people from different parts of this world. That’s at a critical mass of size, where right now it’s really cool and really fun and everyone knows one another — but if it grows, it’s going to become larger, and I don’t know if the existing structures can hold that. I think maybe having different teams that are competing, in some sense, instead of having one superstructure, that seems probably directionally right to me. But I don’t know. Rob Wiblin: Yeah, you end up with a lot of problems as you try to scale a group. Like the trust networks start to break down, because people don’t really know who’s trustworthy because the group’s too big. You also have competing demands, and competing kinds of people who want different things from the group and can’t all be satisfied at the same time. Nate Silver: Yeah. And pluralism is a very robust heuristic, generally speaking. Rob Wiblin: You’ve mentioned you’ve listened to the show a bit, so you might have some sense of the sorts of things that I believe, and the sorts of things 80,000 Hours writes on its website. Do you feel like you have any notable disagreements with the things that we say? Nate Silver: Directionally, probably not. Sometimes I think people underestimate how back-of-the-envelope some of this stuff is. It’s kind of easy to say, going in, “This is just back-of-the-envelope. It’s just a guesstimate.” But then you kind of become overly committed to the model later on, or things are kind of lost in translation later on, potentially. The example I give in the book is trying to estimate animal welfare, for example. Was it Will MacAskill who was like, it’s based on the number of neurons, so actually an elephant is worth more than a person? I think the thing I’m most sympathetic to about EA and rationalism more broadly is that society has to make decisions, actually, right? I give the example in the book of a dog named Dakota that runs into the subway in New York during what’s going to be rush hour in an hour or so. And the New York City Transit network has to decide whether we shut down the entire F train — one of the busiest commuter trains, from Manhattan to Brooklyn — to protect the life of this puppy named Dakota. And they decide to stop the train, actually. So in that case, you actually have to calculate the utility of what is a dog’s life worth versus X amount of delay for Y amount of commuters? And by the way, what if one of them is an emergency worker and can’t get to the hospital in Brooklyn on time and things like that? Maybe an actual human being dies. Maybe an area of undercoverage, or a neglected area, is utilitarian thinking for medium-scale governmental problems. I think this is actually probably more needed. Government heuristics are often very politicised, very cumbersome, with a reluctance to employ cost-benefit analysis. So for things like the COVID vaccines distribution schemes, you had very different schema in the UK and the US. In the UK they were like, “We looked at the chart, and actually there’s an exponential curve where the older you are, the more likely you are to die of COVID, so let’s start with the 95-year-olds and then go down by five years every couple of weeks.” Whereas in the US, we had this whole complicated framework trying to balance equity and trying to balance utility and different things, and it didn’t really make any sense. I think governments should be more utilitarian — with a constraint of, obviously I’m a big liberal: I believe in protecting people’s rights in that classical liberal sense. But within those constraints, there’s so much more government money spent than there is charitable money spent. I think it’s a really neglected cause area, basically. Rob Wiblin: An unusual critique that you make of EA in the book is that it’s not compatible with game theory. It’s not evolutionarily stable. You talk about how, in the long run, people who just use their resources and their time to benefit strangers or people who they’re not in a mutually beneficial relationship with just tend to get outcompeted and disappear. It’s an interesting worry. How do you think that might cash out, and what you would do differently? I guess the natural thing you might think is, in that case, the EA community needs to have clearer boundaries, and needs to benefit people inside those boundaries more and do less to help total randoms or strangers. But I think you can imagine the criticisms that would come from that kind of mentality. Nate Silver: I think there’s two parts to this. One is a more abstract game theory critique, where if you’re not defending your turf, then you tend to get trampled. If you’re having a football match, and one team is super competitive and trains really hard, and the other team says, “Well, we think football is actually destructive. We’re just going to run around and kick the ball around and have fun and be inclusive.” If you’re judging that competition, the competitive team wins every time, right? And maybe there’s a slight aspect of kind of social Darwinism in the book, where I think, in a kind of capitalist system, unless you are able to constrain competition, then the more competitive side usually wins out. But it’s also kind of like meeting people where they are, in the middle, in terms of human nature a little bit more. One of the things I don’t like about the Peter Singer book is that it says that, let’s say you give a little bit and then you feel good about yourself. So now you go and get a nice dinner and have a nice bottle of French wine or something — medium range, I don’t know what it is exactly. Well, you should feel guilty about that too. I think meeting people halfway between common sense and some kind of Singerian utilitarianism would actually go quite a long way. But yeah, I worry about people who are not willing to… If you’re not willing to counterpunch, and you’re not willing to defend your turf and be somewhat partial toward your group or your tribe, then I do worry that you’ll get hoodwinked or trampled over. In part because, in some ways, game theory is the most important concept in the book. We talk a lot about expected value, but game theory is what happens in a world of 8 billion people where everybody’s trying to maximise their EV, more or less. I think that in a world of 8 billion people, disincentives you create or opportunities you have to be exploited will inevitably result in your being exploited, because you have so many opportunities for people to do it. If you have, for example, a financial system where you have moral hazard — where if you make these risky bets, then the government will bail you out, therefore you have more incentive to make risky bets — within some period of years or decades, some or a number of firms in the system will figure that out, and build models, and price that in, and begin to make those bets, almost inevitably. So if you don’t have robust enough fraud-detection characteristics, if you’re prone to getting hijacked by actors who take your good-faith movement and your trust in strangers and use it to politicise their agenda or to build clout for themselves, I think you will actually be exploited sooner or later, at the end of the day. You see this actually in some kind of typical liberal institutions, like the academy, for example. The academy might say, “We want to provide impartial expertise, but we also have values in the US. We’re probably progressive Democrats. We want to be more inclusive and we don’t want to help Trump win, and we want to be anti-authoritarian.” What happens is therefore your expertise gets hijacked by the politicised people. I’m not quite explaining this well enough. Rob Wiblin: You’re saying that over time, the people who are more willing to cut corners on intellectual honesty become more prominent? Nate Silver: That’s the equilibrium, right? If you tolerate intellectual dishonesty, then the equilibrium is that intellectually dishonest people will gain stature within your movement. Rob Wiblin: On this question of how much: you mentioned the Singer book, where he was saying that any money that you spend on yourself, you ought to feel bad. The book both mentions this worry that effective altruism as a community might be too austere and might be too demanding, and say that you should feel bad if you have children, if you do anything for yourself. On the other hand, you also mention, with an eyebrow raise, this very lavish donor dinner that Sam Bankman-Fried organised somewhere. I think it’s been a very difficult balance to strike between, on the one hand, saying, “You shouldn’t spend too much money on yourself or on your projects, because that’s kind of self-indulgent. You might trick yourself that this is a good idea when it’s not.” On the other hand, people can criticise you for saying you’re not actually willing to spend money in ways that are useful, or you’re insisting on too much austerity — more than people actually are willing to live through. Did you have a view on, are we striking the balance wrong in both directions at different times? Nate Silver: I mean, the idea of giving 10% I don’t think is the worst idea, necessarily. I think that meets people in the middle. Yeah, it’s a little tricky. I do think the critique that EA flatters the sensibility of people who are not necessarily rich, but have skills that tend to lead to a lot of financial wealth, I think that’s actually OK. I think people are too concerned with, “This person donated billions of dollars to charity, but they’re the wrong person.” Rob Wiblin: “I still don’t like them.” Nate Silver: Yeah. I like it when multibillionaires give billions of dollars to charity, for example. The other kind of critique I have of strict utilitarianism is that it’s too complicated to implement in practice. You probably need some simplifying heuristics that work pretty well. I’m not sure what heuristics I’d apply. It’s also maybe lacking a little bit of “political common sense.” I suppose that Eleven Madison Park is the most famously opulent vegan restaurant in the world, right? And if you had called me, I’m kind of a New York foodie, and said, “Nate, pick a restaurant that will be equally good, a little bit cheaper, and that you’d never have like a negative headline from, because it’s less well known”… Yeah, I don’t know. I almost wonder whether it was like a flex from SBF, who hosted the dinner there. But maybe it showed a lack of political awareness, I suppose. Rob Wiblin: In the book, a recurring thread is to say that you’re a bit unwilling to fully embrace effective altruism because you have these reservations about utilitarianism. I think many of them are the classic reservations you might have — about lack of side constraints, not valuing a sufficiently wide range of different things, only caring about wellbeing, not caring about autonomy, that kind of thing. I feel like that’s a little bit of a shame, because when we were trying to figure out, “What should effective altruism be? How should it be defined? What should be in and what should be out?” in the very early days, I kind of thought of it as an attempt to take utilitarianism and get rid of the bad parts. There’s a whole lot of commitments that it has that are totally unnecessary. They’re unpleasant, but they’re not load-bearing in any way. Let’s take the good part of it — which is that it’s nice to help people when the benefit to other people is very big, the cost to you is very small — strip away all of the unpleasant parts, and then go with that. And I guess hopefully add in also that you can’t steal, you can’t do all these terrible things. Do you think that’s a good aspiration, basically? To have a philosophy of life that puts benefiting people front and centre, while stripping away the parts of utilitarianism that almost nobody is on board with? Nate Silver: In principle, I think sure, yeah. I think one of the problems the movement has is it’s kind of at a little bit of an awkward teenage phase. How many years old is it? Depending on how you define it? Rob Wiblin: At most 15. Nate Silver: OK, so more like kind of a tween or a teenager. One point the book makes in trying to like trace the intellectual history of EA/rationalism — which I think are actually in some ways less alike than people assume — is the fact that this is kind of born out of these small networks, primarily at Oxford and/or in the California Bay Area and/or on the internet. And it’s all people who know each other, but people who bring different things to the table: prediction markets, animal welfare, some are kind of tech accelerationists, some are maybe a little bit more socialist potentially, some are very concerned about the public perception of EA, some want to discuss genetics and really un-PC things. And as the community gets larger, then maybe some of the contradictions become a little bit more apparent. I think in the current form of EA, there is more Singerian utilitarianism than I might want to be a buyer for. But I don’t know how you change that. I mean, this is where I kind of think that maybe the movement needs to have… Like when Standard Oil broke up or something. Maybe you need a US branch and an Oxford branch, and something that’s a little bit more different and pluralistic. Rob Wiblin: I see. So you could break into different brandings that are more specific, more niche, and don’t have to overlap so much and feel like they’re sharing a common identity. And then you could have them disagree and debate amongst themselves. Nate Silver: Yeah, and you could have one big conference every year that rotates locations or something. I think that might be healthier. Rob Wiblin: You mentioned that you like rule utilitarianism. Should EA try to become rule utilitarianism? So this is something where you say, rather than look at every single act that you’re doing and try to maximise utility from that, you try to figure out what rules of behaviour would lead to the best consequences in general, and then you try to follow those general principles of behaviour. It seems like a pretty good thing in practice. Nate Silver: Yeah. So as a game theory guy, I believe that’s solving for the equilibrium, right? If everyone behaves this way, then what’s the equilibrium that result looks like? I think that’s like an order of magnitude more robust, also harder to calculate. But I am almost ready to endorse rule utilitarianism, I suppose. Rob Wiblin: Yeah, I guess rule utilitarianism struggles a bit philosophically because you can always say, at what level of generality should you be thinking about the actions? And why that level of generalisation? Why not go down to the very specific circumstance that you’re in? — and then you’re just back to act utilitarianism. And I’m not sure how to fix that philosophical question. But in terms of practice, it just seems vastly better. It’s a much better thing to actually advocate for, because it will lead to better consequences, because people will do more useful things with that idea in mind. Nate Silver: So in going around and talking about this book with a lot of people, it’s not that hard to explain expected value, right? Say you’re playing poker: there are 52 cards that can be dealt, and here’s your average outcome. And then you have to be more abstract about that when you’re dealing with situations where there’s incomplete information, whatever else. But the notion of the game theory equilibrium I think is actually the more important concept in the book, because I think people are actually rational within their domains when they have skin in the game and have a repeated problem. I think people are good at getting what they want, the things they prioritise within the constraints that they have, and not treating the rest of the world like non-player characters where they don’t have any agency. I mean, I see this mistake made so often. Rob Wiblin: What’s an example? Nate Silver: In politics you see it all the time, right? After Obama’s rise in 2008, 2012, so I guess 12 years ago now, you had Democrats saying, “Look at these trends! We have all the people of colour and we have all the younger voters. Extrapolate that out, we’re going to have Democratic supermajorities for the next 20 years.” Well, it doesn’t account for the fact that you have a competing party, and that you’re taking for granted what if your share of the white college vote goes down? All these union workers that used to vote Democratic but are now culturally more conservative: what if a Donald Trump comes along and not a Mitt Romney, and it’s much more appealing to them, potentially? The fact that, in American elections, you have elections that are so close to 50/50 is actually a big proof of game theory optimum solutions in some ways: that the parties are pretty efficient at dividing up the electoral landscape, but not treating the other side as though it’s not capable of making intelligent adaptations. Rob Wiblin: Yeah. One benefit of having grown a little bit older is that I remember following elections when I was a teenager, and whenever there was a big victory for one side, they would say, “The other side, they’re going to be out of power for a generation.” And I’d be like, “Wow, huge news. Incredible!” And now I’ve heard that dozens of times in my life, and I don’t think it’s been true a single time. Maybe it was true once. Nate Silver: And even more, the incumbent effects in politics are much more minor now, or maybe even reverse, right? You’d actually rather be the opposition party. People want more balance and want to switch back and forth a little bit more. So you’re always almost like giving money back, 30 cents back on the dollar whenever you win an election, by making yourself more likely to lose the next time. Rob Wiblin: You were talking about the value of game theory. Did you have anything you wanted to add on that? Nate Silver: This is like Tyler Cowen, if you read Marginal Revolution: he’s always saying, “Solve for the equilibrium.” I think that’s something I found helpful: thinking more in game theory terms about what’s the behaviour that results. Even if I’m writing my newsletter or something: What’s the behaviour that results if everyone behaves this way? In some ways it actually makes you behave with a longer-term focus, and maybe even more ethically. One thing that you learn from game theory is that if you try to exploit somebody, then you can be exploited in return. If we’re playing rock, paper, scissors — do you have that game in the UK? Rob Wiblin: Yeah. Rochambeau. Nate Silver: Rochambeau. OK, whatever. And you, Robert, are always throwing rock. And therefore I’m like, “I’m just gonna play paper every time.” Well, all you have to do is then one-up me by then playing scissors, right? So it becomes a circular thing, where the GTO — “game theory optimal” — equilibrium is to randomise one-third, one-third, one-third. And having played enough poker where I’ve been in situations where you think you have the edge, and you think you’re able to not cheat in an actual cheating, using-cheating-devices sense, but you’re trying to like “exploit” somebody is the game theory term. And before you know it, you’re taking a shortcut and you’re the one who’s paying the price for that. Rob Wiblin: One of my favourite tweets of yours ever is, “When they go low, we go high 80% of the time and knee them in the balls 20% of the time.” Nate Silver: It’s important to have mixed strategies, right? Rob Wiblin: In the book you draw a distinction between EA culture and rationalist culture. And I think at one point you say EAs are very reserved and well spoken, very PR concerned. One friend said to me, “I wish that were true. I don’t know what EAs this guy was talking to.” But I guess you are drawing a distinction that is somewhat real between EA culture and rationalist culture: rationalists, I think, are much less concerned about appearances, about comms to the general public. They’re kind of willing to say whatever’s on their mind, no matter whether people find it offensive or not. That’s true to a greater extent. Do you think EA culture should be more freewheeling, and more willing to just say stuff that pisses people off and makes enemies, even if it’s not maybe on a central topic? It seems sometimes in the book that you think: maybe! Nate Silver: Directionally speaking, yes. I think to say things that are unpopular is actually often an act of altruism. And let’s assume it’s not dangerous. I don’t know what counts as dangerous or whatnot, but to express an unpopular idea. Or maybe it’s actually popular, but there is a cascade where people are unwilling to say this thing that actually is quite popular. I find it admirable when people are willing to stick their necks out and say something which other people aren’t. Rob Wiblin: I think the reason that EA culture usually leans against that, definitely not always, is just the desire to focus on what are the most pressing problems. We say the stuff that really matters is AI, regulation of emerging technologies, poverty, treatment of factory farmed animals. And these other things that are very controversial and might annoy people in public, I think EAs would be more likely to say, “Those are kind of distractions that’s going to cost us credibility. What are we really gaining from that if it’s not a controversial belief about a core, super pressing problem?” Are you sympathetic to that? Nate Silver: This is why I’m now more convinced to divide EA into the orange, blue, yellow, green, and purple teams. Maybe the purple team is very concerned about maximising philanthropy and also very PR concerned. The red team is a little bit more rationalist influenced and takes up free speech as a core cause and things like that. I think it’s hard to have a movement that actually has these six or seven intellectual influences that get smushed together, because of all people getting coffee together or growing up on the internet (in a more freewheeling era of the internet) 10 or 15 years ago. I think there are similarities, but to have this all under one umbrella is beginning to stretch it a little bit. Rob Wiblin: Yeah, I think that was a view that some people had 15 years ago, maybe: that this is too big a tent, this is too much to try to fit into one term of “effective altruism.” Maybe I do wish that they had been divided up into more different camps. That might have been more robust, and would have been less confusing to the public as well. Because as it is, so many things are getting crammed into these labels of effective altruism or rationality that it can be super confusing externally, because you’re like, “Are these the poverty people or are these the AI people? These are so different.” Nate Silver: Yeah. I think in general, smaller and more differentiated is better. I don’t know if it’s a kind of long-term equilibrium, but you see actually, over the long run, more countries in the world being created, and not fewer, for example. And there was going to be originally more stuff on COVID in the book, but no one wants to talk about COVID all the time, four years later, but in COVID all the big multiethnic democracies — especially the US, the UK, India, and Brazil — all really struggled with COVID. Whereas the Swedens or the New Zealands or the Israels or Taiwan, they were able to be more fleet and had higher social trust. That seemed to work quite a bit better. So maybe we’re in a universe where medium sized is bad. Either be really big or be really small. Rob Wiblin: I see. Something like liberalism or be much more niche. Let’s say that you’re given $10 billion and a research team as well to start a new charitable foundation to try to do as much good as possible. What are we doing, Nate? Nate Silver: I’d buy an NBA team. No, I don’t know. Yeah, maybe I’d start from scratch, and think about… Maybe you just need a fresh start to say, what are the truly neglected areas now? I mean, I’m sure 10 years ago that AI was very neglected. I wonder if those heuristics need to be updated. I wonder if climate is this example of this issue that EA sees as overindexed. There was a lot of concern about climate, but what if you built climate organisations that are more “rational”? Is that an underexploited niche? Because they get so political and they get so embedded in the progressive politics. Could you build a climate organisation that’s somehow immune from, shall we say, the dangers of wokeness and things like that, and co-option from people who want to adopt that for non-climate objectives? That might be interesting, for example. Rob Wiblin: There are people trying. We’ll link to some episodes of interviews of people trying to do that. One thing that I remember from that interview is saying that renewable energy is actually funded so much, and it’s so popular, that if renewable energy works, then we’re in the clear on climate change. So they focus a lot on like what if renewables is a bit of a bust? What if it’s very disappointing? What stuff do we need in that case? So it’s very River-style thinking. Are there any other problem areas that you think might be underrated, or that you would ask your research team to look into? Nate Silver: The one I brought up before is efficiency of government spending. I don’t know how you persuade governments to do this, but pursuing reforms — like I think lobbying for governments to increase the pay scale and to become less bureaucratic, maybe more Singaporean, I suppose actually probably has a quite high payoff. And things like government waste and corruption and inefficiency seem like they’re such stodgy concerns. But why does it cost 20 times more to build a subway station in New York than in Paris? That is a cause area that I think that EA or EA-adjacent people could start to look into a little bit more. Rob Wiblin: Yeah, I think on the civil service reform, I haven’t heard EA talk about that almost at all. Zvi Mowshowitz might have talked about it, but he might be the only one. On the subway stuff, actually Open Phil does fund Institute for Progress and a bunch of other progress studies organisations. So I think they have dipped their toe in the water there, even if it’s not their main focus. Nate Silver: This is a very niche cause, and this kind of ties into progress studies: I think economic history is a vastly underrated area. So to fund at some seven-figure annual salary an economic history institute, that’s kind of part of the progress studies institute, I think that would actually be quite beneficial, potentially, in terms of like basic research. Rob Wiblin: I remember Open Phil tried to fund a bunch of people to do macrohistory research. I think Holden [Karnofsky] was really into this back in like 2016 and 2017. I don’t know how much came out of that, but I guess it was an idea that they at least thought of. Any other stuff that stands out? Nate Silver: What was the episode a couple of weeks ago with electromagnetic pulses? Rob Wiblin: Oh yeah, EMPs. The nuclear war expert, [Annie Jacobsen]. Nate Silver: Things like long-term data storage and robustness. Maybe the amount of linkrot on the internet. I don’t know who owns archive.org, but you probably want a backup to that. That seems important. Rob Wiblin: Yeah, that’s one that I don’t think has been funded. I guess we had an interview years ago where we talked about, if you wanted to send a message to a civilisation that was going to re-arise in 100,000 years, how would you do it? It turns out to be extraordinarily difficult. I don’t know whether anyone has actually funded an attempt to figure out how you would do it in practice. Nate Silver: But it comes up in various scenarios. If you have a nuclear winter, how do you rebuild and things like that? I think that type of contingency planning might actually be pretty valuable. This is too much of a diversion for this point in conversation, but the loss of ground-truth data is something I worry about a little bit. Having a little bit of ground-truth data you can treat as absolutely true and reliable, I think creates hinge points that make models potentially much more robust, and is something people should think about. Rob Wiblin: How are we losing that? Nate Silver: So if you now go and do a Google News search, everything is a little bit… I don’t know quite how to put it. You can’t quite see the dates of the articles anymore, or how many articles actually meet your search query or things like that. It’s taking away things that are just basic reliability checks as far as data goes. It’s too algorithmised, right? Or I thought the Google Gemini stuff, where it’s like inserting command prompts that the user didn’t ask for, I think it’s actually insidious and coercive. I think that’s quite bad; that’s actually quite evil to like be presenting one thing and then putting a thumb on the scale in a way that is not what the user expected. Rob Wiblin: I thought that was a little bit more boring maybe than people made out. It’s like Google is kind of behind the ballgame on AI; they’re trying to rush out these releases. They’re like, “People complain that the images have too many white people, so we’re going to throw this into the prompt to try to patch that release.” Nate Silver: It is kind of funny, but I worry when data companies become less transparent, I guess. Rob Wiblin: So is your concern that, in the digital era in general, things are becoming more recursive? Or it’s harder to find like what the records objectively are because you don’t have access to them, because the company can just close it off or have a prompt and they don’t tell you what it is? Nate Silver: It’s the kind of recursiveness issue. There’s some metaphor that I’m failing to make here, but you know, the fact that we might have trouble building things, if you don’t keep the original design for things, then that can cause problems down the road — there are probably some things that we’re actually worse at as a society now. You know, red-teaming the concept of what if there were a nuclear winter, and we had to begin to rebuild society? I don’t know. I’m not a big space-exploration knowledge-haver, exactly. But it seems like if you buy the Toby Ord argument that civilisation has a 1-in-6 chance every decade of destroying itself, is it seen as too cliched for longtermists to think more about space exploration or things like that? I don’t know. That seems like a piece that maybe is seen as cringe, but maybe people should be thinking about more. Rob Wiblin: I guess because I think most of the risk comes from AI, that AI would chase after. So there’s that one. I mean, I guess it helps with biorisks potentially, if you have separate groups — although then you wonder, why not put them under the sea? It’s probably easier to be under the sea than to be on Mars. One thing that I wasn’t sure about: Currently, I think the effective-altruist-inspired giving is maybe half in the GiveWell/global health and wellbeing style — so it’s not all bed nets; there’s also funding reductions in air pollution in India and policy change, that kind of thing. But that sort of focus. And then maybe the other half is on everything else — including AI, risks from new pandemics: the whole other more speculative, more future-focused bundle. In the book, you say you maybe want to just demur on the question of whether it’s good to have more people in future. That philosophy doesn’t super interest you. I guess you find some of the idea that just adding more people without limit is massively better is not super appealing. But you also think the risk of doom from AI is like 2–20%. So I wasn’t sure: ultimately, would you fund AI more or less than what you think people do now? Nate Silver: You also get these arguments about whether doing research in AI research capabilities is actually doing good or is actually kind of accelerating, and nerds helping people into spending more investment in AI. So… probably more, but I might try to find ways to make the teams that are tackling these problems more diverse in different ways, and maybe have more of an outside view potentially. I worry a little bit about groupthink in those movements. Rob Wiblin: Do you think they’re too technical, or maybe just too focused on rationalist-style thinking? Perhaps not from enough different disciplines? Nate Silver: Yeah. I made the same critique during COVID, where people who are into public health, but you didn’t have enough economists actually consulting on COVID policy. I kind of wonder if you don’t have almost the reverse problem here. You imagine in all the sci-fi movies, Contact or Arrival, you always have the anthropologist who comes along to talk with the aliens and understands their culture, and the linguists and people like that. I wonder if we need a little bit more of that. I feel like there’s maybe a stigma around people who are theists a little bit in EA. And I actually kind of do think that it raises some questions when you think about consciousness and things like that. You probably want more theists in EA and studying AI and things like that, for instance. Rob Wiblin: I think that is a problem that is slightly solving itself as these worries go more mainstream and more people get involved. I guess it’s starting from a very low base of diversity of styles of thought. Nate Silver: Yeah. Rob Wiblin: I wasn’t going to bring up COVID, but let’s do it. I think there’s an interesting mistake that I think people in the River, and possibly you also, are making. When you’re used to economist-style reasoning, it’s very tempting to think that with COVID there were so many harms that came from all of the control measures that we implemented — you know, massive cost to mental health, massive cost of the economy, all these different things. So what we should have done is move along the marginal cost curve: we should have done somewhat fewer restrictions, and we should have accepted somewhat more spread, in order to not have people’s lives be negatively impacted so much. And I think that what that misses is that this is a very weird case — where basically, if R is above 1, then pretty soon everyone has COVID. If R is below 0, pretty soon nobody has COVID. There isn’t really a middle ground. And what you want to do, you have two options. One is, you could say, “We’re going to accept R is above 1. We’re going to accept everyone basically is going to get exposed to COVID before the vaccines arrive. Probably 1–2% of the population will die.” It could be a bit more, a bit less, depending on how quickly you allow it to spread and how overwhelmed the hospital system is. But you could accept that on one side. Or, “What we have to do is we have to keep R just below 1 — 0.9, to have a little bit of a buffer. And we’re going to try to do that in a way that’s least costly, that imposes the fewest costs possible.” There’s really only these two strategies that you can adopt. And I worry that people are imagining that there’s some middle ground that we could have struck that would have been a lot nicer. When in fact, the choice was just actually quite a brutal one. Nate Silver: Yeah, the middle-ground solutions were actually the worst, which is where the multiparty democracies wound up a lot of the time. In poker, you call it a raise-or-fold strategy: often, in the game theory equilibrium in poker, you either want to raise or fold and not call. So either you want to do like Sweden and be like, “We’re never going to get R below 1, so let’s do more things outdoors and protect old people. But a lot of people are going to die.” Or you do like New Zealand: “Fortunately, we’re an island country in the South Pacific, and there are no cases yet. Just shut down the border for two years.” And those extreme strategies are more effective than the muddling through, I think. Rob Wiblin: So you would say we suffered a tonne of costs socially. People’s wellbeing was much reduced. And at the same time, by the time the vaccines arrived, half of people had been exposed anyway — so we’d already borne half the costs, roughly. Maybe not quite as much, because we managed to spread out the curve. Nate Silver: I mean, the R=1 model becomes complicated when you have reinfection. You start to introduce more parameters when you have some duration of immunity from disease, although clearly severe outcomes are diminished. There are going to be long COVID people getting mad at me. Clearly the overall disease burden from COVID goes down, and probably people are infected with COVID all the time and don’t even realise it right now. There’s a long history of, it’s thought that some flus actually were maybe COVID-like conditions that are now just in the background and aren’t a particularly big deal. And the fact that discussion of “herd immunity” got so stigmatised was one of a lot of things that disturbed me about discussion about the pandemic. Rob Wiblin: An interesting point you make in the book is that you think it would be selfish of rich people — like you or me, or I guess just people living good lives in rich countries — to try to stop AI development for too long. Maybe smaller pauses or temporary delays could be reasonable, but trying to prevent AI progress from happening for decades you think will be selfish. Can you explain why? Nate Silver: Because I think you have countries now — like France, for example, or maybe the Nordic countries — where they have, especially in the Nordics, relatively high equality. And they are maybe taking an off-ramp, and are saying, “Let’s preserve our society as it is. We’ve achieved some high level of human wellbeing and flourishing and goodness.” Look, if every country had the standard of living of Norway or something, then maybe I’d say OK — because at some point you do have to worry about sustainability, right? Maybe then we have to totally focus on how we make human flourishing sustainable for the long run. Maybe that means finding ways to hedge our bets with having colonies in outer space. Maybe it means serious attempts to pursue nuclear disarmament, for example, to protect against some of the lower-risk existential threats. Maybe now it’s time to have asteroid defence systems or things like that. But we’re so far removed from that. Who are we to say now, as wealthy Westerners, “Let’s close the gate right now”? Because there are signs of secular stagnation, especially in the West, but by most measures, world GDP growth actually peaked in roughly the 1970s. The fertility crisis is something which is only beginning to be discussed. I framed it as a crisis, which it might not be, but the decline in fertility. We might never get to 10 billion people globally, and you might have a lot of asymmetries as far as an ageing population, thanks to advances in medical science, and not that many workers to support them. That can create all types of frictions in society. So there’s a pretty strong base case that AI is an ordinarily very important technology. I’m not quite sure how to put it. We have in the book this thing called the “Technological Richter Scale“: a magnitude 7 is something that happens once a decade, an 8 is once per century, and a 9 is once per millennium. I mean, it may be. I think you had Vitalik Buterin on the other day, and he’s talked about how there’s probably a little bit of a slowdown relative to expectations and how large language models are doing — but that might actually be pretty good, right? That we have time to derive mundane utility from them. I was in San Francisco the other week and took a Waymo for the first time, and it’s really cool. I think maybe people extrapolate from trends too much: “Driverless cars are not doing that well.” I would be shocked if they don’t make a huge impact on society. It was enough of a proof of concept, because you’re driving in real conditions like a rainy — I don’t know if it was raining; it was sunny, actually — sunny afternoon in San Francisco. And it’s very intuitively avoiding pedestrians who are jaywalking, and has a very smooth acceleration, and doing things that are quite smart — and I think obviously quite a bit better than human drivers. Apparently, in their own testing, which you can discount, they’re 5x or something safer. So yeah, I think it’s too early to say, let’s kind of lift the drawbridge up and stop technological progress — right when people in the West have it really good. Rob Wiblin: I liked this point, because it kind of flips the script, where I think many people would say: that people in Silicon Valley are pursuing AI because they stand to benefit a lot personally. I think in actual fact, over time, the benefits from AI would end up disseminating to basically everyone. You could listen to the Carl Shulman interview if you are not convinced by that; I think he explains why it would be very surprising if they didn’t. If things go well, at least or reasonably well, then almost everyone will end up benefiting a lot. And basically, just the worse off you are now, the more you stand to gain. If your life is already very good, then there’s only so much better it can be. But if you’re really struggling in poverty, then you just have far more upside potential. Nate Silver: I think it’s probably mostly true, although it can have counterintuitive effects. In poker, there are game theory solutions, called solvers, but they’re kind of slow. And now there are AI tools that you can layer on top of game theory solutions to provide fast approximate Nash equilibria — which creates problems such as cheating, for example, in online poker. But it is funny: if everyone is playing a pure Nash equilibrium strategy, but they still have to physically execute the strategy, then the edge comes all from physical reads, like coolness under pressure. I think it’s maybe not so intuitive what skills are prioritised by AI and which aren’t, necessarily. But I also worry that we can wind up in mild dystopias. One I call “hyper-commodified casino capitalism,” where you basically extract all the consumer surplus for producers and big corporations. They own our data. They have very good algorithms that use fuzzy logic to learn how to make us pay the exact amount we’ll pay for a flight from New York to London, and extract every dollar willing to pay. And they kind of nudge us in ways that are subtly or not-so-subtly coercive when we supposedly have choices, and not when we don’t. And that if you have high agency and have good intuition for what the AIs are doing, you can benefit from that; but if you don’t, then you get kind of suckered in. That seems like a dystopian worry that I think is not existential, but catastrophic. Also, some people’s p(doom) can include situations where human beings basically give up agency, where maybe it’s the AIs plus a few CEOs that have 90% of the decision-making capacity in the world. And that maybe we get to play cool video games and things, and have some nominal agency, but that’s greatly diminished. That worries me quite a bit too. Rob Wiblin: Yeah, people can read the book if they want to hear more about that. So you’re flipping the script a little bit and saying that it would be selfish to slow down AI progress. There’s another sense in which I think the rest of the world and people in poverty are kind of getting screwed that I don’t hear people talk about very much. Let’s say that the world, humanity as a whole, faces some tradeoff between what is its risk appetite: it’s got the potential for massive gain, but you think the risk of doom from AI is 2–20% — less than half, but still pretty material. We’ve got this tricky tradeoff to say, the reward is there, but how much are we willing to delay in order to drive that down from 20% to 15%? Who can influence this? US voters, kind of. Maybe California voters a little bit. I guess a handful of people in the Chinese Communist Party. Possibly some voters in the UK could influence it a little bit on the margin. But that’s basically it. Everyone else in the world, if they don’t like what OpenAI is doing, or they don’t like US voter policy choices on this risk/reward tradeoff, they’re just shit out of luck. And so if people in Nigeria have a different taste for risk versus reward, there’s just nothing that they can do basically at all. Nate Silver: Part of the argument too is based on the fact that, I think if you could press a button to permanently and irrevocably stop AI development, but you get another chance to press it in 10 years, that part is key. And it’s easier to say this now when you’ve had, A, more awareness of AI x-risk, and B, arguably, I think many people would say we’re at a slower period of development in large language models. I think that the fast takeoff scenarios are probably quite a bit less likely. I mean, you have to add more and more compute, and now Sam Altman wants $7 trillion worth of semiconductor chips and whatever else. I think there are probably going to be some plateaus or limits. So you extract all human text on the internet. I think to get to near-human capabilities versus getting to superhuman capabilities, I think that’s not a straightforward extrapolation. It’s one that I think would take a lot longer potentially, but I don’t know. I’m trying to not weight my own intuition that much also. Rob Wiblin: Do you have a thought on the democratic legitimacy issue? Should there be some global referendum on this, in an ideal world? Nate Silver: One Emmett Shear idea that didn’t make it in the book, but I think might be worth talking about here, is deliberative democracy. Which has been tried in different ways. I guess it was in Rome or Greece where they would just randomly call people and, “You’re going to have to be a senator. You have to be a senator for a year.” Or a jury system is very much like this. I recently had to beg out of jury duty in New York by saying, “I’m an author. I’m going on a book tour.” But something like that. Because you worry in a world where it’s kind of statistical sampling. And a poll, in a weird way, is kind of a version of this, right? You pick a random representative sample of people. I think democracy is a more robust metric than people assume. Rob Wiblin: Can you explain that? Nate Silver: I think that there is a lot of value in consensus. I think there’s value in the wisdom of crowds. I think people kind of know areas of their life. I voters actually have fairly good BS detectors. You know, I would never vote for Donald Trump. I’ll vote for Kamala Harris. And I voted for Biden and Clinton before that. But you can understand how a certain type of voter is upset that elites have become self-serving in different ways, and that they’re not being utilitarian. I think probably, from a utilitarian calculus, Kamala Harris is better for various reasons. Although you have to think about what are their senses on x-risks and things like that. I haven’t seen people make those attempts too often. But people aren’t voting based on calculating their utility. They kind of are, maybe for things that directly affect them, like taxes or particular benefit programmes, they might be. Or women might vote on abortion, or gay people vote on gay rights, or trans people on trans rights, and things like that. But they’re kind of voting on, like, “Where am I in this equilibrium? Am I on team A or team B?” And I think sometimes progressives and liberals and Democrats kind of rely too much on a certain model of rationality — where people are kind of expected value maximisers — as opposed to being in a game theory equilibrium, where it’s like, “Are you on my side? Are you on my side defending my interests or not?” The Democratic Party, for example, has been saying, “We are the normal party. And if you’re weird, go be a Republican.” Well, actually, probably a majority of people are weird by that heuristic. So why isn’t Kamala going after the weird crypto voters or RFK, Jr voters or things like that? I don’t know. Rob Wiblin: The game theory thing that I think people miss about democracy is that people evaluate, is democracy achieving their optimal policy outcomes? And it’s like, no, it’s not. But the real virtue that it has is that it’s reducing violence. And this isn’t so evident to us anymore, because countries like the US and UK haven’t had civil wars lately. But if you’re having elections every four years, and you lose, then you think, “What I should do is go away and try to persuade people and win in four years’ time.” But if you don’t get a chance to vote again, then you’re going to take up arms potentially against the government, if you’re sufficiently dissatisfied. Nate Silver: This is the Francis Fukuyama argument about why he thinks liberal democracy and market-based capitalism ultimately prevails. Because people are intrinsically competitive, and you need to have a certain amount of competition in the world. By the way, my other suspicion of EA is that I think it maybe, in the same way that Marxism misstates and underestimates human nature — that people want to compete; they want to compete and they want to have teams — and what’s the optimum level of having healthy competition, where you protect the downside of the losers to some extent? But people don’t want a utopian paradise. Rob Wiblin: I want to talk a little bit about election forecasting. So something that I haven’t heard you talk about before — which some listeners might have heard of, some won’t have — is this alternative election forecasting system called The 13 Keys to the White House. Perhaps it’s a little bit sadistic to force you to explain what this is, but could you explain what the 13 keys are? Nate Silver: So the 13 keys are a system by Allan Lichtman, who is a professor of government, I don’t know if he’s a professor emeritus, retired now, at American University in Washington, DC — which I think are an example of the replication crisis and junk science. One problem you have in election forecasting that’s unavoidable is that you have a small sample of elections since American elections began voting in the popular vote in 1860. Before that, you would have state legislatures appoint candidates. It’s a sample size of a few dozen, which is not all that large. And for modern election forecasting, the first kind of scientific polling was done in roughly 1936 — and was very bad, by the way, at first. One election every four years, so you have a sample size of 22 or something like that. So when you have a small sample size and a lot of plausible outcomes, you have a potential problem that people in this world might know called “overfitting” — which is that you don’t have enough data to fit a multi-parameter model. And there are different ways around this; I don’t know if we want to get into modelling technique per se. But the Keys to the White House is a system that claims to perfectly predict every presidential election dating back to the 19th century based on 13 variables. There are a couple of problems when you try to apply this, forward-looking. One is that a lot of the variables are subjective. So: Is there a significant foreign policy accomplishment by the president? Is the opponent charismatic? These are things that, if you know the answer already, you can overfit and kind of p-hack your way to saying, “Now we can predict every election perfectly” — when we already know the answer. It’s not that hard to “predict” correctly, when the outcome is already known. So when the election’s moving forward, then actually Allan Lichtman will issue his prediction. But it’s not obvious. You have to wait for him to come on stage, or come on YouTube now, and say, “Here’s what I predict here, based on my judgement.” So it’s a judgement call on a lot of these factors. Also, he’s kind of lied in the past about whether he was trying to predict the Electoral College or the popular vote, and shifted back and forth based on which was right and which was wrong. But he’s a good marketer, taking a system that’s just kind of punditry with some minimal qualitative edge or quantitative edge, and trying to make it seem like it’s something more rigorous than it is. Rob Wiblin: So it’s got 13 different factors in it. There’s so many things that are crazy about this. You don’t even need to look at the empirics to tell that this is just junk science and totally mad. So he’s got 13 factors that I guess he squeezed out of… I mean, in the modern era, there’s only like a dozen, at most two dozen elections you could think about — and we’re really going to be saying that it’s all the same now as it was in the 19th century. That seems nuts. So he’s got 13 different factors. Almost all of these come on a continuum. Like a candidate can be more or less charismatic; it’s not just one or zero — but he squeezes it into the candidate is charismatic, or the candidate is not; or the economy is good, the economy is bad — so he’s throwing out almost all this information. He’s got so many factors, despite the fact that he’s got almost no data to tell which ones of these goes in. He hasn’t changed it, I think, since 1980 or something when they came up with it. Nate Silver: Yeah. And he says, for example, that Donald Trump is not charismatic. By the way, he’s a liberal Democrat. And like, I’m not a fan of Donald Trump, but he literally hosted a reality TV show. He was a game show host. I think there’s a certain type of charisma that comes through with that. And that’s the one thing he probably does have, is that he’s charismatic. Maybe not in a way that a Democrat might like, but he’s a funny guy. He’s an entertainer, literally. So I don’t know how you wouldn’t give him that key, for example. And look, sometimes you have a situation where you have a small sample size. It’s not a terrible idea to say, let’s just take a whole bunch of different frameworks and average them out. That’s not actually such a bad idea. Here are all the reasonable approaches we might take, and make a mental average of the different models that you might take. But you want heuristics that you build in ahead of time that you don’t then have to apply subjectivity to. Even for forecasters, I think an election can be an emotional affair. You can have a personal preference for the outcome, or you can get invested in your forecasts — where you probably stand to gain more future opportunities if your forecast is perceived as being right. So you want to set up rules in advance that you’re not allowed to change later, more or less, unless something’s literally broken. Like the Silver Bulletin election model is several thousand lines of computer code. If we caught some actual bug, a minus sign instead of a plus sign, then we’d have to own up to that and change it, potentially. But I think it kind of defeats the purpose of rigorous forecasting. By the way, that model was already wrong this year, because it said that Joe Biden would defeat Donald Trump, and Joe Biden was losing so badly that he had to quit the race. So you have a survivorship bias problem too in evaluating this forecast. Rob Wiblin: Yeah. I think a statistician can just glance at this model and say that it can’t be right; we must be able to do better than this. But it’s incredibly popular. Every four years, it gets a whole lot of attention. I think it used to get more attention back in the ’90s. I guess people worry that academia lacks credibility now, but I think we forget just how bad people were at assessing evidence back in the ’80s and ’90s and early eras. Nate Silver: In some ways, it kind of fits a stereotype of what an expert is supposed to provide you with, right? Where it’s saying, “Here’s the hidden…” I mean, if you go and look at bestselling nonfiction books, every subtitle is like, “The hidden factor behind X,” or, “The secret factor behind X.” It’s a very well-marketed system, where it’s like, “I, the expert, I’m going to reveal these secret keys. And you put them together, and you unlock the White House,” basically. Whereas for the Silver Bulletin model, it’s just a fancy average of polls, basically. And actually, it’s a very hard problem statistically because of the way polls are correlated, and there’s lots of things you have to figure out. But yeah, just the polls, basically. And we’re never going to be certain which can be probabilistic. It’s actually kind of harder to sell to the mainstream, I think. Rob Wiblin: Yeah. Are there any other really popular, kind of fake quanty things in the media that get covered a lot that are similarly dubious to the 13 Keys? Nate Silver: There must be. There’s Groundhog Day: if the groundhog sees its shadow and things like that. Kind of quasi… But I think election forecasting is kind of unique in just the tempo of it. You have it every four years. I mean, there’s some stuff, if you’re watching a football match, either American or European football, then you’ll see “keys to the match” and things like that. And it’s often really obvious things, like, “The team that scores more points will win”; “The team that gains more yards will win, probably.” So there’s some of that: you’re saying obvious things and making them seem profound. I think that’s probably something of a universal. But elections are in this really weird zone where they happen once every four years — or once every two years, counting midterm elections — which is enough to have some regularity, but never to quite have certainty. Rob Wiblin: On that topic, I recently saw a paper titled, “Assessing the reliability of probabilistic US presidential election forecasts may take decades.” I think you might have seen this one. Nate Silver: Yeah, I tweeted about it. Rob Wiblin: Yeah. So I asked Claude to give a brief summary of the paper and some of the points that it pulled out were: Presidential elections are rare events. They occur only every four years. This provides very few data points to assess forecasting methods. The authors demonstrate through simulations that it would take 24 election cycles, or 96 years, to show with 95% confidence that a forecaster with 75% accuracy outperformed random guessing, and that comparing the performance of competing forecasters with similar accuracy levels could take thousands of years. What do you make of that? Nate Silver: So I challenge these guys to a bet. If they think that it’s no better than random, then I’m happy. I mean, right now, our model — as we’re recording this in early September, full disclosure — is close to 50/50. So yeah, if they think that that’s no better than a coin flip, then I’m happy to make a substantial bet with these academics. Because, look… Am I allowed to swear on this show? Rob Wiblin: Of course. Nate Silver: It’s like, OK, you have an event when it’s every four years. To get a statistically significant sample will take a long time. Yeah, no shit. You don’t have to waste a slot in an academic journal with this incredibly banal and obvious observation. But I’d say a couple of things. One is that when you actually have a sample size which is not just the presidential elections, but presidential primaries and midterm elections: in midterm elections, there are roughly 500 different races for Congress every year. Of course, they’re correlated, which makes this quite complicated structurally, but there’s a little bit more robustness in the data than they might say. But also, they’re kind of caught in this… I consider it the replication crisis paradigm of, like, you hit some magic number when it’s 95%, and then it’s true instead of false. And that’s just not… I mean, I’m a Bayesian, right? I don’t think that way. One of the authors of the paper was saying, based on one election, you can’t tell whether… So in 2016, models had Trump with anywhere from a 29% chance — that was a then-FiveThirtyEight model — to a less than 1% chance, 0.1%, let’s call it. And they said that you can’t really tell anything from one election which model is right or which model isn’t. And actually, it’s not true if you apply Bayes’ theorem, and you have a 0.1% chance happening on a model that’s never actually been published before, and it’s wrong. The odds are overwhelming that model is inferior based on that sample size of one to the 29% chance model. So to me, it kind of indicates a certain type of rigid academic thinking, which is not fast enough to deal with the modern world. In the modern world, by the time you prove something to an academic standard, then the market’s priced it in. The advantage that you might milk from that has already been realised. It’s interesting to see effective altruism: which comes out of academia, but understands that you’re having debates that occur quickly in the public sphere, on the EA forums, for example. And they’re big believers in being in the media. And that part I like: that the velocity of academia is not really fit for today’s world. Rob Wiblin: Yeah. I think presumably the authors of this paper wouldn’t really want to say that your model is no better than a coin flip. I guess what they’re saying is, imagine that there were two models that were similarly good — your model, and one that was a bit different, that gave a bit more weight to the fundamentals versus polling or something like that — and say it gave Trump a 27% chance when you gave it a 29% chance. It’s actually quite difficult to distinguish which of these is better empirically, and so you might have to turn to theory, and then that’s not really going to be decisive. What do you make of that sort of idea? Nate Silver: I get a little perturbed because we are the only… So the legacy of FiveThirtyEight, and now Silver Bulletin models, this is a pretty rare case of having forecasts in the public domain where there is a complete track record of every forecast we’ve made, both in politics and sports, since 2008. And they’re very well calibrated: our 20% chances happen 20% of the time. You get a much larger sample size through sports than through elections. But yeah, it’s this abstract of basically no other model in this space has a track record over more than one election. And we also are having presidential primaries and things like that; there’s quite a long track record now. And I would think academics who are interested in public discourse would be more appreciative of how it’s much harder to make actual forecasts where you put yourself out there under conditions of uncertainty, and publish them so they can be vetted and observed, than to back-test a model. And look, I think there’s probably some degree of jealousy, where… I mean, there is, right? You take these ideas and you popularise them and there’s a pretty big audience for them. But also, I’m taking risks every time I forecast. I mean, we’ve had 70/30 calls where we’re perceived as being wrong, and you’re taking reputational risk. So I don’t know. Rob Wiblin: Yeah. I guess the real reason that I care about your forecasts and think that they’re credible is less the track record and more that I’ve looked at and I understand how it operates internally. I think, yes, that is the process that’s actually generating the outcome, that this is where the randomness is coming in. But this paper made me realise I didn’t understand it perhaps quite as well as I thought I did. So given that you only have a dozen, two dozen data points in the modern era that you can think about, how exactly do you figure out how to weight, say, the fundamentals versus the polling, and how to change that over time? It seems like you might not really have enough data to specify that very closely. Nate Silver: This is where it’s both an art and a science for sure. There are things like, I probably err on the side of weighting the fundamentals — meaning things like the economy or incumbency — less, because there’s more researcher degree of freedom in that. There are dozens of economic variables that are published, or actually more than dozens: there are thousands of variables that are published by the Federal Reserve or other organisations and updated quarterly or more often. So part of it is having experience of model building. You don’t want to fit every parameter to the back-tested data; you want to say, let me come up with a good index of the economy. For example, how would someone trying to determine if the economy’s in recession (is kind of the idea) create an index of the economy which is not binary — between 1 and 0 — but fluid? And then having picked that definition of the economy, combine that with the polls and see how well that it does. And then, when you’re doing that, understanding that there probably still is a little bit more overfitting in one part of the model than the other. So you have a lot of decision points if you’re designing a complex model like this. It’s, again, several thousand lines of code. You might have like 40 or 50 legitimate points of decision to make. I think it’s worthwhile to keep mental notes on maybe in which direction those different decisions might err, right? If you make a decision like, “This is a slightly cleaner way to do it, but might be a little bit overconfident or might lend itself toward overconfidence,” maybe the next decision that you make, you would say, “OK, this is going to be a more cautious assumption.” But it’s tough. I think anyone who says that you just kind of feed data into a computer — I mean, maybe with AI it becomes more where you lose legibility — but I’m still working in a space where, if you read Silver Bulletin, there’ll be these like 2,000-word-long posts that explain, “Here is what the model is doing. Here is why, when I designed the model, I designed it this way. Here is why that assumption may or may not be right, right now. And you can look at the raw data before we get to a certain stage of the model and make a different assumption.” But that’s the thought process that you actually have in the real world. I’ve done consulting from time to time for financial terms, and they’re like, “We want both your model and we also want your experience and your brain,” so to speak. A model is a tool or a device. It’s important not to use the model as necessarily being oracular. It’s a disciplining mechanism to force you to think through questions more rigorously. Now, when you get into truly data-rich environments, like some sports applications, then we have thousands and thousands of games to test this on. We worry a little bit about overfitting the model, but for the most part, you can be more strictly empirical. You can’t do that in these kind of small-N problems — which are, again, more like the problems that you might face in EA or rationalism, right? Where it’s about how do you make a good estimate? Sometimes parts of it you can model very rigorously and sometimes parts of it that you can’t. You hope that the parts that are more rigorous are kind of higher leverage. One thing that’s very important to do is robustness check. If there are two reasonable ways to specify a parameter or a function in the model, and they give radically different answers, then almost for sure you have to find some way to take an average of those two. If there are two different ways they give you the same answer, then you can simplify it by saying, “I’m just going to use one of these, because we don’t need the extra degree of freedom” or whatever. So understanding that — when you kick a tire on a model and how robust it is to changes in assumptions — that’s the skill. That’s the experience and art, I guess, of model building. Rob Wiblin: On this question of how low of a data environment this actually is, the paper makes the point that you might think you’re forecasting 50 different states, so in fact you have 50 different data points, but actually you don’t, because these are things that are super correlated. If you get half of them, probably you’ll get the other half as well. And likewise, if you get the first half wrong, the other one is probably going to have errors as well. It seemed like quite an American-focused paper, because couldn’t you think about this modelling operation or this forecaster, and they’re going to forecast elections both in the US and all around the world using similar mentalities, using similar methods? Then you could suddenly have a massively larger dataset on how good that modeller is at thinking about elections in general. You might have some external validity questions, maybe the accuracy for just one class of elections might be a bit different, but I still think you could expand the N enormously. Nate Silver: Yeah, for sure, if you looked at all European parliamentary elections, for example. You have to be careful. In India, for example, the polling is incredibly inaccurate. It’s off by an average of about 11 points, whereas in the US and Europe it’s about three points on average. The reason why is partly that there are so many different ethnic groups and racial groups. But also it’s a country where there is not a culture — and I’m stereotyping, and I apologise, but I have spoken to local experts on this and done some work myself — where it’s not a culture where you’re necessarily terribly forthcoming with strangers, right? If someone asks you for your political views, maybe you’re not going to communicate that in a way that reveals your true preferences. Whereas in the Anglo cultures, especially the US, we tend to be more forthright about that stuff. Rob Wiblin: On this question of people treating forecasts in an oracular way, there’s been a real flourishing of prediction markets. There’s Manifold, Polymarket (which you do some consulting for), Metaculus. I’ve spent years trying to get people to pay more attention to these things. Do you think now possibly they’re a bit overrated? Especially the small markets, rather than big presidential elections — niche things like, “Is Elon going to drop his lawsuit against someone or other?” Nate Silver: There can be times when — and this is true, by the way, of other types of markets; it can be true of sports-betting markets, for example — people ascribe too much wisdom to the markets, and it becomes kind of circular logic, right? So during the Democratic National Convention, a rumour circulated that there’s going to be a surprise guest appearance. And of course, Polymarket, I think maybe Manifold too, had betting markets on who the surprise guest would be. Would it be Taylor Swift? Or some Republican, like George Bush, endorsing Kamala Harris? However, the market decided that it was going to be Beyoncé, and these rumours began to swirl and circulate. So people would tweet, like, it’s now 80% for Beyonce, and then go all the way up to 96% or 97%. And then her representatives had to send something to TMZ saying that she’s not even in the same city right now as the convention’s being held, so this is not going to happen. So that might’ve been a case of circular logic, where everyone thinks everyone else knows something. You have this kind of false focal point. For the most part though, for political betting markets, you finally have a lot more liquidity. They’re very different. To me, the leading two are Polymarket and Manifold. Or Manifest? Manifold. Manifest is a conference that Manifold Markets held. Manifold is play money, but has an incredibly dedicated community that cares a lot about reputational stakes and skin in the game. Polymarket has real liquidity and is well structured. Some of the ones in the past would make it such that you had asymmetric problems with betting on 2% outcomes and things like that. These markets are now pretty well structured and have enough volume and liquidity to be quite a bit better. I know that as everything becomes kind of eaten by politics, then financial firms, investment banks, and hedge funds want to consider political risk, right? If you’re trying to forecast interest rates in the long term, whether Trump or Harris would be president is quite relevant to that. Or US–China foreign policy, and the long-term value of Nvidia, or things like that. So you have a market need to forecast political risk, and now you have some tools to help with that. Rob Wiblin: One of my colleagues has been a fan of prediction markets over the years, but is now a little bit worried that they’re overrated. He wrote this into me, saying: It’s useful to make models about complex questions. But when they spit out a number, that doesn’t necessarily mean that you’ve learned that much. And prediction markets can launder what are little more than guesses into numbers that have much more respectability than a few people’s opinions. But sometimes they’re not actually that much more than that. Also, the Brier scores of even superforecasters aren’t that great. So forecasting is really more like poker than chess: the experts can have an edge, but often go on very long losing streaks to even amateurs. Do you agree with that? Nate Silver: I agree with that for sure. In markets in general, including gambling, you can have small edges that persist for a long time if you’re very good, or large edges that persist for a short time if you’re very good — but you never have a long edge that persists for a long time. There’s too much efficiency in the market. You can also have recursiveness. So some people say, why don’t you use prediction markets as an input in the Silver Bulletin model? Well, the Silver Bulletin model moves prediction markets sometimes, so you have a certain amount of recursiveness. And it can be easy to say, “All these independent indicators suggest the race is going to go this way” — when in fact there’s non-independence; when in fact, the reason why experts think it’s going to be Harris or whatever is because prediction markets say that. And the reason prediction markets say that is because the experts say it. And if you then have polling models that are non-independent from that, then you wind up in a place where you can become overconfident and have these big asymmetric tail risks. Rob Wiblin: Yeah, it’s interesting. That’s a case where possibly the forecast would be more accurate if you added in other indicators like experts or prediction markets — but it would be less useful in a way, because it’s just merging in your thing with other stuff in a somewhat unclear way. Nate Silver: Yeah. This is another thing that you learn by building a lot of actual models, is the amount of… The trader term is “alpha,” the amount of value you provide. (That might not be quite right.) But an indicator that’s highly correlated with other indicators — for example, let’s say, if you had the polling average, the polling average plus 0.1% for the Democrat will be almost as accurate as the polling average, but it provides no additional value, because it’s just a linear function of the polling average. Whereas something like the number of yard signs in a political campaign in Manassas, Virginia, or something is totally uncorrelated with the polling and may be a very poor predictor on its own, but might provide an extra 0.01% of R2 or something. Rob Wiblin: So in my world, people who have qualified for the “superforecaster” qualification tend to be treated a little bit as oracles, or their opinions are given a lot of extra weight. There was this forecasting tournament, which we’ve actually had an interview come out about: the Existential Risk Persuasion Tournament. Ezra Karger ran it, which you talk about in the book. And in that experiment, they found that superforecasters who didn’t know about AI in particular thought there was a 0.2% chance of AI causing doom — in broad strokes, 0.2% chance — whereas experts in AI who weren’t superforecasters thought that there was more like a 2% chance of AI causing extinction by some end date I can’t remember. And somewhat to my surprise, you said that you would probably go with the AI experts over the superforecasters in that case. Why is that? Nate Silver: I do think that AI experts have enough grounding in the rationalist community and being aware of these kind of cognitive biases that forecasters have, so that I think they’re actually incorporating those heuristics into their models for the most part. I’m giving them credit for being kind of super nerdy — probably in many cases, high-IQ nerds who are already accounting for that — but then add domain knowledge to that. Usually the issue is that if you have the inside view, you have more information, but your heuristics may be worse for varying reasons — ranging from that you’ve never really studied the kind of meta stuff about forecasting, to the fact that you’re a little bit close to the data and you may have perverse incentives to some degree. Rob Wiblin: Well, you might have been selected. Maybe you’re an AI expert because you had this view. Nate Silver: Yeah, obviously it’s “an every hammer looks like a nail” problem. Am I saying that right? Rob Wiblin: Yeah, I think that’s right? Every nail looks… Nate Silver: “When you have a hammer, everything looks like a nail” is the correct metaphor, I guess. So yeah, I think there’s something to be said that if you survey only people who are in AI risk… But I think the Katja Grace survey also surveys engineers for the AI labs, and not just people who are in AI safety per se. And they also are fairly worried. Rob Wiblin: They say more than 2% chance, I think. Nate Silver: Yeah. Some of them, the medians could get up to 5% or 10%. So yeah, I’m giving credit to the broader EA/rationalist community for having better meta heuristics than your typical pundits, I suppose. Rob Wiblin: Something you say in the book that surprised me is that venture capitalists talk a big game about taking risks and revolutionising the world and changing everything and being willing to upend it all, but you actually think they don’t take that much risk. Why is that? Nate Silver: Are you a basketball fan, or a sports fan? Rob Wiblin: Soccer sometimes. Nate Silver: In American sports, we have the draft, which is a mechanism for giving the worst team more competitive equality over the long run. If you’re the worst team, you get the first pick. You therefore get the next good player. For the top Silicon Valley firms, it’s almost the reverse of this, right? Where if you’re Andreessen Horowitz or Founders Fund or Sequoia, and you’re very successful, then you get the first draft pick: the next founder that comes over from Ireland or Sri Lanka or across the country or whatever else will want to then work with this firm that has bigger network effects. I mean, Marc Andreessen even told me that it’s kind of a self-fulfilling prophecy, their success: they have access to the top founder talent all over the world; the expected value of any given bet is quite high. And yes, there’s high variance, but he actually gave me some data — and if you do the math, almost every fund they make is going to make some money. The risk of ruin is actually very, very low. Rob Wiblin: Because they’re diversified across hundreds of different…? Nate Silver: Diversified. You have a fund that has 20 companies a year. And by the way, it’s not true that it’s totally hit or miss. There are plenty of 1xs and 2xs, or even getting half your money back. That helps actually quite a bit over the long run. So it’s a very robust business where they’re guaranteed to make a really nice profit every year. Look, I think a lot of them are risk-taking personality types, but they kind of have a business that’s too good to fail almost. Whereas a founder may be making a bet that is plus-EV in principle, but the founder’s life may not be that great much or most of the time. To commit to an idea that has a 10-year time horizon that results in complete failure some percentage of the time, results in a moderate exit after a lot of sweat equity some larger percent of the time, and has a 1-in-1,000 chance of you’re now the next Mark Zuckerberg, or 1-in-10,000: that’s less obviously a good deal, depending on the degree of risk aversion you have. And you have to have some risk-takingness — or risk ignorance, I call it — in order to found a company in an area that has not achieved market success so far and has this very long time horizon. Rob Wiblin: Reading this made me wonder: Are founders getting screwed by VCs? Why is it that they’re in this relationship where the VCs are taking very little risk, but they get a high, nearly guaranteed return? The founders are taking enormous personal risk. They also some of the time do really well, but lots of the time they don’t do that well at all. They bear much more risk than the VCs do. Why don’t the VCs give them more of a stable salary, like give them $200,000 every year so that they’re more insured against the risk that their business goes badly, given that they can easily afford it? Nate Silver: Yeah, it’s interesting. Maybe there are opportunities for different VC business models. Maybe whichever VC has a big falling out with one of the big VC firms should try an alternative model, where it’s a little bit more founder-friendly potentially, or you get more equity early on. I think they sometimes want founders who are more hungry, and sleeping in the group house and things like that. I wonder, and I think, frankly — again, I’m not super PC — but I think they have to be missing talent from founders who were women, or founders who were not white or Asian men, basically. I mean, that has to be true. Rob Wiblin: Something you explained in the book is that VCs talk about being very contrarian, being different than other people — but you actually think that they are super herdy, that they tend to be very conformist within VC. Why is it that that is the better strategy, rather than doing the thing that’s different from what everyone else is funding, so you can get new and different opportunities? Nate Silver: For one thing, I don’t think that this market is necessarily hyper-efficient. It’s such a good ROI if you’re a top-decile firm that I don’t think they’re optimising every angle necessarily. I think they’re getting the big heuristics right, which is: have a long time horizon and bet on companies with big upside. If you make enough of those bets, that’s a very high expected value on average. And they get a lot of things at the margin quite wrong. There are other things too, so it’s very hard. You can’t really short an early-stage company, so therefore it’s a culture where there’s not a lot of negging. And they all invest and reinvest at different stages of one another’s companies, so they have a lot of correlated interest. I think one reason why they’re so bothered by criticism as compared to the hedge fund guys: the hedge fund guys are always looking for where is conventional wisdom wrong; they’re critical, and used to being criticised. Whereas Silicon Valley is a no-negging culture as much. It’s also just a small number of people. The number of VCs that really are movers and shakers is a couple of dozen at most, probably. Whereas the number of movers and shakers in Wall Street, the New York hedge fund/investment bank/private equity world is hundreds or thousands, probably. So the scale is really small, and maintaining group relations in some way is… What they’re trying to do is predict the behaviour of their friends. They are the influencers, almost, who are going to the hot new club and they want to ensure their friend will have a good time at the hot new club. If it feels played out, they’ll lose. If they’re too early on the trend, they’ll lose. It’s more of a social activity than people assume. And it’s a small, tight-knit group of people. Rob Wiblin: Final question: With the venture capitalist world, you say the reason that they have to herd so much is that they don’t want to be 100% of the funding for any company as it scales up. So it’s a big problem for them. Imagine that they make a great investment in a company that has a lot of potential, but it’s weird in some way. You talk about how a Black woman founder might be an example of this. Where someone thinks this business has great potential; however, they think that other VCs won’t be interested in funding it because of their prejudices, say. And so they don’t want to get in, because this company won’t be able to raise enough funding from enough different groups to go through its series B, series C, and so on. VCs could solve this problem individually if they were willing to take more risk. If they were willing to say, “I’m willing to be 100%. I’m willing to back this person 100%, and even if other VCs don’t like it, I’m going to take them all the way just with my money. And even though that exposes me to less diversification, I’m willing to do it.” Should they do it? Nate Silver: I agree. They have plenty of money. Maybe they should be more of that. Maybe it’s mission-driven VCs. Some VCs are more mission-driven for companies they think will be good for the environment, for example, or good for global welfare and poverty reduction. I think there’s room for more diversification within the VC model. Again, in the whole history of like gambling or investing, to have edges that are very large, that persist for a very long time, that’s very rare. So maybe in 20 years someone will write the next book about how 2024 was the time of peak VC, before people realised that this is too good to be true. And maybe between other private options and maybe governments and founders taking more revenge — or not revenge, but saying, “Actually, I have more leverage as a founder than you might assume” — and maybe chipping away at these extremely high and impressive but extremely high excess returns. Rob Wiblin: My guest today has been Nate Silver, and the book is On the Edge: The Art of Risking Everything. Thanks so much for coming on The 80,000 Hours Podcast, Nate. Nate Silver: Of course. Thank you so much. Rob Wiblin: If you liked that episode, some others you might like include: #163 – Toby Ord on the perils of maximising the good that you do #200 – Ezra Karger on what superforecasters and experts think about existential risks #176 – Nathan Labenz on the final push for AGI, understanding OpenAI’s leadership drama, and red-teaming frontier models All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris. Video editing by Simon Monsour. Audio engineering by Ben Cordell, Milo McGuire, and Dominic Armstrong. Full transcripts and an extensive collection of links to learn more are available on our site, and put together as always by Katy Moore. Thanks for joining, talk to you again soon.