Latest / 80,000 Hours Podcast / #254 – Max Nadeau on why ambitious people should start AI safety nonprofits
Transcript
- Max Nadeau: I think the most important bottleneck, maybe the only bottleneck, is talent. In absolute terms, you can get a lot of money for ambitious nonprofits. … I mean, $200 million is not like an upper limit on the largest size of a grant Coefficient Giving will ever give to an organisation in the technical AI safety space. So I think that there’s a lot of opportunity for people who are willing to get down in the mud of these speculative questions, and are willing to entertain these sorts of out-there hypotheticals. I’m hoping we’re going to put out this call and then a lot of really, really talented people are going to raise their hands and say, “Yeah, I want to work on that.” Zershaaneh Qureshi: Today I’m speaking with Max Nadeau. Max works at Coefficient Giving, which is the philanthropic funder that was previously known as Open Philanthropy. His team, the Technical AI Safety team, makes grants to support technical efforts to mitigate catastrophic risks from AI — so that’s things like interpretability, alignment research, evaluations of AI behaviour, and so on. I’m talking to Max today because Coefficient Giving is launching a new funding initiative, basically getting people to start nonprofits that plug the biggest gaps in technical AI safety. And there’s quite a lot of money on the table: there’s between $200,000 to $200 million of funding available for a new nonprofit — depending, of course, on the stage of the grant and how promising your proposal actually is. But I’m just super excited to dig into this, so Max, thank you so much for joining us. Max Nadeau: Thanks for having me. Zershaaneh Qureshi: And before we begin, I should disclose that Coefficient Giving is 80,000 Hours’s largest donor — although we haven’t received funding directly from Max’s team. So you were previously at Redwood Research. I’m interested to hear what you got up to while you were there. What did you do? What’s your origin story? Max Nadeau: The primary work I was doing at Redwood was actually fieldbuilding rather than direct technical AI safety research. So I was running the machine learning for alignment bootcamp that Redwood ran, and then I also was running the Redwood model internals experiment, which was shortened to REMIX. Both of those were programmes in which Redwood had something like 30 to 50 people come for a short period of time to learn more about AI safety research and develop their skills. And I’m happy to report that many of the people who went through those programmes went on to full-time work in AI safety. I also dabbled a little bit at Redwood in doing interpretability work directly, but frankly, I didn’t really like doing the technical AI safety research myself very much, so it was a better fit for me to do grantmaking work instead of being a researcher directly. I think maybe now that the AI agents do all of the drudgery, I might enjoy technical AI safety research more these days if I were working on it, but I’m pretty happy in my current career trajectory too. Zershaaneh Qureshi: Got it. So you’re launching this new initiative called Project Tailwind. Can you tell us a bit about what this project is, specifically? Max Nadeau: Yes, definitely. Tailwind is a new initiative that we’re starting at Coefficient Giving to get more philanthropic startups in the AI safety space. And these are primarily, but not exclusively, technical startups. We’ve got a long list of I think [36] project stubs on the Tailwind website which you can read, which are all ideas for projects and organisations in the AI safety space that we’ve identified as being major opportunities for impact that we need someone to found. We’re making grants of a couple of different types: We’re making preseed grants to founders who may not have established a team yet, may not have a super detailed vision for what the organisation will do, but have some preliminary ideas. Those preseed grants are [typically] between $200,000 and $2 million. Then we’re making seed grants, which we can do right out of the gate if the application has the important qualities. Those important qualities are: There needs to be some amount of a founding team There need to be multiple people who are covering the important bases for the organisation And some initial, obviously very preliminary, results — either technical or otherwise — showing that they have the capacity to make progress on the goals that they have as an organisation. And those seed grants are going to be between $2 million and $20 million. And for some organisations — this could be the first grant that we make them, but more likely it will be the second or third grant, once they’ve had some time to establish themselves and set up a track record — we’re willing to make grants up to $200 million, even for new organisations right of the gate. And longer term, $200 million is not like an upper limit on the largest size of a grant Coefficient Giving will ever give to an organisation in the technical AI safety space. I personally would predict that at some point in the next year or two, we’ll make grants that are higher than that. But obviously that will depend on many factors, like how good the organisations are, and how much the funders want to support that, and all sorts of other things like that. Zershaaneh Qureshi: And in terms of your approach these days, one way that you’re kind of replicating the venture capital model, I understand, is that you’re just moving quite fast. Are there any other things that you’re borrowing from the VC world? Max Nadeau: One thing, which has always been a part of Coefficient Giving’s philosophy, is the notion of “hits-based giving,” which is a borrowing of “hits-based investing” from the investment venture capital world. That’s the idea that you make a bunch of grants, and the majority of them don’t have any impact, but a couple of them are so great that they make the whole portfolio worthwhile. That’s something that we’re leaning even harder into right now in light of the rapid progress in AI. We’re trying to make sure that we’re not falling into a common failure mode for grantmaking institutions — which is to be overly focused on scrutinising budgets and making sure that every cent is going to good use — and making sure that, in addition to due diligence, we’re also paying a lot of attention to ensuring that the best projects have all of the funding they need, and that we’re really paying attention to the upside that’s available from moving fast and supporting things generously. There are some other more tactical strategies that we’re borrowing from the venture capital world. For example, my colleague Jake Mendel has run a couple of “pitch days” — where he has interested founders come and spend 15 minutes describing their plan for a new AI safety philanthropic startup, and then gets them a decision very quickly, like in [a couple weeks], about whether or not we’re going to make them a preseed grant. And we’re also using scouts to try and help us identify some of the best people in the world to run these new startups that we’re supporting. Zershaaneh Qureshi: Yeah, super interesting. I am wondering, though, whether you lose out on anything by taking this faster, more dynamic approach. Does it mean that you get to be a bit less diligent than you ordinarily would like to be with your grantmaking? Max Nadeau: Well, I think tradeoffs of that sort are unavoidable. We’re trying to keep them to an acceptable level, in particular by hiring more grantmakers so that we can get answers to as many grant applicants as possible as rapidly as possible without trading off against diligence. Zershaaneh Qureshi: Got it. Something I know that you’ve done recently is you gave a $160 million grant to an organisation called Resolution. That’s obviously massive. Can you tell us a bit about what makes a grant application exciting enough and promising enough to earn that amount of money from you? Max Nadeau: Yeah, totally. With Resolution, the CEO, Geoffrey Irving, is really just one of the luminaries, the leading lights in technical AI safety. He was the chief scientist at the UK AI Security Institute. And before that, he led teams doing AI safety work at Google DeepMind and at OpenAI. And when he came to us, in addition to being a great founder, the plan he had on paper sounded right up our alley. He wanted to start a large, ambitious research centre that was going to work directly on the problem of aligning superintelligence — which is where we think a lot of the existential risk from AI comes from, and is a subject that we think is sorely neglected by the existing portfolio of research bets in the alignment ecosystem. Just to throw on even more positive qualities, part of his plan is to spend a lot of time and effort on using AI productively to accelerate his safety work, which is something that’s really important to us — because we think that there will be, and to some extent are, opportunities to make five years of progress in one in AI safety, while that is also happening in AI capabilities research. And we really want to make sure that our grantees are taking advantage of those opportunities. So Geoffrey’s enthusiasm for that approach to the problems he’s solving was also a really big plus in our books. Zershaaneh Qureshi: Yeah, awesome. So obviously Geoffrey Irving is, as you say, quite an esteemed guy. For these quite big-money grants, are you looking for founding teams that have quite big names in them, or does a regular person have a chance of getting that amount of money for a really exciting project idea? Max Nadeau: I think right out of the gate you need to have some pretty hard-to-fake signals of being a really thoughtful and well-intentioned founder if you want to get the sort of money that we were able to support Geoffrey with. But I think that we are very enthusiastic about taking new organisations that are demonstrating really great results and having a lot of impact, and scaling them up to that scale within a relatively short period of time after they’re founded. But that would be a case where we’d be making the grant primarily based on the results of the organisation, rather than on the profile of the founder. Zershaaneh Qureshi: So as I understand it, there are quite a lot of overheads involved in starting a new organisation, and other downsides — like smaller or newer organisations may just get taken less seriously than more established ones, or be less well connected, and overall have a harder time or take a longer time to have an impact. If very risky, advanced AI systems are indeed coming quite soon, then I guess my question is, wouldn’t it be better for Coefficient Giving to just go totally all-in on scaling up existing organisations and helping them expand their focus areas? Shouldn’t that be what you’re prioritising? Max Nadeau: So yes, we are definitely doing that as well. And in fact, I’m very excited about the grants that we’re making in that space. I think that for a lot of people who are going to potentially be interested in founding new organisations, they should really consider going to work at some of the existing third-party nonprofits in the space for all the reasons you list. I mean, these organisations already exist, they already have credibility, they already have processes in place. They already have, in many cases, years of thinking about what their theory of impact is. So if you’re a founder, you should be thinking like, “Maybe I do want to start a new team within Redwood Research, within METR, within TruthfulAI, within Epoch, or within one of these other third-party organisations.” So yeah, totally agree. I think, however, there are a lot of projects that are not being done right now and that are maybe not a very good fit for these sorts of organisations, and those projects should happen too. Basically we’re trying to do an “all of the above” strategy. So for people who don’t want to work in one of these existing organisations, or who want to do a project that wouldn’t be a very good fit for the existing startups in the space, that’s what Tailwind is for. We want those people to be able to go their own way and have their own startup that is able to pursue their vision for an impactful direction within AI safety. Zershaaneh Qureshi: So I’m pretty curious here about what you think the bottlenecks actually are to getting all of this kind of work done. What are the main reasons that the field isn’t already in a position, given that it already has several AI safety organisations, to do all of this kind of work? Max Nadeau: I think the most important bottleneck, maybe the only bottleneck, is talent. There are a number of really exciting AI safety organisations, but at each one of them, there’s a long list of projects that they would love to do, engagements with AGI companies and with governments that they would love to take on, that they just do not have the staff to execute on. Which is a big reason that we’re doing Tailwind, and why we’re trying to scale other parts of the AI safety field: we just need more talented people to come work on these problems. Zershaaneh Qureshi: So you’re saying that talent is the biggest bottleneck right now for AI safety. Can you tell me more kinds of talent we’re most missing right now, especially in terms of the skills and profiles needed in founders and founding teams? Max Nadeau: For the purposes of this conversation, I’ll mostly talk about the founders specifically, although I do want to just make clear to the audience that I think there are lots and lots of opportunities for people who are not interested in founding new organisations to have a lot of impact in the AI safety space, and in the third-party AI safety space. But I think that founders are a really important bottleneck. When it comes to what we’re looking for in founders, and what we think is most important for a founder in the AI safety space, I think there’s a lot of attributes which are shared with normal for-profit startup founding: It’s really valuable to have perseverance — you know, to be willing to run through walls. It’s really important to have energy for not just heads-down technical work, but also managing employees, dealing with funders, generally communicating what your organisation does to a wider audience. And then I think there’s also some distinctive attributes to AI safety philanthropic startups which are not as important for the normal for-profit world. The biggest one is being really discerning and careful about your theory of impact — really thinking deeply about what your organisation is doing, how it’s supposed to make a difference on future risks from AI systems that haven’t been created yet, and being willing to rethink your strategy, to pivot dramatically even, if you see a higher opportunity, higher-impact avenue. Relatedly, I think it’s very important for people founding organisations in the AI safety space specifically to be willing to think about speculative questions that don’t have clear answers, to think about what the future of AI holds and take best guesses about where these trends are going, and then try and be prescient and try and act in preparation for the AI systems and the risks that are emerging down the line. That’s something that’s very unique about AI safety, even compared to other philanthropic cause areas: the problems that people are working on mostly haven’t manifested yet. And a lot of people are very averse to that; they want to only work on problems that are more grounded or more empirical or more demonstrable today. So I think that there’s a lot of opportunity for people who are willing to get down in the mud of these speculative questions, and are willing to entertain these sorts of out-there hypotheticals. We’ve seen that strategy work very effectively in the case of organisations like METR [editor’s note: METR is not a Coefficient Giving grantee], for example, who were willing to, before AI capabilities were really taking off, come up with a way of evaluating AIs that aged much better than other benchmarks made in that era, because they had that additional foresight and that additional willingness to be speculative. Zershaaneh Qureshi: How embedded in the AI safety world do founders actually already need to be to have a good shot of doing this? I understand from what you’re saying that they probably need to have a fair amount of knowledge and opinions about AI, but do they need to already have a strong network in this space and so on? Max Nadeau: Having a network definitely helps. I don’t think it’s absolutely essential, but I do really think that it’s important for founders to be pretty high context on the existing work that’s happening in the AI safety space: the existing arguments that are taking place about what the future of AI holds, what the disagreements are, what the fault lines are, what the different models are that people have for the way that the future is going to go, the way that AI development is going to proceed. That takes a lot of time to get up to speed on, so I don’t want to say that somebody can just come into the AI safety field without having a lot of preexisting understanding of the domain that they want to start an organisation in. Because unlike with for-profits, where you can learn on the job and learn from this impossible-to-fake demand signal of, “Are you making money?,” in AI safety, no one will tell you if you’re not having impact. I mean, we as funders can try and provide some signals about that — but ultimately, organisations are independent, and they have the effects on the world that they have. So there’s really no substitute for a founder who’s taking that stuff really seriously, and spending a lot of time thinking about it. Zershaaneh Qureshi: Do you have a theory about why it’s so hard to find good founders for AI safety organisations at the moment, given the kind of profile that you’ve sketched out? Max Nadeau: Well, I’m hoping it won’t be hard. I’m hoping we’re going to put out this call and then a lot of really, really talented people are going to raise their hands and say, “Yeah, I want to work on that. I want to start a new organisation like that.” I think to some extent there’s just like a chicken-and-egg problem — where people haven’t realised how much money they can get from Coefficient Giving and from other founders in the AI safety space to start philanthropic startups, and we haven’t heard about them. And I’m hoping that Tailwind helps improve that. Zershaaneh Qureshi: OK, so if you are trying to found something new, I think there are some pretty obvious ways that could go wrong — like the founding team could fall out, and that could be pretty bad. But are there any less obvious mistakes that you see founders or early employees making that you think people should be trying to avoid? Max Nadeau: I think one of the biggest ones is not being ambitious enough about the magnitude of the problems that you’re targeting, and getting sucked into things that are either smaller scale than the sorts of risks that your organisation could be working on; or maybe they’re more present day, and so they’re easier to work on, but they’re just not as large, so in a couple of years you’ll be thinking, “Why was I working on that? Why didn’t I pivot sooner to working on this other thing, which felt speculative and early stage at the time, but like now seems really important?” So I think just trying to head that off in advance is a mental move that I would want a lot of founders at these new organisations to be making. Another failure mode that I would call out, especially for nonprofits, is that, because you don’t have to actually get customers in the world to fork over money for your goods and services, there’s a common issue of not thinking hard enough and not spending enough time focusing on the people in the world who you actually want to change their ways on the basis of the work you’re doing. Either you want them to read the reports or the evidence that you’re putting out and change their mind about some important topic, or you want them to implement some practice that you’ve developed that they’re not implementing, or you want them to pursue a line of research that they can pursue but you can’t. As a philanthropic startup, you need to really be thinking about, like, the entire supply chain, or like every step in the recipe of having impact — which both involves having good ideas, but also involves actually getting those good ideas into practice in the world, and into the minds of other people. And those two parts of the process can rely on different strengths and involve different styles of thinking. Zershaaneh Qureshi: Got it. You’ve talked a bit about the need to sometimes make a big pivot if you’ve started a new organisation. Do you have any case studies of pivoting going especially well or especially badly? Max Nadeau: Yeah. Shortly after I left Redwood Research, the two leaders of the organisation, Buck Shlegeris and Ryan Greenblatt, decided that the research field that they had been working in, of mechanistic interpretability, was not producing as much progress as they had thought, and was not the most impactful thing that they felt they could be working on. So they made the really difficult decision to downsize their organisation, and shrink down to just the two of them and spend like a year just thinking about the landscape of AI safety and what the biggest holes were and what the most underrated approaches were for making AI safe — and then emerged from their contemplation with a bunch of great ideas about AI control and other techniques for reducing risks from AIs, and have now scaled up their organisation again. At the time, they got a lot of flak for this, and it was an unpleasant experience for everyone involved. But I think now, looking back on it, they — and I, and basically everyone else I know — thinks that they’re having a lot more impact than they would have had if they had stayed the course. And this is a case where there was no market force that pushed them to do this; there wasn’t really pressure from funders to do this. There was pressure in the opposite direction from a lot of the people who they knew. But because they were really just personally internally committed to trying to have as much impact as possible, they decided to make this change. And that’s exactly the spirit that we’re looking for in other founders. Zershaaneh Qureshi: Yeah. I want to talk a bit more about the Redwood story, because I’m sure this really sucked for the employees who ended up being fired in this situation. Max Nadeau: Oh, definitely. Zershaaneh Qureshi: I don’t know, maybe you caught a bit of a sense of how people were feeling about it after this happened. From my perspective, it seems like a really hard call to make with some definite costs. And I think the thing that I am thinking about here is how I would feel very squeamish about making a decision like that. And if I was also an employee at an organisation like that, I would be feeling kind of nervous to work with people who I thought might do a big overhaul of everything that could be costly for me. I’m kind of wondering how much you think that being a good founder requires you to have an appetite for being a bit ruthless and do things that are going to have some short-term costs like this? Max Nadeau: I think it totally does require that appetite. I think there are ways of mitigating the short-term costs — and I think Redwood did not take all of them; I think they took some of them — but yeah, I think ultimately at the end of the day there’s just inevitably going to be some stress and some pain that is involved in these sorts of decisions, and that’s just par for the course. I think also this aspect is maybe not unique to AI safety or to philanthropic startups. My understanding is that it’s pretty common for startups just in the regular tech world to decide to pivot dramatically. And that definitely involves focusing on a different set of priorities and a different set of tasks, and not everyone who joined with the initial vision in mind is going to want to stick around or is going to be able to stick around for that new vision. But just like how in the for-profit world sometimes that can be necessary to actually find your product-market fit, ditto in the AI safety world and in the philanthropic startup world: that can be necessary to really find your path to impact. Zershaaneh Qureshi: I guess something that I’m interested in here is how people should decide when to pivot. Like if there are any heuristics for making that kind of decision, or if it’s very kind of instinct-based or something. Max Nadeau: I think ultimately it’s going to depend on your assessment of the impact that you have had and the impact that you can have in the future. And that again relies on the founders, the leadership of these organisations, having really deep models of the risks of AI, and the future of AI development and the present of AI development; and having really in-depth, gears-level takes and opinions about how the work they’re doing is going to actually have a causal influence in the world. And then you can make predictions, and you can look at how your predictions do, and whether or not your models were correct or not, and you can say, like, “OK, this is what I thought would happen. These are the beliefs that I held about the technical research direction we were working on: did they hold up? These were the beliefs I had about the social landscape that were working in, and did they hold up?” And just noticing over time when you had a vision that didn’t actually work out when you got into the details and when you actually tested things out in practice. Zershaaneh Qureshi: Let’s dig in a bit to how founding a new nonprofit compares to other ways of trying to make AI go well. One thing that feels salient here is: if I’m worried that we’re going to get advanced AI systems very soon, and I really want to quite quickly influence how that goes, you might wonder if it makes more sense to go to an AI company instead and do safety projects there on the safety teams that already exist, rather than going through the process of founding my own thing, basically. Does that seem fair to you? Max Nadeau: Yeah. This is a great question, and I think there are a lot of arguments on both sides. There’s a lot of opportunities that are really impactful that can only be done outside of AI companies, and that are currently being neglected right now because so many people are working inside of AI companies. The biggest argument to me for the importance of working outside of AI companies is that we’re in a really brutal, really perilous race to the bottom on safety between many different AI developers. And there’s this really tough pressure that the safety researchers and safety teams at AI companies are under to develop safety techniques that are not very costly to implement and to put in practice; to perform their predeployment evaluations of models very quickly, so that the model can either get deployed internally and start accelerating the company’s research progress, or deployed externally and start making the company money; and to work on problems that are manifesting from the AIs right in front of us, that are happening today — you know, there’s like a lot of fires that are happening at companies that there’s a strong pressure to put out immediately because of this race to the bottom. And I look at that situation and I think, wow, there’s really not enough willingness to pay at these AI companies. They just don’t have the top-down organisational prioritisation and resource allocation that is necessary to make safety go well. And obviously that fluctuates over time, and there are exceptions to this; sometimes there are some teams or some efforts that have all the resources they need. But broadly speaking, we do not have good methods for aligning the AI systems of tomorrow, or even the AI systems of today. And we’re going to need more slack and more willingness to pay — for both research, and then also, just in practice, putting in place costly measures while we’re using AIs that are going to slow down things and reduce revenue and otherwise get in the way in order to prevent large negative externalities from being imposed on the world. And I think that within AI companies there’s a lot of great work that can be done that tries to make the most of the very limited willingness to pay, but that if you want to increase the willingness to pay and release this really harsh pressure of the race to the bottom, I think the best opportunities to do that are outside AI companies. Zershaaneh Qureshi: Yeah, this makes sense. If people are particularly concerned about making sure that the work that they do doesn’t get obsoleted by an AI company’s safety team who’s already working on the problem and has way more resources to make a difference, on that basis, what types of projects would you suggest that they prioritise or deprioritise? Max Nadeau: Yeah, I think that is a really important consideration that organisations in the third-party space should be taking very seriously. I think the main thing that it implies is that there’s a lot less impact to be had in developing new prosaic — or empirical, or call it what you will — techniques for making AI safer. Over the last couple years, we’ve seen some progress on some AI safety issues — for example, it’s a lot harder to jailbreak models than it used to be — and I think very little of that progress has been attributable to researchers outside of AI companies. And I think there’s a lot of reasons for that, but the biggest one is that when you’re outside AI companies and you’re trying to develop these new techniques — for example, defending against jailbreaks — you just don’t have access to the details of the constraints and the costs that your techniques are going to actually have to be imposed under in practice in these AI companies. And also, you don’t have access to a bunch of really valuable resources for doing this work, like lots of data that AI companies have access to. However, I think there are lots and lots of types of AI safety work and niches that organisations can fill outside of AI companies that really cannot be obsoleted by work happening within AI companies, and that are basically immune to the concern that you’re raising. One is work that just provides more evidence and more analysis and assessment of what the state of play is on these AI risks. That’s work that AI companies can’t really do — because they’re not credible voices on it, and because it often involves talking about multiple different AIs from multiple different companies and comparing them against each other. It’s also work that, in practice, I think these AI companies are not doing effectively. I think you would have gotten a very different sense of the state of AI alignment if you just read system cards than if you were reading work like the risk assessment on misalignment that METR put out recently. And I think that you’d be much less surprised by the incidents of misalignment, like the OpenAI Hugging Face incident, if you had been reading the METR work than if you’d just been reading the system cards. So that sort of work is just not going to be done, it isn’t getting done by AI companies. It has to happen outside. There’s other work that could happen in AI companies, but isn’t, for just more contingent reasons, I think. There are types of AI safety research that are more speculative and more ambitious and more principled that are just not happening in AI companies. And those are areas of work that totally could be obsoleted by the alignment work that’s happening within AI companies — we can’t rule that out — but I think that’s really true of any work on alignment. You know, maybe whatever technique you’re developing won’t end up mattering, because other alignment techniques were good enough. Or the opposite: maybe your technique just wasn’t good enough, because nothing was good enough anyway. I think with these sorts of speculative problems, there’s always a risk that the work you’re doing ends up not really fitting in and not really making an impact, and that’s just par for the course. Zershaaneh Qureshi: Yeah. I guess this is part of the difficulty of trying to do some really big stuff in potentially not very much time: you kind of have to take some swings under the knowledge that some of these things will actually not have ended up moving the needle that much further forward, because maybe somebody else does a better swing than you and gets there first or something. But ultimately, you kind of need to throw the whole kitchen sink at things if you want to prepare for a world with these ultra-powerful AI systems. Max Nadeau: Yeah, totally. And slightly contrary to that, I think there are lots of really important projects and startups that, when you look at them, you’re like, “Well, obviously someone should be doing that. Ideally five people should be doing that.” But unfortunately nobody is doing it. So I think for a lot of these things, the bigger risk is not that somebody else takes a better swing at it and your work is obsoleted, but that no one even tries. So that’s why we’re trying to bring more founders, more talent into the field to work on some of these problems. Zershaaneh Qureshi: Yeah, totally. Something I’m interested in here is whether you think that Anthropic is basically swallowing the field at the moment in terms of absorbing all of the talent and the funding and the influence and so on — whether you think that, and also, if so, how good or bad that is for AI safety overall. Max Nadeau: Yeah, I’ll focus primarily on the talent one, because that is, I think, the main bottleneck that a lot of efforts outside of AGI companies face. It’s definitely true that Anthropic has been hiring very rapidly and they’ve been pulling people away from lots of different organisations and institutions across the AI safety field. One thing I would say is that even for people who are working inside AI companies, and inside Anthropic in particular, there are often lots of opportunities for them to be having more impact than they are in their current role and on their current trajectory. I think a lot of people are just not taking those opportunities, not pursuing those higher-leverage opportunities, even within AI companies. I think it’d be even better for many of those people to leave Anthropic — and maybe other AI companies, although I think it’s a more complicated case — to pursue things in the third-party ecosystem for all the reasons that we’ve been discussing in this podcast. But I think the fact that a lot of these people, even within Anthropic, are not necessarily pursuing the highest-impact roles or projects or opportunities that they could be just goes to show that in a lot of cases the people working there are not primarily motivated by trying to have as much impact as they can. And, you know, I don’t think that’s objectively wrong. And I think also many people will say that when you ask them. But I think it’s an important fact about the world, because when people who are new to the field or who are thinking about what they want to do to have the most impact see that there are so many people working on AI safety at Anthropic, it’s easy to assume that’s because they’ve all thought really hard about what the highest-impact thing for them to do is, and decided that their current role is the highest-impact thing. And that is true in some cases, but I think it’s not true in a bunch of other cases too. So if you’re very morally ambitious, and you’re trying to have as much impact as you can, I think those are not necessarily the role models that you should be looking towards for what it looks like to try and have as much impact as you can. Zershaaneh Qureshi: Yeah, that makes sense. One kind of dynamic that’s emerging with Anthropic is that Anthropic is going for an IPO — which means that employees will soon, it seems, be able to sell their shares, and will probably end up in a situation where there are some quite rich people who are at least somewhat aligned with AI safety goals. I’m wondering, in this kind of landscape, whether you think that people should maybe consider working at a company like Anthropic for now and then start new organisations later with their IPO money. What do you think about that? Max Nadeau: Like they would be self-funding these new organisations that they’d be working at? Zershaaneh Qureshi: Yeah. What do you think of that? Is that crazy? Max Nadeau: I don’t think it’s crazy. I mean, I could see that style of reasoning being a bit of a crutch for people who are sort of looking for an excuse to just go make a bunch of money. I think this middle ground where you plan to self-fund your own effort primarily makes sense if you have really strong reason to believe that you’re not going to be able to get adequate funding from other sources, and we’re trying really hard at Coefficient to support a wide range of people doing a wide range of efforts. And also, a lot of the other funders who are coming into this space are doing the same, and have different beliefs from us about which stuff is most important to support, and in many cases are funding things that we don’t think is worthwhile. But I think that’s great. I mean, that’s what you want from a diverse funding ecosystem. Zershaaneh Qureshi: Got it. So I think that there are a few various different reasons why people might want to work at an AI company: One is the motivation to shape how the trajectory of AI is going in a quite direct and fast way. One is wanting to make lots of money. And then one, which is related to making money, is that some people seem to want to work at AI companies out of fear of something like a permanent underclass — getting left behind if everything gets very automated and there’s really serious job disruption, and wanting to escape this permanent underclass. We talked a little bit about these motivations in a previous episode with Jasmine Sun. I’m wondering whether you think this fear of a permanent underclass feels like a fair reason to go down the AI company route rather than founding something new, because it’s a way you could potentially shape things for the better while also kind of securing yourself a much more comfortable position or something. What do you think about that? Max Nadeau: I think that the concerns about being in a permanent underclass… That term means very different things to different people. For a lot of people, I think it primarily just means that they want to get rich, and they’re not really thinking through it; that’s sort of all there is to it. And that’s a very familiar debate: how important is it to get rich versus to have impact or do other things in one’s life? So I don’t think I have anything new to contribute to that conversation. For people who have some view that is more specific to the AI situation, like the one that you were describing — about maybe there’s some limited period in which your labour has any value to the world, and so it’s even higher stakes now — I would break down the argument in a couple of different ways. One world is like, maybe in the future there will be a lot less value in human labour to the economy, but there will still be a lot of value in having capital — and having capital now or having capital then will be able to translate into having the sort of consumption that people are looking for. In that situation, the amount of capital that you’re going to have working outside AI companies and working inside companies is different. But you know, in like the distribution of American incomes, let alone world incomes, it’s really small. Zershaaneh Qureshi: OK, to move on a little bit. Some people who are interested in founding might be thinking about whether they should be going down the nonprofit route or the for-profit route. I’m interested in what you think the main differences between those two worlds are for them to be aware of. And I think in particular, I want to know if you think that you can be as ambitious at a nonprofit as you can for a for-profit, given that there’s more limitations in terms of the amount of capital that you can raise in a nonprofit environment. Max Nadeau: Yeah, it’s a great question. It’s one that we hear a lot. On the question of how much capital you can raise, I think it depends obviously on what sort of work you want to do how much capital you can raise. So if you’re trying to do work that doesn’t really make sense as a product, it doesn’t have a clear paying customer, then you’re not going to be able to raise a lot of capital from the venture capital world. Yes, you can get money from VCs if you’re a talented person, but it imposes a really major restriction on the sorts of projects that you can undertake at your organisation. And I think that for a lot of the most impactful projects, it’s valuable to pursue them as a nonprofit — primarily because it allows you to not have to have these conversations with VCs who are not going to be interested in the primary mission of your organisation, and are going to have their own motivations for wanting to fund you or wanting to support you, which is that they want you to be making money. Also, I would just say that in absolute terms, you can get a lot of money for ambitious nonprofits. As you said at the start of the podcast, there’s up to $200 million in funding for new organisations through Project Tailwind. And in future years, I predict we’re going to end up making grants that are even bigger than that in AI safety. Zershaaneh Qureshi: I’m actually curious about whether, based on the evidence that you can see so far, it’s true that the nonprofits in this space are being less ambitious than the for-profits, or if your evidence points in a different direction. Max Nadeau: From my perspective, I see the nonprofits in the AI safety space as being much more ambitious with respect to impact and with respect to the problems they’re tackling than the for-profits. When you’re doing a for-profit, there’s an inclination or a tendency to focus on problems that are already better understood — you know, that have customers who are already dealing with them and who are therefore interested in paying money for them; problems that are already well understood by the venture capitalists who are funding the work — and problems that affect people with money to pay for them, rather than affecting people more broadly than that. I think that the nonprofits can stay really focused on what they see as the biggest risks from AI and the biggest opportunities to have impact using AI or on AI just in general, the biggest opportunities to solve problems and make progress in the AI safety space. That means that they can focus on issues that haven’t manifested yet, because they’re funded by impact-motivated funders who are happy with them doing things that are more speculative. So just to give an example of what I have in mind, there’s a lot of problems we’re seeing crop up with AI agents today that are attracting a lot of new AI startups to help solve them. These are problems like, “My AI agent is misbehaving in ways that I didn’t want,” or, “I’m worried about my AI agent getting prompt injected” or something like that. These are definitely real problems, and they’re problems that are causing issues for people today, but the size of these problems in dollar terms or in terms of the amount of suffering or joy that is on the table is just really, really small in comparison to other risks from AI that have not happened and that are still speculative and hypothetical — risks ranging from the risk that people could use AI to release pandemics on society or that the AIs themselves could lose human control and potentially wipe us out or outcompete us in some way. And those sorts of problems, it’s hard to raise money to work on them from VCs — because what’s the product? Who’s the customer? But for people who are really being ambitious about tackling the world’s biggest problems, those are the sorts of issues that you’re going to want to work on. Zershaaneh Qureshi: Before we move on, are there any other big differences between the for-profit world and the nonprofit world that you think people need to be aware of? Max Nadeau: I think one difference, which is actually becoming smaller over time, is the funding landscape. Obviously if you’re doing an AI startup in San Francisco, there are a bazillion different venture capitalists that you can work with as a founder. And historically, people have been under the impression that if you’re doing a nonprofit in the AI safety space, and you want to be spending lots of money per year — you know, tens of millions of dollars or even more than that — that really you only have one option, which is Coefficient Giving. I’m happy to say that that’s really not true anymore. There are other big funders that have come into the space, groups like Macroscopic and Astralis, and one I saw recently, Lightcone Commons. And then I think there’s a lot more money coming into the space — probably, although it’s kind of too early to say — from the OpenAI Foundation and from individual Anthropic donors. And I think a lot of those funders are going to be interested in supporting nonprofits in the AI safety space at very ambitious scales and at high levels of expenditure. Zershaaneh Qureshi: OK, we’ve talked a bit about why people maybe should found one of these new nonprofits, and what kind of person ought to do this. But on the flip side to that, I’m wondering if there are any people you’ve come across who are interested in founding but you would actually advise against them doing it? Max Nadeau: So there are a lot of people who are coming into the AI safety field and who may have some preliminary interest in founding things. I think that the main reasons that people should think again about that are just that it requires a different and complex set of skill sets from other roles in the space, and they may not realise that the skill set that is required is different from what they might be expecting. So you have to be willing to both think in a lot of depth about the vision for your organisation — which will generally require being really in the weeds about the technical details of the work you’re doing — and also the social, or geopolitical even, objectives that you’re trying to accomplish. It also requires being willing to shoulder a lot of responsibility and take on the duties of making sure a big organisation is functioning well, and being willing to manage other people and to make tough decisions. I’ve spoken to people who initially think that they want to do that, and then they just spend more time thinking about what that will actually entail and decide that really their heart is in IC [individual contributor] work being a researcher. And I think there’s a lot of impact to be had and a lot of impact that has been had by researchers too. Zershaaneh Qureshi: Yeah, that seems quite fair. Something that I worry about when we’re trying to grab the attention of people who would want to found technical AI safety organisations, I sometimes wonder whether at least some of the skills that are valuable in these kinds of roles are a little bit in tension with or anticorrelated with the personality types who are likely to be most interested in founding. I don’t know if you find this, but something I’m thinking about here is founders probably in this space need to be extremely thoughtful, and often cautious and considerate about things — and possibly the people who are this extremely thoughtful are maybe less likely to be very confident in their abilities, as seeing themselves as a leader and a visionary and stuff like that. Because I think sometimes with that thoughtfulness you get a fair amount of self-doubt and thinking things through a lot more and overthinking and so on. Is this a tradeoff that you see, or is it an issue or a gap at all? Max Nadeau: I think it probably is a tradeoff, and I think we don’t really have enough data points to say confidently, but in the cases I’ve seen, I agree that those things can sometimes be in tension with each other. I think there’s kind of no way around it, other than being comfortable trying things that will probably fail, and being willing to accept when you did something that was wrong, and try it again or do something different. And being honest with yourself so that you notice when something failed, because that’s evidence that you won’t necessarily get from the world. What I would say is: in business, there’s also this need to be comfortable with failure and be comfortable with viewing your past efforts or your current efforts with some amount of honest scepticism and doubt, and saying, like, “I don’t know whether this is going to work; it might fail.” I think that that doesn’t necessarily prevent people from being confident and assured that they’re doing the best they can and just being comfortable in the course of action that they’ve chosen in business, so I think the same can hold in more impact-motivated work too. Zershaaneh Qureshi: Yeah, got it. I want to get a bit more specific now. What are the organisations that you actually want to see being founded in technical AI safety? I know that you have a very long list of 40 different ideas, but maybe you could start here with a quick overview of the different types of work these nonprofits would be doing? Max Nadeau: One really important one, maybe my personal favourite, is new organisations doing independent auditing and assessment of AI companies and their safety practices. I think that’s something that we’ve really seen the importance of recently in the Hugging Face incident and the Mythos UK AISI incident — in which AIs that were deployed in the real world acted in ways that were definitely not in the interests of, or not at the wishes of, the people who had deployed them, and exhibited types of misbehaviour that had not been caught ahead of time. But also there are broader and more systematic ways of assessing the safety of AI companies that involve looking all across the training runs, looking across the safety procedures that they’re using internally, and forming independent, disinterested views on whether or not these safety practices are adequate and how much risk they’re imposing on the world. So that’s one category. A second category is new research centres making progress on new techniques for aligning and making safe, powerful AI models. One example is that we think it would be valuable for there to be more work on understanding chain-of-thought monitorability: to what extent can we use chains of thought to oversee AI systems, to catch them when they’re behaving in ways that we don’t like? How can we train AIs so that their chains of thought are more easily monitorable? What’s the status of chain-of-thought monitorability right now in different AI companies, and what’s the nature or the flavour of the ways in which chains of thought cannot be useful? Which I think we’ve already seen differ a lot between different AI companies. A third category of organisation that we’re excited to see founded is work on better evidence generation. Right now it’s very hard to understand what the state of AI is — how capable the models are, how risky they are. And we think that a lot of the times, some of the best evidence about these questions for informing the most informed people in the world even could go unnoticed or underappreciated — either because it relies on running lots of experiments that are expensive and logistically complicated to run, or because things just happen that nobody knows. For example, the misalignment incidents inside OpenAI were only detected by Hugging Face, which was a third party outside of OpenAI. That just goes to show that some of the most important evidence-generation work doesn’t need to happen inside AI companies. A fourth category of nonprofit that we’re excited about people founding is new organisations doing technical research related to security and verification. There’s a lot of different threats that this can be relevant to. One is supply chain vulnerabilities in AI — which can include things as high up as issues in the hardware that AI is trained on, but can also include poisoning of the training datasets, either by a malicious AI or by a bad human actor, that the companies developing AIs wouldn’t even be aware of. So having good security methods to make sure that we understand that the supply chains — that go all the way from the initial inputs to the AI to the inference happening at the end of the day — are secured and are as we expect, there’s a lot of important technical problems to be solved there. A fifth category of organisation that we’re excited about founders starting is public goods for the field of AI safety. The MATS Program, which centralises fellowships for AI safety organisations, is a really great example of how much impact can be had by leveraging economies of scale and helping out many organisations at once by putting things under one roof. And we think there are other opportunities to take that model and apply it to other public goods for the AI safety field. One potential example of this might be a new compute cluster that’s intended to support AI safety research. Then the sixth category I’ll mention — and you should go look at the project list on the website if you want to see even more — is new organisations in the AI safety fieldbuilding space. AI is really in the news; it’s a kitchen table conversation topic now. And there’s a lot of people all over the world who are getting interested in the risks from AI and want to use their careers to work on it. We think it’s really important to help those people get integrated into the field, and to do matchmaking between them and the organisations, so that they can leverage their skill sets most effectively and so that they can get up to speed on what’s already going on in the field and how they can help. Zershaaneh Qureshi: Yeah. As you lay these things out, obviously all of these things do sound important to me, but some of them do feel a little bit dense. If you imagine that I am a potential founder, can you sell it to me a little bit? What do you think would be exciting about founding an organisation that’s working on monitoring chain of thought or something like that? Max Nadeau: I mean, I think different organisations are going to appeal to different people, so not everything is going to be exciting to everyone. If you’re the sort of person who wants to solve really hard technical problems, then the organisations that I mentioned concerning security and verification and concerning new alignment techniques, that’s going to be a more exciting pitch to you. If you’re somebody who wants to be really scrutinising and looking carefully at all the claims that AI companies are making about their safety practices, and applying a sort of journalistic ethos to that, then the third-party auditing, third-party evaluation orgs might be more appealing. Maybe the throughline that’s part of the pitch here is that there’s an opportunity to be a really influential, independent voice in the AI safety world, and in the AI world, and in the world more broadly. There’s a lot of demand from politicians and from journalists for people who are credible, who are independent from the powers that be, and who can speak to these really important issues that everyone is waking up to — especially around AI misalignment — and who understand what’s going on and can explain that to important decision-makers in the world. Zershaaneh Qureshi: Yeah, awesome. Very inspiring stuff. For listeners who are interested in founding one of these technical AI safety nonprofits, what concretely should they be doing now to get the best shot possible of winning a grant? Max Nadeau: Before you reach out to us, it’s definitely valuable to spend time thinking through all the different organisations that you might want to found, and come up with a vision that you’ve thought through in depth that you are a great fit for, and that you think we would be excited about. And that can be something taken from the list on our Tailwind website, but we think that we’re going to make a lot of grants to people who have their own ideas that we didn’t write down. So developing your vision for what the startup will do, that’s a really important step. And then once you want to reach out to us, you can go to cg.org/tailwind, and there will be a form you can fill out. There’s going to be a lot of questions, most of them are going to be optional. Also, if you’re listening to this and you personally don’t want to found an organisation through Tailwind, but you know someone who you think would be a great fit for one of the sorts of organisations that we’ve spoken about here, we would love for you to either tell them to express their interest through our website or you can just tell us directly. You can just shoot us a very quick email with just this person’s name and we can reach out to them. The email address to do that will be in the description of this podcast. I don’t have it, unfortunately. Zershaaneh Qureshi: Yeah, we can stick that up. I think that’s all we have time for. But Max, you’ve been a pleasure to have on the show. Thanks so much for coming. Max Nadeau: Thanks so much for having me.