Latest / 80,000 Hours Podcast / #153 – Elie Hassenfeld on 2 big picture critiques of GiveWell's approach, and 6 lessons from their recent work

Transcript

#153 – Elie Hassenfeld on 2 big picture critiques of GiveWell's approach, and 6 lessons from their recent work · 80,000 Hours Podcast

• Rob Wiblin: Hi listeners, this is The 80,000 Hours Podcast, where we have unusually in-depth conversations about the world’s most pressing problems, what you can do to solve them, and getting to know the man behind the cost-effectiveness analysis spreadsheet. I’m Rob Wiblin, Head of Research at 80,000 Hours. It’s a bit crazy that we’ve never done an interview with the cofounder of GiveWell, Elie Hassenfeld, but I’m glad we’re fixing that with this episode. Many of you will be familiar with the idea behind the charity evaluator GiveWell. Their slogan is “we search for the charities that save or improve lives the most per dollar.” But if you haven’t paid close attention, you might not realise how much they’ve grown and changed in recent years. For a few years they kept the Against Malaria Foundation as their top-recommended charity, and for some became almost synonymous with insecticide-treated bed nets. But they now have research into many more interventions; have been looking at approaches that have weaker evidence, where you don’t expect to be able to prove they work; and have made grants to dozens of organisations, including RESET Alcohol, Nutrition International, Sightsavers, and Alliance for International Medical Action — to name just a few that I’d never heard of. This interview is split into two very different parts — the first more empirical, and the second more theoretical — and I want to highlight that because I could see different people being interested in each of them. For the first half of the conversation, Elie and I go through six interventions and organisations that GiveWell has investigated and made grants to in recent years. He explains what generalisable lessons they learned in the process of researching them. Those six organisations are: Dispensers for Safe Water from Evidence Action Syphilis screening and treatment in pregnancy programme from Evidence Action Kangaroo Mother Care helping premature infants survive MiracleFeet, which treats clubfoot Alliance for International Medical Action, which treats severe malnourishment And Helen Keller International’s vitamin A supplementation work The second half of the conversation challenges GiveWell’s approach at a higher level. I put to Elie two different critiques of GiveWell’s approach that I’ve seen repeatedly over the years and have some sympathy for. The first is that GiveWell should use increases in subjective wellbeing as an outcome measure, rather than lives saved or doublings of income. Some people argue that this approach is more theoretically sound and would lead to substantially different recommendations. The second is that, in the long run, what really helps people escape easily prevented diseases and the suffering of poverty is sustained economic growth. And that therefore GiveWell should focus on recommending organisations that increase economic growth rates in poor countries, and that that’s going to look very different from the sorts of global health projects GiveWell usually supports. If you’d rather listen to that second half of the conversation, you can skip forward to around 1h45m — or if your podcasting app has chapters, you can skip forward to the chapter called “The subjective wellbeing approach in contrast with GiveWell’s approach.” All right, without further ado, I bring you Elie Hassenfeld. Today I’m speaking with Elie Hassenfeld. Elie is the cofounder and chief executive officer of GiveWell. There, he oversees much of what GiveWell does, but has a particular emphasis on setting strategy. GiveWell, if you don’t know, is among the most well-known charity evaluators in the world, and it aims to find high-impact, cost-effective charities backed by evidence and analysis. In 2021, they think that they moved around $600 million to charities that they either recommended to the public or directly made grants to, or suggested grants to other organisations. Before that, Elie worked in the hedge fund industry, which he left to cofound GiveWell in 2007, along with Holden Karnofsky. And before that, he was at Columbia University, majoring in, of all things, religion. Thanks for coming on the podcast, Elie. Elie Hassenfeld: Thanks for having me. I’m a big fan of the show, so really excited to be here. Rob Wiblin: I hope to talk about a whole host of new charities that you’ve made grants to over the last few years, and whether you should maybe measure the impact of programmes in units of subjective wellbeing. But first, what are you working on at the moment, and why do you think it’s important? Elie Hassenfeld: Two big things. The first is recruiting. We’re aiming to grow GiveWell — both on the research side and across the entire organisation — so we can do more and do it better. And I’m very focused on that. And then I’m also focused on fundraising. We’ve found a huge number of outstanding opportunities that we want to direct funding to, and we don’t have enough funding to fill all of them. So we’re very focused on trying to increase the amount of money we raise and direct. Rob Wiblin: It’s been a couple of years since we last spoke to someone at GiveWell on the show. Maybe you could give us a little bit of a description of how things have changed over the last couple of years. I think it’s been a period of a lot going on. Elie Hassenfeld: Yeah. The short story is a period of major growth. We now are raising approximately $600 million a year, and 60 staff work at GiveWell, and that’s enabled us to do much broader grantmaking. I think in the popular mind, GiveWell is sometimes conceived as top charities — or even, in the simplest version, just equivalent to the Against Malaria Foundation, which is funny. But we’ve broadened out a significant amount, directing more than $100 million to non-top charities over the last couple of years. We’re directing that annually in areas like malnutrition, water, policy-related grants, and research. So that’s been a major expansion in the work we’re able to do. Rob Wiblin: Yeah, it seems like people’s perceptions of GiveWell might be kind of trailing six or seven years behind — because I think that long ago, most of the money that you were moving was by recommending things to the public, right? I guess something like the Against Malaria Foundation was pretty prominent in your recommendations. It’s just that it’s a much bigger operation these days. Elie Hassenfeld: That’s right. I mean, it’s still the case that 75% or so of the funds we direct go to top charities. I think for a long time GiveWell’s list of recommendations stayed fairly static — and that static memory has remained in people’s minds. Rob Wiblin: Yeah. How many organisations do you think you’ve made grants to or recommended to the public over the last 12 months? Elie Hassenfeld: Oh gosh, I would say upwards of 20 — but you’d have to check the website to be sure. Rob Wiblin: In 2021, I think you moved around $600 million. Do you have a sense of how that’s going to change over the next couple of years? There has kind of been a decrease in wealth since 2021. I suppose the picture might be a little bit more difficult in the immediate term? Elie Hassenfeld: Yeah. So for 2022, data is still coming in. One of the things we do to get complete data is go to the charities we recommend and ask them about donations they received as a direct result of our recommendation. That takes us a few months, and we usually release that data in the summer. But our sense is that 2022 came in right around where 2021 was — which we see as a success, given some of the challenges in the financial markets. When we look forward, we have, unsurprisingly, a wide range of uncertainty about what the future will hold. I think there’s some possibility that over the next few years we’ll grow quite quickly, and reach $700, $800, or $900 million a year. There’s also a possibility that growth may stagnate and we’ll be moving about the same amount in the future. We’re obviously working very hard to increase that number as much as we can, but with the uncertainty in the financial markets, we don’t know what the future holds. Rob Wiblin: You’re directing funding to a lot more organisations than you were seven years ago. Someone might wonder, why is it so many different projects and so many different organisations? Because you might think you should find the most cost-effective one, or the most cost-effective handful, and then just throw as much as you can at those ones. What’s the reason for spreading out? Elie Hassenfeld: Yeah. And we agree with that. We want to direct marginal dollars to the place where marginal cost effectiveness is highest. If we believed that there was one organisation that would deliver the highest impact per dollar, we would just fund one organisation, even if that’d be pretty boring. We’re willing to be boring, I guess. The reason we’ve expanded is that now, if we were to direct the marginal dollar to one of our top charities, we think it would be less cost effective than giving it to some of these other organisations. Just to make that concrete, we talk about cost effectiveness in multiples of just giving cash: just giving someone cash would be 1x. Our current bar is 10x. If you want to go deeper into what all this means and how it works, it’s obviously not quite as simple as that. But our bar is 10x — and so the next dollar beyond what we would otherwise try to fund to our top charities might be 9x or 8x, and we’re finding opportunities that are better than that in some of these newer programmes. And that’s the reason for expansion. Rob Wiblin: Maybe the first thing that you fund — the thing that has the greatest multiple on just giving cash — is it that, by the time you were trying to give them $600 million, basically they’ve run out of ways to spend that meaningfully? Or that they wouldn’t be able to grow that quickly? Or maybe they’d have to operate in a country where the thing they’re dealing with is less of a problem? Elie Hassenfeld: I think the last one is the most common. Let’s just use vitamin A supplementation as an example. This is delivering vitamin A to young children twice a year to reduce child mortality. There’s great evidence for this. Some countries have much higher rates of vitamin A deficiency than others. If you deliver it in the first set of countries that have high rates of vitamin A deficiency, this programme will be extremely cost effective. But as you go down the list of countries — towards countries with lower vitamin A deficiency, or lower rates of child mortality, or higher costs to deliver the supplements — it’s less cost effective. There are places where these programmes are very cost effective, and then other places where, frankly, there’s still amazing programmes to support, but we think we can do better at the margin, and so we try to. Rob Wiblin: Yeah, that makes sense. We’ll come back to a bunch of those themes throughout the interview, but just before we dive in, I wanted to mention for listeners that there’s kind of three different big ways that this conversation connects to pressing problems and doing good with your career. The first is that plenty of listeners are trying to do good by giving money in high-impact ways. Obviously that’s super relevant, because GiveWell is exactly trying to advise people on how to do that best within a particular area. The second thing is that GiveWell is trying to do research to figure out how to do the most good for people in poverty, or people who are suffering in the world’s poorest countries. In the process of doing that, it investigates all kinds of different sub-problems related to poverty, and many different possible approaches and methods that one might take to reduce those problems. I think that the practical and methodological lessons that come out of all of that work are obviously very relevant to people who are working on global health and development anywhere else. But I suspect that they can also be pretty instructive for people who are working on other problems — maybe especially those who are working in areas with very poor feedback and little data that they can collect, who are going to find it hard to learn all that much from their own work because it’s just not possible in that case. And the third reason is that GiveWell itself is looking to hire researchers and operations staff and fundraisers and so on, roles which people in the audience might be able to fill. I think we’ll talk about that at the very end, but maybe at this point, could you briefly list a couple of the different roles that GiveWell is hiring for at the moment? Elie Hassenfeld: Yeah, let me just list a couple, and by the time the show actually airs, there may be new roles, because we’re aiming to grow quickly. But at a high level, we’re looking for senior researchers — these are the folks who really lead GiveWell’s research work across the board. Some of them are focused on more technical questions, like, “What should our moral weight be for subjective wellbeing versus increasing income?” Some of them are programme officers who are making decisions about which organisations to grant money to and which questions about them we most need to look into. That’s on the research side. We’re also trying to hire folks to support operations. Operations makes GiveWell hum, tries to enable us to do as much good as possible across the board. One type of project that we want to improve on — but we need more staff to get there — is doing a better job investing some of the assets that GiveWell holds. We aim not to hold a lot of money, but we’ve grown and we have some funds, and we could do more if we were investing those funds more effectively. Rob Wiblin: Yeah, fantastic. As I said, we’ll come back to that at the very end. For now, let’s push onto the body of the conversation, the torso of the interview, which is about a whole bunch of new nonprofits and interventions that you’ve been looking into and started supporting in the last few years. Basically, it’d be good to go through maybe six or so of these, and find out why you ended up directing funding to them after looking into it, and maybe what transferable lessons or general lessons you learned in the process of investigating them. The first of those that I wanted to bring up — because it seems like it’s actually quite a big deal for you — is your funding of Evidence Action’s Dispensers for Safe Water programme. GiveWell wrote an article on that last year, titled “A major update in our assessment of water quality interventions,” and there you explain that you looked into Dispensers for Safe Water years ago and decided not to fund it. But over 2020 and 2021, you took a second look, and changed your mind, and ultimately recommended a grant of $65 million to them. Let’s start at the beginning. What does Dispensers for Safe Water do? How does the intervention work mechanistically? Elie Hassenfeld: So in many parts of low-income countries, people don’t have access to clean water, and drinking unclean water can lead to diarrhoeal disease, which most importantly leads to death among children under five. This intervention puts a small chlorine dispenser near a water point, so that when someone comes to collect water from a spring, a pipe, they quickly push down on the chlorine dispenser into the jerry can that holds the water. That puts chlorine into the storage container that they then carry home. This intervention is potentially much more effective than other attempts at chlorination in the past, because the individual collecting water only needs to remember to put chlorine in their container one time, right at collection. Also, the chlorine remains effective while the water is in the container once they bring it back home. So it reduces the need, say, for an alternative programme, which would require someone to go to the store to purchase chlorine tablets, or get them from a nonprofit, have them at home, and then use them each time they choose to consume water. And that easier behavioural intervention makes it more effective. Rob Wiblin: Yeah. I’m not sure how much I would sanitise my water if I had to stick something in it every time I poured a glass of water. Sounds super annoying. So that’s how it works. What’s the prima facie or conceptual, high-level case for why this wouldn’t just be good, but it could be amazing and one of the best things for you to fund? Elie Hassenfeld: It’s that unclean water leads to a great deal of mortality in low-income countries. Having a diarrhoeal disease not only can lead directly to death, but can exacerbate malnutrition, which itself is a risk factor for death from other infectious diseases. So it’s a major problem. Then, this is a very low technology type of intervention. It’s very simple. I’ve used it — I’ve visited this in Kenya, and it just requires pushing down on this thing to deliver chlorine. So it’s easy to implement and easy to monitor and follow up on; it’s easy to check. You put that all together, and it’s a fairly low-cost programme that has a direct effect on a major public health problem globally. Rob Wiblin: Yeah. Just so I can picture it in my head, people are getting a big bottle of water from a well, or from a common tap, and then they have to stick a little chlorine tablet in it? Or is it kind of a chlorine spray that they stick in the bottle once they’re done? Elie Hassenfeld: Basically, imagine that the person is carrying a jerry can. This is often a yellow, several-gallon container. They’re bringing that container up to maybe a pipe, or even without a pipe, just a spring that water is flowing from. Right next to that water point, there’s a stand that’s maybe two to three feet high with a little plastic container that holds liquid chlorine. You press down on the pump one time, almost like a soap pump that you’d find in a public bathroom, and out of that pump comes the appropriate amount of chlorine for that jerry can. So it’s just dispensing it directly into the water container. Rob Wiblin: OK, amazing. Can you tell us a short version of the story of how GiveWell looked into these clean water dispensers many years ago, and then ultimately decided not to fund it? Elie Hassenfeld: Yeah. A few years ago, we looked at the evidence for the reduction that chlorination would have on diarrhoeal disease. There’s a fair amount of evidence that demonstrates that chlorination does, in fact, unsurprisingly, reduce diarrhoeal disease. And we put that through our normal process of trying to understand what that would mean for Dispensers for Safe Water in the field: how often we expected people to use it, how often it might break, et cetera. We came to the conclusion, based on that analysis, that this programme was not going to be cost effective enough to be above that bar for funding where we had it at the time. Rob Wiblin: OK, so basically, just given the cost and given how bad diarrhoea was, and given what an impact it made on diarrhoea, it was good — it just wasn’t amazing. It wasn’t good enough. Elie Hassenfeld: Yeah. Just to put some numbers on it, our bar historically has often been around 10 times as cost effective as cash transfers. I think at the time, we estimated that this programme was in the range of three times as cost effective as cash transfers. So in the scheme of things, still a very good programme — effective, helps people a lot — but we thought at the margin we would give money elsewhere, rather than supporting this programme at the time. Rob Wiblin: OK, yeah. So what changed in 2020, 2021? Elie Hassenfeld: There’s two big things that changed: one on the research side, and then also the amount of funding we were directing changed too. I’ll talk about how that affected our decision. On the research side, Michael Kremer — who was awarded the Nobel Prize in Economics — and his colleagues conducted a meta-analysis of the effect of water programmes on mortality. And in general, the trials that had been done for chlorination were using mortality as a secondary endpoint: they weren’t powered to detect a reduction in mortality, only in diarrhoeal disease. But by pooling the studies together, they were able to find an effect on mortality as a whole. And this meta-analysis showed a significantly larger effect on all-cause mortality than we had estimated based on diarrhoeal deaths alone. It is really that that makes the biggest difference: we had seen the mechanism of impact as flowing through diarrheal disease, but when you look at all-cause mortality and you take into account the likelihood that diarrhoeal disease leads to mortality that is not specified in the data as diarrhoea specifically, that caused a pretty big update in our overall estimate of the impact of the programme, and we decided to fund it. Rob Wiblin: These trials of Dispensers for Safe Water that you were aggregating, were these randomised controlled trials where people did or didn’t get the dispensers at random, or were they something else of a more observational style? Elie Hassenfeld: These were randomised controlled trials of chlorination and chlorination plus programmes. One of the reasons that we selected a subset of the trials that had been included in this meta-analysis conducted by Michael Kremer and colleagues is that we were trying to select the trials that were predominantly just chlorination interventions in similar contexts. Some of the other trials included components related to hand washing, and we discarded those from our analysis because we wanted to be focused on ones that we thought would generalise more to the context Dispensers is in. That said, it’s not directly many trials of Dispensers itself. So as always, there’s still a fair amount of uncertainty in how the results will translate. Rob Wiblin: OK, so one thing is you could get a better resolution on the size of these effects by pooling together a bunch of different studies that each individually might not have been quite big enough to tell exactly how large the effect was — at least if the effect wasn’t extremely large. The other odd thing is that previously you were looking at these studies thinking, “This will reduce diarrhoeal disease by X percent, and diarrhoea causes this many deaths, so we should expect diarrhoea deaths to decline by 20%.” But then these studies looked at death from all causes among children, with and without the chlorine dispensers, and found that it was reducing the number of deaths by more than the total number of deaths from diarrhoea — which suggests either that something’s gone wrong [in the studies], or the chlorine dispensers are preventing many deaths that aren’t related to actual diarrhoea, which is a little bit odd. I mean, I feel suspicious, but maybe I just don’t understand what the mechanism of operation would be. Elie Hassenfeld: Yeah, it’s somewhat surprising at first glance, but I think it makes a lot of sense. What’s happening is that children who have diarrhoea are less able to hold on to essential nutrition, and essentially move on that malnutrition spectrum. When children become weaker, they become more susceptible to the whole range of infectious diseases: malaria, pneumonia, and others that are prevalent in low-income countries. So it’s not surprising that a programme that improves health in general would have some effect on deaths that are not directly described as diarrhoeal deaths. I think that case is quite strong. Another potential explanation — that is similar but slightly different — is that the data collection that is aiming to allocate deaths to specific causes is imperfect. We saw some of this in high-income countries with COVID. But we just don’t always know. A child could have had diarrhoea recently, come into a clinic, be suffering from respiratory distress, and that could be categorised as a death from a respiratory infection rather than diarrhoea — even though diarrhoea in that case is much more clearly a relatively proximate cause of death. Rob Wiblin: Yeah. I’m curious to know, could these issues be quite widespread in investigations that you and other groups do of other interventions and other programmes? So each individual study of whatever other interventions finds no effect, but then if you added them all together, you’d find that there was a large effect? Or maybe if they focus on malaria, they look at deaths from malaria — but in fact it’s had a much larger effect on mortality than what’s apparent, because people are attributing deaths incorrectly. Did you worry about that? Elie Hassenfeld: It’s definitely something that we’re very focused on. For a long time, when we’ve looked at malaria data, we’ve focused when we can on all-cause mortality. And of the randomised trials that were done on malaria nets, historically a large number were on malaria rates, but a number were on all-cause mortality as a whole, because of this reason exactly. One of the lessons we took away from this is that, years ago, when we first did this analysis, we insufficiently brought our conclusions to experts outside. I think, had we done that, it’s possible that they would have raised this question in 2019 and we would have more quickly updated because we would have realised that we were too narrowly assessing the impact of the intervention. And that is a change we’ve made with some of the other programmes (which I think we’ll talk about in a minute): we’ve taken our estimates to outsiders and they’ve helped us see a broader picture of what they might be doing so we can home in on the best possible estimate we can. Rob Wiblin: If I recall, another thing that Kremer did (who was the person who did this early aggregation of all of these different studies, and tipped you off that maybe you would want to take another look at this): he had access to a bunch of data that wasn’t entirely public, or maybe some studies that hadn’t come out yet that allowed him to get a larger sample and notice this. Is that a common problem? That studies get done and either the data is not published yet for a long time, or perhaps you don’t have access to the specific numbers that you need from that study in order to aggregate it to get a clearer picture? Elie Hassenfeld: I think it is a pretty common issue. And in many cases, when we go deeper on analysis, we’re doing that via reaching out to authors and getting the underlying data itself, so we can understand what’s happening. We’ve done that a number of times historically. Mostly we’re relying on publicly available data because the time costs involved in trying to track down that data and get more of it are high, relative to just relying on the data that’s already out there. Rob Wiblin: Yeah. Are there any kind of personal or practical barriers to GiveWell overturning a past finding like this? Or maybe is it almost the opposite? That you’re excited to have something new to say, so maybe you’re almost even biased in favour of it? Elie Hassenfeld: I think it’s a positive for us, I’ll say. I don’t think we’re biased in favour of having something new to say. It’s not that we’re particularly excited about saying we were wrong in the past, but GiveWell has built up, honestly, a brand of being transparent, open to mistakes, and changing our mind — whether it’s about past organisations we recommended or research decisions. That’s so strong that I think we’re all somewhat excited about the opportunity to be able to demonstrate publicly that we really hold those values dear. That’s good both for how we communicate with the outside world, and how we build our culture internally. Ultimately, I think we’ve reached the point now where it would be harder to not change our mind than to change our mind, because it’s so much a part of what we are and what we’re trying to do. Rob Wiblin: Yeah, that’s great. I was pottering about in the cost-effectiveness spreadsheet for this one earlier today, and people who are interested should definitely go check it out — it really reveals the amount of work that goes into this. It’s very nice. You can see how you’ve gotten, I think, these five studies that passed your various different criteria, and you can see how you aggregated them all into producing this estimate that it would reduce all-cause child mortality by 15%. You’ve even gone through and you’ve got the population age distribution for all of the different countries where these dispensers are going out. And then you place the value on how bad it is for someone to die at each of these different ages and estimated what effect would it have on the mortality. It’s a lot. Elie Hassenfeld: Yeah, it’s a lot. And I’ll say one of the things that we’d like to improve a lot on is making that information easier to consume for someone like you, and maybe for people like those who are listening to this episode. We ran this contest last fall called the Change Our Mind Contest, where people looked at our work and tried to find mistakes. And they did a great job. But I think people also sometimes struggle to really identify either the most important parameters that, if were wrong, would change our decisions the most, or understand what we were doing. That’s something that we refer to as “legibility” — where we would love to say, “Hey, Rob, if you want to spend an hour poking at our spreadsheet, we want to support you in finding the most important places to poke, rather than finding yourself down a maze of spreadsheets and Google Docs.” Rob Wiblin: Yeah, I think you’re doing reasonably, but it’s challenging to make something totally decipherable to someone who’s really working in a different area. Is there anything else that you think you could have done differently in the past to maybe get the right answer earlier? Elie Hassenfeld: I think that’s the main one. I think the other possibility, which we’re doing more of now, is being quicker to fund additional research in areas that are close to our line, or where additional work could lead to a result. We provided some support to Michael Kremer in his work on this additional research. But when we see areas now where we think additional funding to support researchers who are spending some time collecting some data or running an additional trial could lead us to change our mind, we support it — because we want to get that information back as soon as we can to update our views. Rob Wiblin: Have you learned anything new in the last year or two to give you a sense of what the longer-term prospects are for this one? Elie Hassenfeld: The longer-term prospects here are really interesting. We’ve been talking about Dispensers for Safe Water. There are two other programmes that relate to clean water that we’re very excited about exploring. I don’t know how much we’ll end up supporting them. One is in-line chlorination, which is a device that serves people who get their water through pipes. This is often not a pipe that’s reaching their household, but pipes that go to a water tank, and then they’re collecting water from the water tank. A small chlorination device can be put into the water tank to automatically chlorinate the water as it’s coming through. This is an important change, because Dispensers for Safe Water monitoring shows that only about 50% of the people who live in the communities served have chlorinated water when Dispensers does their backchecks. So it’s very cost effective at a 50% level — if you can improve it, that’s great. The challenge is it’s a newer technology and is more complicated, so it’s more likely to break. So there’s that. Then the final programme is vouchers. The idea here is to give households a coupon to go and get free chlorine at a local store, local market, and then they would use the chlorine in the traditional way in the household to clean their water. The downside of this programme is that it requires that user behaviour, but the benefit is you might think that the people who decide to go to the market and utilise these vouchers are more likely to use the chlorine. It’s also standing on top of existing infrastructure in a country, in the private market that’s getting chlorine out nearer to people. If it’s effective, it can be very cheap. Our plans in the near future involve continuing to follow and support all of these programmes at small levels as we think about which are going to be most cost effective. I think the answer will be some are going to be more cost effective in one context versus another. And then we’re fairly likely to support a large randomised control trial of these interventions — most likely vouchers — to both understand its impact and get an additional data point on mortality. We still have a fair amount of uncertainty about our current estimate of how much chlorination reduces child mortality. Rob Wiblin: Yeah, it makes sense. It seems obviously better if you can just automatically chlorinate the water as it’s coming through. But I suppose technically, that’s a harder challenge to install that, and have it always put out the right amount of chlorine as the water is coming out, and to make sure that it’s always restocked with chlorine. Is that the reason not to do it that way? Elie Hassenfeld: Yeah, it’s more expensive upfront and the technology is newer, so it’s not clear that it works in every location. But certainly if you could, holding those two considerations constant — of course, they’re very big considerations — I mean, that’s the obvious path. Rob Wiblin: I would say it’s unbelievable that only 50% of people chlorinate their water when the chlorination thing is sitting right there. But I have seen what fraction of people — I guess men in this case — wash their hands in public bathrooms after using the bathroom. And 50% sounds about right! Elie Hassenfeld: I’ll just say I think that 50% number is a good anchor to hold in mind for the challenge of improving lives in low-income countries. If you imagine that you have this dispenser, it’s right at the water point, it matters — and still what you’re measuring is 50%, then what you know is this is hard. We have to be focused on getting this right, because there are so many ways that it can go wrong. What I think is particularly great about Evidence Action and its work, and why we support them, is that they’re out there collecting that data so we know what that number is — because normally you would have no idea, and you would just assume 100%. Of course, why not? Rob Wiblin: Yeah. OK, let’s push on and talk about a totally different approach to doing good: improving screening and treatment for syphilis among pregnant women in particular. In 2020, you directed about $4 million to Evidence Action again — this same overarching organisation — in this case, to provide technical assistance to governments in Liberia, Cameroon, and Zambia in order to improve how they do screening for syphilis among pregnant women as part of their public healthcare systems. Can you describe more of the programme? Elie Hassenfeld: Yeah, I’ll give the short story here. Pregnant women are often infected with syphilis in low-income countries, and if a child is born to a mother infected with syphilis, this can lead to stillbirth, neonatal death, or birth defects. It’s easy to treat syphilis with penicillin, but in many low-income countries, clinicians didn’t have a diagnostic to test for syphilis. Recently, the World Health Organization approved a new diagnostic that combined an HIV test with a syphilis test. This was very effective, because many people are coming during pregnancy to be tested for HIV. So Evidence Action supported first the government of Liberia in switching over to this dual test: they had to help them procure the dual test, ensure that the test reached the clinics, train clinicians to utilise them, ensure a supply of penicillin in the clinics, and then develop a monitoring system to know whether or not this was actually happening in practice. We supported the Liberia programme several years ago, and then more recently Cameroon and Zambia, based on initial positive results from the programme there. Rob Wiblin: So it’s really bad for pregnant women to continue to have syphilis during pregnancy: it’s bad for them; it’s bad for the child. And there was this new test that combines the HIV test that was already being delivered with syphilis. Does this combined test not cost more? It’s not more difficult to deliver? Basically, you get a two for one? Elie Hassenfeld: That’s my understanding. I’m not sure of the exact cost differential, but it’s fairly straightforward in that way, as easy to use and roughly the same cost. Rob Wiblin: And here the technical assistance was talking directly to hospitals or medical providers in order to convince them to switch over to this combined test? Elie Hassenfeld: It starts with working with the government and ensuring that the Ministry of Health in Liberia is ready and excited for this support. And then supporting their work in procuring this new test, to make sure they get it and can utilise it, and then supporting the whole programme along the way — this is ensuring that clinicians are trained to use it and know how to use it, ensuring that penicillin is in stock. So generally it’s coming to the government and not only convincing them that this is a good programme, but also saying, “We can help get you there, and we will provide the support to ensure that this programme actually goes the way that you ideally want it to.” Rob Wiblin: So as you mentioned, you compare almost everything to just giving people cash. How much better than that do you hope this programme might be? Elie Hassenfeld: It varies by context, but we see this programme as being somewhere between 10 times and 30 times as cost effective as cash. The reason why this programme is potentially so cost effective is two parts. Number one, it largely relies on what we call “technical assistance,” which is providing support to the government in its delivery of a programme. So the programme is not purchasing the commodities itself; it’s not directly paying for distribution. It’s ensuring that the existing funds that the government is already spending are going into more cost-effective uses. And then second, we assume that there’s a decent probability that eventually the government of Liberia, say, is able to take over this programme with significantly less support from any outside funder. And we think that this programme will be more likely than average to go on, over the long run, owned by the government. Rob Wiblin: Doing the technical assistance thing — just encouraging a country to improve how it provides healthcare — I suppose you get big leverage because you’re not having to pay for all of it. Or in this case, it might not even really be more expensive on an ongoing basis. You maybe only have to support them in doing the switch temporarily, and then you get to have the gains for many years after that. Is it surprising at all that a country needs external people to come in and tell them to do something like this? From one point of view, it’s very obvious that they should make this switch, and it’s not involving anything that’s particularly high tech. You might expect that perhaps hospitals just review and improve their practices automatically, without people coming in from overseas to do this. Elie Hassenfeld: I don’t think it’s that surprising to me. I think there’s many steps in the chain that all have to go right for this to work. Someone has to decide to procure this new test. They need to go out and do it, then they need to ensure that everyone across the entire country is trained to utilise the new test. There’s an additional step of ensuring that this drug that otherwise would maybe not be in stock, is in stock. When I think about that, I guess I try to look at it in the examples that are closest to me. I think about GiveWell as a very small institution that is nowhere near the size of the government of Liberia or the Ministry of Health of Liberia. There’s so many times when someone has an idea that is prima facie good, but is hard to execute on — because it’s hard to move through all the steps that have to happen to make it take place. But when someone shows up and says, “Not only is this a good idea, but I’ll ensure that all the steps happen to get it to the finish line,” and you have confidence in that person — this is now thinking of the GiveWell context — that’s a very easy decision to make. I think in that context it actually makes a lot of sense to me. And I think this whole area that we call technical assistance is essentially policy-oriented intervention, and offers high leverage and I think fairly high potential success. Rob Wiblin: OK, then I suppose I can almost flip it around: If we think at a high level that technical assistance could be really highly leveraged — and there’s maybe many ways that medical treatment in poor countries, and indeed probably rich countries as well, could be much better, and maybe not even ultimately more expensive — then that creates a big search space, where you could look at all of these different countries and all of their different practices and see what’s the biggest blunder, the biggest unforced error that someone’s making, where technical assistance could pack a big punch. Is that something that you or other people have done? Elie Hassenfeld: It’s definitely something we’re thinking about. We see technical assistance or policy as a very high leverage point, and so we want to understand it. There are big constraints on the other side: it’s not enough to see something that would be good; it also has to be something that the institution on the other side — whether it’s the government or whoever else — will be excited to action. And there could be other reasons why it’s hard to make policy change. I think we often can relate to this most easily when we think about our local communities or countries, where we know policy change is hard even when ideas seem obviously good. We are searching in that space, and the types of things we’re finding are either like syphilis — meaning highly, highly technocratic, and sort of boring, clear wins to improve people’s lives. Or sometimes they’re driven by government priorities, where the government says, “This is something that we care about” — and then it’s usually not GiveWell, to be honest; it’s the groups that we support, who have these conversations with government, who can come to us and say, “We have an opportunity to make a big difference here. We’d like your support in making that happen.” Rob Wiblin: Yeah. This is maybe one of your earlier technical assistance grants. Do you think you’re going to learn much about the mechanism here by seeing whether, say, you do get uptake in the change? Or will you learn anything else from making the grant? Elie Hassenfeld: I think we’re going to learn a tonne. When we were going through this investigation — I think it was three years ago that we were kind of in the early stages — we were wondering about and had no real decent estimate of the likelihood of success for so many lines in the chain of logic for this to be effective. Would the Liberian government decide to go along with this? Would they want to do this, or would they just say no? (And that was a possibility.) Would they eventually procure these tests even if they said they would? Would we get decent evidence that people were using them? A lot of what we’ve seen so far in this case is generally positive results. Obviously some that are not — happy to go into details if you want — but it just helps give us better grounding in how likely it is that something will happen. And I think that will enable us to keep moving forward with more realistic forecasts of what’s going to take place. Rob Wiblin: Yeah. I think this is something you often say in your grant pages, that you’re hoping to learn about a programme, or learn about a general area, by making a grant in it and then seeing how it goes. Something that’s a little bit surprising about that is you might think that in the world as a whole, probably there’s been many grants made to technical assistance programmes that kind of look like this. You might think you could learn from possibly dozens of those, that you could find out about them through the grapevine or in published papers. And then having done that, just learning about this one extra one that you happen to fund might kind of be a drop in the bucket of all of the evidence out there. What would you make of that? Elie Hassenfeld: I think it’s a really good question. I’ll give you my answer, but I think it’s a really good question, and I’m not sure we’ve done enough on it is maybe the short answer. The instinctive answer I have is that it has been extremely hard — and harder than I would have expected — to learn about success from cases that were funded by others that happened outside of us. I could think of a lot; I’ll just give one example. Way back when, GiveWell supported an organisation called VillageReach. If you want to read about them, they’re on our website. We were very highly supportive of them in 2009, 2010, 2011. Rob Wiblin: I donated to VillageReach back in 2009, 2010. Elie Hassenfeld: Right when we first met. I think our first conversation might have been about VillageReach, which is amazing. Rob Wiblin: Yeah, I think that’s right. Elie Hassenfeld: We were not the original supporter of VillageReach; we were relying on evidence from its work in 2006, 2007, 2008 that had been funded by others. And I think we wrote about this on our blog. VillageReach was an organisation that provided logistics support for immunisations, and they had decent evidence that they had increased immunisation uptake in the places they served. We supported them on that basis. And we tried to go back and understand what was the cause of the increase in immunisation in Cabo Delgado province in Mozambique. Even though that programme had only happened two or three years before, it was incredibly challenging to figure out what had happened. I still think we don’t really know. I think another place you see this is Open Philanthropy has funded some work on the history of philanthropy. That was something that I was pretty involved in way back when, when GiveWell and Open Philanthropy were combined. I think we got some decent perspectives on how different philanthropic programmes have gone, and had an idea of what happened. But when I think about a case like the Clinton Health Access Initiative’s impact on lowering HIV drug prices — HIV drug prices fell precipitously from the time they were developed until a few years later when they started being distributed widely — it was hard to really home in on what exactly happened and how much they contributed to that reduction. We found it a lot easier to understand a situation when we were there from start to finish, because we have a better understanding of what’s there and what’s not there. All that said, I do think this is a reasonable critique. And I think it is very possible that with the resources and staffing we have today, we could put more time and energy into it and probably learn more from other cases. Rob Wiblin: Speaking of resourcing, what’s the number of person years that goes into making a grant like this one? And maybe also, how many person years might go into making the much bigger grant, the $65 million grant to the water chlorination one? Elie Hassenfeld: It’s really hard to say. For the water chlorination one, very roughly, I would say it’s starting on the basis of all the work we had done previously on water. So let’s start the clock in 2021: that included a previous review of research, and a site visit to Dispensers for Safe Water. Probably six months full-time equivalent to get to where we were, and then we continue working on it. Something like syphilis was probably also a lot, because it was the first time that we did something like that, and whenever we’re looking at something like technical assistance for the first time, or it’s early, we tend to look at it more carefully. I’ll just make up a number and say maybe today that would be closer to 1–1.5 months of full-time equivalent work to get to the point of being comfortable making, I don’t know, a $10–15 million grant in a relatively new area — but one where we have a decent grounding in the key parameters that we’re going to need to think through about how the programme has impact. Rob Wiblin: That’s a lot less than I was expecting. Actually, maybe it’s something going on that you also spend time looking at water quality interventions in general before you look at that specific programme? Or is this all in? This is both looking at the intervention and coming up with a cost effectiveness thing, and looking at Evidence Action in particular? Elie Hassenfeld: So it’s all in. It starts in 2021, so there’s a lot of work that it’s building on top of. If you counted that, maybe you’d get up double or triple that, but still, that’s 1–1.5 full-time equivalent. It definitely helps that it’s Evidence Action, because they’re a partner we’ve worked with for a very long time, and it’s very easy for us to communicate with them. By that I mean if we ask a question, we get a very clear answer, positive or negative. That is a 90th-percentile outcome for communication. So it does make it easier. But there’s a lot that we’re doing to explore the wider space. We do a lot of investigations that don’t end up leading to grants, but we expect in a given year we might do, I don’t know, 60 total grant investigations — of which half result in grants. And then we’re aiming to write about everything we’ve done publicly so we can be critiqued. Altogether that takes a lot of time. Rob Wiblin: If I recall, you said the headcount for GiveWell is now about 60. What fraction of those folks are working on research, broadly speaking? Elie Hassenfeld: I’d say it’s roughly 35 people on research, 15 or so people on outreach — which is raising funds for our recommendations — and then the remainder on general operations. Rob Wiblin: OK. If you’ve got about 35 researchers and $600 million in money directed each year, then you can just do the ratio that I suppose a year of a researcher moves something like $20 million, as a ballpark? It’s just quite a lot. I might be stressed out if I was a GiveWell researcher thinking about that. Elie Hassenfeld: Well, I do think our team is smaller than it should be, given the amount we’re moving. And that’s both because of that figure that you gave, but also I think we would like to do a better job engaging more with the outside world and getting more critique, thinking through critiques, bringing things to others. And I think that requires a larger team than we have. Our funding went up faster than we were able to grow the team, and we’re aiming to catch up. Rob Wiblin: Yeah. OK, just coming back to the syphilis technical assistance programme. I suppose many people will associate GiveWell with recommending things that are just proven to work: extremely firm, strong evidence; randomised trials; and so on. In this case, of course, you just have to form an estimate of how likely it is that this programme is going to cause an uptake of this new test and what fraction of people in Liberia are actually going to end up using it, and how long they might. It’s quite a bit more speculative. How do you handle that? Elie Hassenfeld: Mostly it’s handled in, first, conversation with the organisation that we’re supporting — in this case Evidence Action — and then debate internally. All of that just leaves us with an estimate that ultimately we know we’re very likely to update from as we learn more about what happens. We’re just aiming to honestly do our best, with the understanding that it’s far from perfect. The way we think about it is our first grant here was roughly $5 million. That’s about 1% of the portfolio that we’re giving each year. It’s worth making some bets that we’re highly uncertain about, for the sake of learning more over time that they could be really effective. Rob Wiblin: Yeah. I painted this picture of GiveWell as being focused only on the best randomised controlled trials and so on, but I think that has always been a misunderstanding, or at least has been a misunderstanding for a long time. Maybe you’re quite a bit more comfortable with risk and estimating probabilities and doing expected value calculations than people appreciate. Elie Hassenfeld: We’re certainly aiming to do that. Well, let me say two things. First, there is a large portion of our donor community that really wants high-confidence interventions. As an example, as we’ve been clearer with people about the nature of the case that we make for deworming — which is there is an exceptionally strong randomised trial behind it, but the results are weird and we don’t have high confidence that that study would replicate today. Therefore, we see it as a risky bet. As we’ve said that, we’ve seen some people move away from deworming to support other things — because there’s this contingent in our community that says, “We want those high-confidence options,” and so we want to continue to offer them those high-confidence options. Those are essentially the GiveWell top charities. We think about them as the blue chips or something, to use a finance analogy. Then there’s also a large contingent in our donor community that wants GiveWell to use its judgement to try to offer the opportunities that maximise expected value. Even in that world, we’re definitely further on the spectrum of, “Let’s really understand this, let’s give to things we’re excited about.” We’re not taking an explicit hits-based giving approach, but there’s a lot that we’re doing that is from that expected value framework, and I think that serves another portion of our donor community. I think both what they want — and what, I should say, importantly, we believe — is one of the best ways to give. I think one of the most important parts of this, for me and my temperament, is that mostly we’re in a position to follow along and learn — in, I guess short order in the world of philanthropy, long order in other places — over the course of several years: we learn what we are right and wrong about, and use that to update and make better decisions going forward. Rob Wiblin: Let’s push on and talk about another technical assistance programme which is going to have some similar elements to it, and that is Kangaroo Mother Care. You funded researchers from a small nonprofit called the Research Institute for Compassionate Economics (r.i.c.e.) to work on encouraging uptake of Kangaroo Mother Care in a hospital in Uttar Pradesh state in India. Can you describe the intervention here? Elie Hassenfeld: I would describe this programme as closer to direct delivery: really supporting the implementation of the programme itself. The way that this works is we supported r.i.c.e., they support a small local NGO, and this local NGO pays for nurses to deliver Kangaroo Mother Care. What is Kangaroo Mother Care? It’s encouraging mothers of newborn infants who are low birth weight — low birth weight infants are at high risk of mortality in their first month of life; 15–30%, say, mortality risk — encouraging those mothers to hold those newborn babies close, do skin-to-skin contact, and initiate early breastfeeding. This programme has seen significant — 20–30% — reduction in neonatal mortality as a result of the programme. This organisation that we’ve been supporting is directly supporting the implementation of this programme in hospitals in the state, via paying for dedicated staff that will be there to train caregivers — meaning mostly mothers of newborns — to initiate this programme. And then also paying the hospital to allocate space, so that mothers have a place to be over these first few days after their child is born, so they can do this programme. So that’s the basics of the programme. Rob Wiblin: The issue is, maybe for all babies, it’s good for their growth and for their health and their survival to potentially have as much skin-to-skin contact with their mother, or their father as well. Probably good for them in all kinds of ways, but I suppose it’s particularly important for premature babies or low birth weight babies — in part because I think it’s particularly difficult for them to maintain a good body temperature, and skin-to-skin contact keeps them warm. Doing this programme of lots of extended skin-to-skin contact and early breastfeeding reduces mortality among low birth weight infants by about 15–30%. I think that was compared to hospitals where they have facilities for treating low birth weight babies, where they actually have alternatives other than this Kangaroo Mother Care programme. Whereas I think in some hospitals in particularly poor locations, in fact, the alternative is nothing, because they just don’t have the facilities. Is that right? Elie Hassenfeld: Yeah. This programme, Kangaroo Mother Care, demonstrated a significant effect when compared to hospitals in low-resource settings that did have a neonatal intensive care unit with a warmer to support the infant. I imagine that that would be very different than the hospitals where my children were born, in the Bay Area. So it’s not at that level. But when comparing Kangaroo Mother Care to hospitals with those facilities, Kangaroo Mother Care showed this significant reduction in child mortality. Rob Wiblin: Yeah, OK. What’s the prima facie case for this one being not just good, but amazing? Elie Hassenfeld: Basically it’s that low birth weight infants need to be kept warm. This keeps them warm. The instruction is to hold the children close to the mother’s body or the parent’s body for eight hours a day for several days. It’s very time intensive. It keeps the babies warmer and also provides them comfort. And then early initiation of breastfeeding — where breastfeeding has some evidence that it reduces mortality, especially in low-resource settings. This programme has this large effect. And it’s not a high tech solution — this is something that, if hospitals can be convinced to do it, it is doable often in existing settings. Rob Wiblin: Yeah. I guess it raises the question of why this isn’t already standard practice, if this is the best that you can do for helping out these babies, and it doesn’t really require any particular resources, other than people who would support the mothers that you might have anyway. I suppose rooms for parents to be in with their children at the hospital, or possibly they could be instructed in how to do it at home. Why isn’t this already just how things are done? Elie Hassenfeld: I think there’s two big reasons. First, the actual lever for change is training hospital staff, and there’s high turnover among hospital staff. So if you train one person, but then they leave their job immediately after, that doesn’t help implement the programme. A big difference that this r.i.c.e.-initiated intervention has is that they’re paying full-time staff to be there consistently over time. And then second, often hospitals just don’t have space for people, and space is expensive. So the other way that this programme is having an impact is by ensuring that that space is created. I do think, looking both at this and syphilis from an outsider’s perspective, you might say this seems obvious; why isn’t it already happening? But again, I imagine the hospital administrator who has to make a lot of changes in how they operate their organisation, and often that’s just hard for institutions to make that kind of change. Rob Wiblin: Yeah. They’re potentially run off their feet, and so they don’t have time to step back and think, “How should we change these practices?” It sounds like the evidence for this being good is very strong. You might think that maybe what we should do here is go to the minister of health for this state, or maybe for India as a whole, and get them to kind of champion this, and say, “Hospitals, you really ought to be doing this. This is very cheap, and it’s going to save an awful lot of lives.” Rather than go hospital by hospital, maybe you can do something a bit more centrally. Is that possible, or is it just not how things work? Elie Hassenfeld: I think it’s possible. We initially did our research on this several years ago, and then talked to some organisations and didn’t feel very optimistic, based on what we were hearing about how likely they were to work. So we were excited when r.i.c.e. came to us, because it was a very dedicated model that was focused on solving some of these implementation challenges that seemed so key. Also, I think you’re right: I think that often there are a lot of different pathways through which philanthropy can have impact. We’re excited to keep exploring ways that this programme could be scaled up further — either top down, as you’re describing, or maybe bottom up, with r.i.c.e. being able to direct other larger organisations about how they can bring this to a wider scale. I do think one big difference is this programme does end up being fairly expensive on a per-person-reached basis. Our rough estimate is this is $400 per low birth weight infant reached with Kangaroo Mother Care, and that ends up being pretty cost effective because the reduction in mortality and the mortality rate are fairly high. But it’s not hard to understand why it would be difficult for, say, the state of Uttar Pradesh even — never mind an even lower-income location — to have the resources available to just implement this programme. This is something that I remember you talked about with Karen Levy when she was on the show, and I thought it was very telling: that when the challenge is funding, funding is scarce in low-resource settings, obviously — so one of the major levers for change that outsiders have is to provide funding where needed for very effective programmes. Rob Wiblin: One aspect of this, and maybe many other technical assistance programmes, might be that inasmuch as something isn’t already being done, and perhaps there’s some resistance to changing practices, you might worry that perhaps that you’re missing something — that maybe the hospital staff knows something that you don’t; perhaps there’s a reason why this advice isn’t as good. Or maybe doing Kangaroo Mother Care is going to create other problems for the hospital, and it’s actually more difficult than what you’re appreciating. Is this something that you look into? Elie Hassenfeld: Yeah, we both look into it and worry about it. A couple of things. In this case specifically, the fact that we’re supporting an organisation that is very small and very focused I think mitigates this concern to some extent, though not completely. Meaning r.i.c.e. is very focused on a small part of this state, and that enables them to be more aware of what’s happening — this is not GiveWell running a global programme to roll out Kangaroo Mother Care all over the world. And they’re supporting a local NGO, whose acronym is PHI, to do this. It’s supporting groups that are fairly close to the ground, number one. Then, number two: everything we do, we’re aiming to talk to people who have local context about the programme that we’re supporting, to just ask them what we’re missing. That could mean like a district health commissioner in a state of Nigeria about a programme that we’re considering, to get feedback and hear from them about what we might be missing. All that said, obviously we could do more, and I think we probably should. As we increase the size of our staff, we will do more — because if there’s probably one story about how development has failed most egregiously historically, it’s being insufficiently attentive to what people on the ground know. We’re very aware of that. While we’ve done the things we’ve done to mitigate it as much as we can, I think we could also do a lot more. It probably would lead us to find things that would be surprising. Rob Wiblin: I guess in this case, maybe it’s not quite as severe a concern, because it doesn’t sound like there are people who disagree with this advice or are refusing to do it for some reason; it’s more just that people aren’t super paying attention to this. I guess it will be more of a red flag if you were advocating for some changes that other people were opposed to. Something that’s odd about the whole way that this is being done is that you’ve kind of got an organisation looking at one aspect of medical practice in one hospital at a time. In a sense, it feels surprisingly piecemeal. You might think hospitals are dealing with all kinds of different patients and all kinds of different medical problems and offering many different ways of curing them. And presumably they have some kind of guidebook about how they would do this, or people are taught to do things a particular way in a medical school, and then they implement that. It feels like maybe there should be some more holistic thing, where someone goes to the hospital and tries to update them on a whole lot of different practices simultaneously, rather than just looking at one that’s particularly salient to them, given that there could be so many areas for improvement. What do you think? Elie Hassenfeld: Yeah, it’s a really good question. I think there’s two things I want to say about it. First — this is going to be a very general answer to the high-level question of why so much bottom up and not more top down — often the way that solutions come about is that people experience a very particular pain point in a location that they are, and they aim to fix it. To use GiveWell as an example, Holden and I wanted to give to charity. We were really frustrated by what we could find. That leads to GiveWell. And I think that is just often how solutions come about: that you just have individuals noticing problems and then aiming to fix them. I think the criticism of it might be something more that GiveWell institutionally has been an organisation, historically, where we mostly are responding to people coming to us with ideas, rather than being out there pushing particular ideas on individuals, organisations, whomever. That has a lot going for it. But I also think that in a case like this, we could probably do better at noticing this as a possible path, offering it to people, and seeing if someone wants to take it up. Could you go and identify, say, the three most obvious procurement changes that low-income country governments should make, or the three most obvious hospital-based changes that should be made? That would take someone some work to do, but I think it’s a pretty good idea. And it’s not surprising to me, based on how GiveWell has historically been set up, that we haven’t done more of that. Rob Wiblin: Do you have any sense of how many multiples of giving cash this programme might look like? Elie Hassenfeld: Our central estimate is 13 times as cost effective as cash. That number is way too specific. I mean, we don’t really know. You could call it 10–15, but like everything else, I think we’re going to learn a lot. This was roughly, I think, a $2 million grant. We’ll get some data on folks reached: the organisation is doing a baseline survey of mortality and then going to track their own effect on mortality — not a randomised trial, but a pre-post study — and so we’ll know some more, and hopefully we can triangulate that number and learn more about where it actually is over time. Rob Wiblin: We maybe should have said something earlier to help people understand these multiples of cash a little bit better. So yeah, you have this kind of baseline programme, which is just find people who are very poor and give them $100, or give the household a year’s income or something like that in cash, for them to spend on whatever they want. That’s going to become this baseline against which you compare all other programmes. Depending on how good the opportunities are that you’re finding, and how much money you’re moving, the threshold above which you fund something has gone up and down a bit over time. I think at the moment, it’s something like if something is better than 6x or 7x giving cash, then you’re interested in making a grant. If it’s below that, typically you won’t. Is that roughly right? Elie Hassenfeld: Everything you said was correct, except the threshold is different. Our current threshold is 10 times as cost effective as cash. The reason this has moved around a fair amount is that back in, let’s say, the fall of 2021 — when financial markets and cryptocurrency were doing extremely well — our range of future projections for funds raised were fairly high. We said that in order to keep up with that forecast demand for our research, we expect that our bar will be in that 6x or 7x range. The funds we have actually raised have stayed fairly flat, but certainly our future forecasts have fallen relative to what they were several years ago, and that’s caused us to increase the threshold up to 10x. I do want to say two things about this, and happy to go into them if you want, but we don’t need to. The first is we talk about these numbers as if they are clear, objective truths — and nothing could be further from the truth. They’re a useful way to boil down our view of a programme quantitatively, and we are aiming to do that. Also, there’s a huge, huge uncertainty around each of these numbers. To give an example, if I had two programmes in front of me and one was 11x and one was 9x, but I strongly, for whatever reason, felt like the organisation implementing the “less” cost-effective programme was going to do a better job, I would pick the 9x programme over the 11x programme, notwithstanding what the spreadsheet told me to do, because there’s a lot of uncertainty involved. And then it’s also the case that in order to compare a programme that delivers cash to enable people to buy more things to a programme that improves health and reduces mortality, you have to come up with a translation between those. We call that “moral weights.” I think it is probably obvious to every single person listening to this that there is no true answer to how you should trade off between those two things, but that is baked in. Rob Wiblin: What? You couldn’t just look at the philosophy literature and find out the rate? Elie Hassenfeld: I mean, we have to wait another year, and then I’m fairly confident GPT-7 will have the answer. But for now, for the next year, we’re stuck in mode of uncertainty. But obviously, that also introduces a huge range of possible differences. So we use this as a shorthand. It’s very useful, but it only is useful insofar as you remember its weaknesses — because it’s not “the truth.” Rob Wiblin: Yeah. Here we’ve been talking about these cost-effectiveness multiples on giving cash, which is quite abstract in a way. Another way that you sometimes cash things out is in terms of how much you have to spend in order to save the life of, typically, a child. Is it possible to evaluate some of these programmes that we’ve been talking about in terms of dollars per life saved? Elie Hassenfeld: We definitely can and do. I don’t know those numbers off the top of my head, but we could go and find them and plug them in if you want them. Rob Wiblin: For the Against Malaria Foundation, do you have a sense of roughly how much you think it costs to save the life of a child under five these days? Elie Hassenfeld: Very roughly, a number that I would anchor on would be about $5,000 per life saved. There are a lot of programmes and a lot of locations where it costs less than that. There are also areas where it costs more than that. But it’s a useful benchmark to think about: a central estimate of single-digit thousands of dollars to avert the death of a child in a low-income country. Rob Wiblin: Off the top of my head, I think that it sounded like the multiple you were suggesting for Dispensers for Safe Water was similar, in terms of cost effectiveness, to the Against Malaria Foundation. Hopefully, if these two technical assistance programmes worked out, that would be somewhat more cost effective. Maybe they’re able to save a life or do the equivalent of that in terms of health improvement for a bit less than $5,000. One challenge with these programmes is that you have to strain to see what is the true cost here? What is really the limiting factor, or what’s the scarce resource that’s being used up? I suppose one thing is the research staff at GiveWell; that’s very salient. Then there’s, I suppose, these economists or technical experts at the Research Institute for Compassionate Economics who could be working on this or could be working on some other programme. Another more hidden cost might be that the nurses, the doctors, the administrative staff at these hospitals, although this might not cost more to provide the care, ultimately, they’re having to think about this — they’re having to rejig their systems and think about how to change in order to deliver Kangaroo Mother Care, rather than maybe making some other improvement to their practices. Maybe that’s actually the biggest cost of the programme, even though it doesn’t have an apparent cost in dollar terms. How does GiveWell handle something like that? Elie Hassenfeld: We’re aiming to think about this holistically. In the example of Kangaroo Mother Care, let’s say that the hospital is convinced to provide additional space for caregivers. Well, what was that space being used for before? Does that mean that someone who comes in for another condition is not being treated? I don’t know for sure that we explicitly thought about that question in this case, but that is the type of question we’re aiming to cover as best we can. It’s also another way in which the results we land on are our best estimate of what’s happening. They’re useful primarily to compare programmes to one another, but the real world, what is the “true effect” of giving $2 million to r.i.c.e.? Hard to say, because there’s so many knock-on effects of what happens. Anyhow, for sure, we’re aiming to try and understand. Let’s say you run a programme that pays highly trained medical professionals more to come and treat condition A as opposed to condition B. The impact is not that you’ve created more impact on condition A; you’ve also created less impact on condition B. In everything we do, there’s a section you can find in our reviews where we look at negative or offsetting impact of our support, which is aiming to get at this question of what are the most obvious and direct secondary effects — often potentially negative or offsetting effects — that might counterbalance the positive impact our programmes are having. Rob Wiblin: Yeah, it’s a very general problem. I think the technical term that economists use when they’re doing modelling with this is the “shadow cost.” It’s kind of hidden, whatever was given up in this entire system in order to deliver one thing rather than another. It doesn’t only make it hard to evaluate the impact of this programme, but just everything. Indeed, in normal life, whenever you do one thing, that’s time that you could have spent doing something else — and even if it was useful, if it was worse than the alternative then in fact, you’ve done harm. It could be a little bit mind bending, and it’s super non-transparent to us. So yeah, just a tricky thing. Elie Hassenfeld: I think this is a good example of where getting the local context is very helpful. You could imagine talking to the hospital administrator about Kangaroo Mother Care, and they could say their hospital has space that’s not being utilised in a lobby. I remember a hospital in Udaipur when I was in India, and there was a lot of excess space that I could just imagine someone rolling some beds into. On the other hand, you could imagine them saying the opposite. This is why purely relying on academic papers and modelling is far from sufficient to make good decisions. Because you certainly cannot make a true estimate of everything, but I think you can get a better sense of where there are higher and lower magnitudes of unmeasured effects, and take that into account as you’re making grant decisions, which we try to do. Rob Wiblin: Yeah. It’s a great example of the importance of local knowledge, that whether this programme is good or bad might turn on the question of whether the hospital has a spare room at the moment. Anyone who’s dealt with an office in a growing or shrinking organisation knows that you can really flip from having too much to too little space quite quickly, just because buildings are very lumpy basically. Elie Hassenfeld: Totally. Yeah. Rob Wiblin: OK, let’s move on from that and cover our next intervention and approach, which is the nonprofit MiracleFeet, which GiveWell made a grant to recently. What does MiracleFeet do? Elie Hassenfeld: MiracleFeet serves a condition called clubfoot. Clubfoot affects about one in 1,000 children, when they’re born and their foot is turned sideways or backwards. This makes walking either impossible or challenging. MiracleFeet supports an intervention called the Ponseti method, which treats the child’s condition with a series of casts that slowly turn the foot towards the correct direction. Eventually, the leg and foot is braced and ultimately prevents the child from having lifelong disability. Rob Wiblin: So the kids are born with the feet pointing in the wrong direction, but as they’re growing, it’s possible to redirect it so that the feet are pointing in the right direction. The prevalence of this is something like one in a 1,000 births? I’d never heard of this, despite it being so common. I suppose that’s because in rich countries, basically this is just always picked up, and people get a good standard of care and basically you get a 100% cure rate by the time you’re an adult. Elie Hassenfeld: That’s my impression. Rob Wiblin: Do you know how much it costs to provide this treatment to one child? Elie Hassenfeld: I believe our central estimate is $100 to $200 per child. But this is an area where we really don’t know. Because this is a case where what the organisation MiracleFeet is providing is trying to get the clinicians who are present when the child is born to be aware of this condition; to know that there are steps they can take; to ensure that the supplies, most notably the bracing, is available when the child needs it. The casting is fairly low cost, so is mostly available. This is not a case where most of the cost is providing the commodity: most of the cost is providing salaries for people, who are then doing work to raise awareness both about the problem and what can be done about it. What that means is that in any location, the fixed costs of implementing this programme are quite high. You have to set up a country office, set up local staff, hire them. But once you’ve paid those fixed costs, the additional costs for each additional child reached are very low. We’re just not sure how this will all play out. We have some data from MiracleFeet historically, but a major component of the support we’ve provided them now is seeing how they are able to scale up, and what that means for children who are counterfactually reached — who wouldn’t have been reached before — and their cost structure. Rob Wiblin: I suppose the simple dream would be if MiracleFeet was just paying for all of the care and maybe treating people directly, and then you could just say, “How many kids did you treat for clubfoot, and what was your total budget per year?” And then you could get a ratio there. But here they’re doing something more complicated, so it’s a bit hard to figure out how much it costs per person. Elie Hassenfeld: I guess the simple dream from the analyst’s perspective — but the ideal here is that by running the programme the way they do, we get better return on the money and reach more people ultimately. Rob Wiblin: Of course, yeah. It’s simpler to analyse, but probably way less cost effective. Elie Hassenfeld: Oh, it would be great if it were just super simple and then we could know the truth — because, of course, that’s what matters most. Rob Wiblin: Yeah. Could everyone just eat one thing so that it’s much easier to keep statistics on nutrition? “People, you can only drink Huel now. I need you to be legible.” OK, what’s the case for making a grant to MiracleFeet? What’s the case on its face that this would be amazing? Elie Hassenfeld: Again fairly similar, I think, to some of the other programmes we talked about, in that it’s fairly low technology, fairly easy to train people to do. The reason it’s not already being done is some form of it being neglected — meaning it’s not high on the agenda of the ministry of health or hospitals or local clinics. Because one in 1,000 births means that a very large number of children globally are affected by this problem, but for any individual country or individual hospital even, this is not high in the list of things they see on the list of public health problems that they’re aiming to fix. So it really pops and looks great when you’re taking a cost-effectiveness perspective, taking into account how much it costs and the benefit that’s provided. Rob Wiblin: The benefit would be very large to not have, because if you grow up and this isn’t treated, then I suppose it’s quite difficult to walk and you’re going to have all kinds of issues as an adult — it’s a permanent disability that was totally preventable. Why is it that this one is going untreated? You’d think the parents would really be worried that their child is growing up with their feet pointing in the wrong direction, maybe, and might go get followup care. You’d even think, from the perspective of a country, the internal return to curing this problem must be enormous — because it’s so cheap to cure upfront, and if you don’t do that, then you’ve got this massive problem permanently for this person as an adult. Is there an explanation for why? I suppose the answer with all of these things seems to be these are extremely resource-constrained places; there’s a lot of health issues to treat and this is just one of many? Elie Hassenfeld: I think that’s right. If you’re the minister of health in a country, you’re looking at a public health burden from malaria, pneumonia, diarrhoea, tuberculosis, HIV, et cetera that are far higher than this. This is not coming to the top of your agenda. I think looking at it from the perspective of cost effectiveness — which I think is somewhat rare as people are looking at opportunities to prioritise — is what brings this up. It is harder for me to really understand the caregivers’ mindset. I shouldn’t say their mindset, but their situation. Largely I think that’s because our experience in high-income countries is so fundamentally different: we’re used to feeling as if “I have a health problem; I will get care immediately.” I mean, I told you I recently dislocated and fractured my shoulder, and it was actually amazing to me how quickly I got care. I was skiing, the ski patrol came, I went to urgent care, I saw a surgeon 12 hours later, and I was in surgery like three days later. This is the expectation we have, and I think it’s very different in low-resource settings, where one might not even be aware of the possibility of treating certain conditions. I know we’re going to talk about malnutrition later, but one of the activities that organisations focus on when they are trying to reduce malnutrition is helping caregivers — meaning most often parents — know that their children are undernourished by giving them a tape measure to measure their upper arm circumference. Otherwise they may not know, because malnutrition is very common in certain communities. If you look around you and see that your child looks fairly similar to a lot of other kids, you might not have another thought about it. I do think that makes some of the intuitive understanding or comparison hard, because our experience is so different. Rob Wiblin: Yeah. As I understand it, this intervention you could group in a bit with a bunch of other corrective surgeries that GiveWell has looked into. What’s this cluster and what do they have in common? Elie Hassenfeld: There’s a number of programmes we’ve looked at over the years, including cleft palate surgery and others, and the big question that we’ve always had about these is: Are you able to support higher volume of this particular health problem without having the offsetting impact of allocating limited, highly skilled medical professional capacity away from condition A to condition B? The reason that this one really stood out is that the vast majority of this treatment is casting and bracing, which can be implemented by lower-skilled medical professionals. I forgot this earlier, but there is a surgery called a tenotomy that has to happen, which loosens the tendons in the foot and ankle, but we understand it’s fairly straightforward and so easier to conduct. And that’s ultimately the reason that, of all the different types of interventions we’ve looked at over the years, this is the one that we’ve seen as most promising: because the surgery that has to take place is relatively simple, the overall care is simple, and therefore the likelihood of offsetting impact seems less. Rob Wiblin: This is highly related to what we were talking about earlier, that it sounds like you’ve been wary of funding some of these things in the past because you’re worried that maybe the limiting factor to the number of people who can be treated for these conditions is not funding; rather, it’s the staff available to treat it. Maybe you could spend more money on it and in fact, no more people are treated, because it’s just not possible, given the number of surgeons available. I guess an alternative thing would be that maybe more people would be treated for this, but then someone else would go untreated for some other condition that they needed a surgeon for. So maybe the impact is much less than what you would think. That’s basically it? Elie Hassenfeld: I think that’s right, and that might lead you to say the one approach is investing in surgical capacity in low-income countries, which I think is definitely an interesting option, though it comes with significant challenges around knowing whether what you’re giving to will lead to these positive results far in the future — because training medical professionals and surgeons is a long road. Rob Wiblin: Right, yeah. It feels like something about this… I guess the expression that we use is that “it proves too much” — because it seems like we don’t normally think, in ordinary life, that if we buy one thing or if we ask doctors to do one thing, that’s just going to cause an equal amount of damage in them stopping doing something else. In part because we think that if you pay for more medical care, then hopefully you’ll train more doctors over time, and more hospitals will be constructed, and so you might be directing more resources into medical care as a whole. So these offsetting effects, in the long run at least, won’t be quite so severe. Is that something that you’ve looked into? Elie Hassenfeld: I guess we’ve mostly thought about it from the perspective of what happens over the next few years as we’re supporting more of this activity. I remember talking to a surgeon, and he was telling me that more money to his organisation would be great, and then also said that he was working 18 hours a day, seven days a week. I was like, I’m not really sure. More time to work with more money. So yeah, I don’t know, maybe it would have the effect over the long run that you described. Rob Wiblin: I guess, as an economist, you think you might bid up salaries for surgeons and then more people will go and become surgeons. I suppose that does highlight how that might be actually difficult in very resource-constrained or poor settings: that maybe there isn’t a lot of labour market flexibility or educational flexibility that allows many more people to be trained up in these skills and stick around, merely because more money is being put towards paying for surgeries. Elie Hassenfeld: Yeah, intuitively it seems to me that, at the margin, you’d be better off funding opportunities that have less of that offsetting impact in the short term. So use school-based deworming as an example: mostly this is distributed through an existing infrastructure in schools, and students are with their teachers at school, teachers give them deworming pills. This doesn’t have a significant offsetting impact. If you can get the same short-term, let’s say, cost-effectiveness measure for both — but one has a major negative offsetting impact — I would probably choose the former over the latter. I’m not sure that’s right, but that seems intuitive. Rob Wiblin: Totally. So flipping it around now: let’s take it that this is a really important effect, at least in the short run, and maybe in the long run as well. Then you might almost just want to make a list of medical treatments that are important in poor countries, and just list them from the most trivial, the most basic — something that could be provided by a fool — and then try to fund that, because it’s going to be so much easier to scale up, and the opportunity cost of providing that will be much lower. I suppose just giving people a deworming tablet would look pretty good there. Maybe antibiotics and vaccination delivery are also relatively straightforward. Elie Hassenfeld: And I think that ends up playing a role. So in our prioritisation, one thing we’re doing is we see a lot of programmes, and we have to make a choice about which ones to look into most. There’s some work that we’re doing up front to decide which ones to prioritise, and one factor is: how straightforward is this to deliver? That’s a major component. If something is straightforward to deliver, it moves higher on the list, because it’s more likely that we’ll reach a point where we can direct funding to it. Rob Wiblin: Yeah. What are your biggest reservations about MiracleFeet, if you have any? Elie Hassenfeld: I think the biggest reservation is honestly how little we know about two major components of how cost effective this programme is. Number one: how many additional children will be reached because of our support to MiracleFeet? We really do not know. At this point, we’re guessing, and we are going to know more in a few years. And number two: we don’t know how much it costs to reach each additional [child]. So it’s how many will be reached and how much it costs: those are going to be two major components of what we believe over time, and we’re going to learn a lot. We have our best guesses, but similar to what I’ve said in the past, this is an area where we at this point are optimistic because of our support, but fairly uncertain. And we’re effectively making a bet based on our best estimate that’s going to settle in a few years. Rob Wiblin: OK, let’s switch on to food-related programmes now. The first of these is treating serious malnourishment among young children, which leads them to be underweight and have stunted growth and development. I think you made a $1 million grant to the Alliance for International Medical Action (ALIMA) to provide treatment for malnutrition and paediatric emergencies in Niger back in 2021. What’s the exact problem we’re talking about here? Elie Hassenfeld: Many children in low-income countries are severely malnourished, which is really undernourished, meaning they’re just too skinny. That could mean low weight for height. Also, there’s a measure of their upper arm circumference, which is the clinical diagnostic for malnutrition. This is then categorised as either moderate acute malnutrition or severe acute malnutrition. Being malnourished, we think, leads to children being more susceptible to death from infectious disease. Therefore, ultimately, it’s a major factor that leads to high mortality rates among children in low-income countries. Rob Wiblin: Yeah. I think of all of the problems that I looked into preparing for this interview, this is the one that somehow stunned me the most. You’re estimating that something like 50–200 million young children are suffering from malnutrition at any given point in time. An interesting phenomenon with GiveWell is that your mission is to search for the most outrageous, stupid, harmful things that are happening that would be incredibly easy to solve. So all the time, you’re identifying things that it’s kind of crazy that this is happening, because it would be so easy to fix. And that’s why we’re looking for it and funding it. This may be a stupid question, but why are so many young children not eating enough food? Is it poverty or is there more to it? Elie Hassenfeld: I guess what I would say is in order to treat a child who is severely malnourished, it’s fairly costly. We estimate it’s $100 to $200 per child treated. With the prevalence of malnutrition globally, it would be extremely expensive to reach all of those children. I don’t personally have a clear and specific understanding of what the root cause of children not having enough calories is, and why is that happening — aside from the somewhat superficial take of when people are in very dire conditions, often they have insufficient food. Rob Wiblin: Maybe this is talking too much about my feelings, but one thing that blew me away about this was that this was a week when GPT-4 was launched, and we’re seeing these just incredible technological breakthroughs. It feels like the things that humanity is capable of are staggering, and yet we still fail to get such a large fraction of children enough calories. I’m not sure there’s any lesson from that, but the world is a crazy place. Elie Hassenfeld: I think “the world is a crazy place” is the right take. Just to go through all of the things that we’re supporting, I mean, 500,000 people die from malaria every year. Many, many people don’t have access to clean water. Insufficient food is an even crazier reality than the craziest reality: that people’s lives in low-income countries are shockingly hard compared to the wealth that we have in high-income countries. Because of that, we in high-income countries have an amazing opportunity to improve people’s wellbeing a huge amount if we’re giving and supporting making their lives better. Rob Wiblin: Yeah. So why does it cost about $100 or $200 to treat a child for malnutrition? So we’re not just giving them basic food? There’s more going on here? Elie Hassenfeld: What’s happening, what are these organisations doing? Every country has a programme that is targeting this problem. This is not a neglected problem; this is a known problem. But the countries don’t have the resources to provide the programme at the scale that’s needed. So an organisation like ALIMA comes in, and on one hand, they are providing some basic health commodities. The literal food that is being provided to the kids, the high-calorie-density food, is provided by nonprofit support. They are training health workers both to identify malnutrition in communities but also screen and treat children when they come in. They’re providing incentives for staff, so that staff are there and staffing the clinics where children show up. One of the things that has been challenging historically — and is something that ALIMA has worked on — is aiming to have a standard protocol for treating children, whether they are moderately malnourished or severely malnourished. Before, there was a totally different protocol for each — and that made it challenging, because you had to have two different setups to treat kids when they came in. And they’ve tried to simplify that, where basically you do something more severe, more intense for severely malnourished children, and less severe for moderately malnourished children. But in addition to bringing them in, giving them this specialised food that’s very high in calories to treat them, often children present with infectious disease. So part of reducing mortality from malnutrition is receiving kids who have this additional risk factor, and treating their other health conditions at that moment to get them back to full health so that they’re able to live successful lives. Rob Wiblin: So what new stuff did you learn in the process of researching this one? Because as you said, this is a known issue, and one probably that you’d thought about years ago but hadn’t yet funded. Elie Hassenfeld: So there’s a lot of issues, like we were saying, where it’s a catastrophe that people are still suffering in 2023 from these problems. Our work in trying to understand this problem was what does giving to the organisations working on this accomplish? How cost effective is it? And the first challenge in malnutrition that prevented us from working on it earlier is the available data — which would enable you to estimate how often children who are malnourished die, and the treatment effect of giving them high-calorie food — is all from observational studies that were conducted in the 70’s, 80’s, and 90’s. Researchers basically followed groups of children, tracked their level of malnutrition and their eventual mortality rates, and sort of put that together to come up with an overall estimate of the mortality risk from malnutrition and the treatment effect of treating for malnutrition. Observational data like that is very hard to interpret, and to come up with a central estimate of how the world of the 70’s, 80’s, and 90’s compares today was very time consuming for us. It took us some time to get to it. Maybe that’s the starting point of what made this challenging on its face. Rob Wiblin: OK, the big uncertainty was around how many lives would you save per person who the service was delivered to? The problem was that there weren’t randomised trials where some people were not treated for it. How do the observational studies establish this at all? What do they compare the treatment group to? Elie Hassenfeld: What they’re doing is tracking children of all different nutritional statuses, and finding that children who are very severely malnourished have a much higher mortality rate than children who don’t have that same underlying condition. Now, the obvious question is the unobserved variables. But basically we’re relying on that. I think there’s a, I don’t know, highly plausible/intuitive case. We know that insufficient food ultimately leads to death, and so we’re starting with that as a very important anchor in an overall assessment of this. Rob Wiblin: Humans need food to live, yeah. Elie Hassenfeld: But we’re then using that data to form the estimate that we need. Because ultimately our job is not just to say it would be better than not if everyone were treated for malnutrition, which is obviously true, but to decide where, when, and to what extent to support malnutrition, versus the array of other global health problems that we’ve been talking about. Rob Wiblin: I suppose you could have a reverse causation thing, where children that have some other health condition, maybe they have an ongoing infection, perhaps that’s causing them not to eat so much because they don’t have much appetite, because they’re ill. I guess you could imagine other causal pathways as well. Elie Hassenfeld: Yeah, exactly. It’s complicated. You could imagine a lot. Rob Wiblin: Yeah, right. OK, so basically it was just a question of it took a very long time to look into this. It sounds like this was close to your threshold for funding and not funding, and that this question of just how bad it is to be malnourished could make the difference between it being worth funding and not funding, which is kind of interesting. It surprises me a little bit the degree to which many of these different interventions seem to have quite similar cost-effectiveness estimates from your analysis. You might think that they could just be radically different numbers, where something could be 100x cash and other things could be much worse than cash, I suppose. Well, you wouldn’t look at the ones that are much worse than cash, but you could imagine much more variance. Is there anything interesting about that or any possible explanation? Elie Hassenfeld: I mean, I think the explanation is that’s exactly what we observe. If you looked at our cost-effectiveness spreadsheet — which I know now you’re in love with, you can go back to — if you look at each individual programme we’re supporting, say vitamin A supplementation, you’ll see numbers that are very high — in that 50x, 60x zone. Of course, what we’re talking about are the programmes that we decided to support, which are nearly always going to be 10x or higher. But there are plenty of programmes that we end up estimating, I don’t know, 2x, 3x, 1x, maybe less — obviously very far below 1x we probably don’t spend a lot of time on — but we see this very wide variation in everything we consider, from very low to very high. Rob Wiblin: So how does the malnourishment treatment now compare? Do you know the multiple roughly off the top of your head? Elie Hassenfeld: Well, I think one important thing to know, that we’re not talking about as much, is malnutrition as a whole is not going to have some estimate of what its cost effectiveness is on average, globally — because basically, there are some places where underlying government resources and treatment are quite poor, malnutrition rates are quite high, mortality rates are quite high, and there’s relatively higher population density. So if you have those four characteristics, those are going to be the locations and the contexts where malnutrition will be very cost effective. Then, as any of those four factors changes, you’ll be in a context where it is much less cost effective. To be honest, the same is true for every other programme we’ve discussed — maybe with the exception of the very small ones, where we’ve actually focused on a location, and largely the grant as a primary learning purpose. Malaria nets, vitamin A, water: these are all programmes where there are going to be locations where it would make no sense to support this programme because it won’t be very cost effective, and then some locations where it’s extremely cost effective. What we’re aiming to do is find those locations where it is. For malnutrition, the areas where we’ve supported it we believe are in that 10x or so zone. That’s not to say that malnutrition everywhere would be in that zone. We’re looking for those places where we’re going to get the cost effectiveness that will enable us to deliver money to organisations that then help people above where we would otherwise spend that marginal dollar. Rob Wiblin: It’s a slight guilty pleasure of mine getting you to give these precise multiple figures — because I know, having done cost-effectiveness analysis myself sometimes, that you feel so uncomfortable saying it, because there’s so much uncertainty. Sure, you say 10x, but it could be 2x and it could be 30x. Elie Hassenfeld: I’m glad to know that you’re just trolling me. That’s good to know. Rob Wiblin: But yeah, there’s a reason why you do say these numbers, which is that in the process of pinning it down, you learn a whole lot, and it kind of focuses the mind rather than just allowing things to be vague. Elie Hassenfeld: And the way historical GiveWell communicated, we said, “Programme X, organisation Y is good, and here is its cost effectiveness.” And that is just not reality. Like, there are some locations where malaria prevalence is much, much higher than it is elsewhere. There are some locations where underlying child mortality is much higher than it is elsewhere. And that has a major effect on the cost effectiveness. A big part of what we do is not only to say this programme is worth considering, but then identify a location, an organisation, and a context where that programme will be delivered above our cost-effectiveness threshold. Rob Wiblin: In November 2021, GiveWell published this article, “Why malnutrition treatment is one of our top research priorities,” laying out why you wanted to look into this much more. How have your views changed since then, if at all? Elie Hassenfeld: I think back then, based on this complicated assessment of the evidence that we had gone through, we thought that malnutrition was exceptionally cost effective — like, we were going to find maybe hundreds of millions of dollars’ worth of programmes around the world that were going to be above that 10x bar. Because it seemed like it might be so much of our funding, we decided to dig fairly deep into the evidence to see what we could make of it. I’m happy to go into that process and what we learned, but the bottom line is that that whole process then led us on a long investigation over the course of another year that meaningfully updated us about the cost effectiveness of malnutrition programmes globally. At one point it took us all the way down where we thought mostly they would be in the 2x range. Then we realised we had made a mistake there, and kind of came back to a point where we expect to fund malnutrition programmes in certain contexts based on the factors I described, but moderated a lot from where we had been previously, based on going deeper into the analysis. Rob Wiblin: It sounds like there might have been a lot of little modifications there that you found by scrutinising things more closely. Is there maybe one or two that you could explain? Elie Hassenfeld: Yeah, I’ll explain what ended up being the biggest factor. We were relying on this observational data, as I described, and there was a meta-analysis of all of this which had effectively an aggregate ratio of the mortality risk to a severely malnourished child relative to the mortality risk of the, I don’t know, “normally nourished” child. That was an aggregate data point that came out of a meta-analysis of this observational data. And when we actually went back and got the data, what we realised is that all of the individual observational studies had different followup points for reaching a conclusion about the mortality risk. I don’t remember the exact time frames, but imagine one assessed nutrition status at day zero and then assessed mortality after two months, and then another one maybe did the same, but after a year. There was a very clear relationship between time to followup and mortality risk. Basically, the shorter the followup, the higher the ratio of malnourished mortality was to regular mortality, because often malnutrition can move around. Over the course of a year, many other factors introduce themselves to a child’s life that lowers the strength of the relationship between these two factors. And so we had David Roodman, of some fame, dig into it; another expert, a statistical expert, dig into this data. We had mistakenly used this ratio as an annualised number in our estimate of the effect of malnutrition on mortality — that was an error on our part. When we did this big reanalysis, it led us to significantly reduce that figure and lower the cost effectiveness from where we previously were. Rob Wiblin: Got it. So of course, the time immediately after someone comes in for malnutrition is a time of particularly acute risk of them dying because of malnutrition. So you were taking numbers that were looking at the rate of death over those few months, and then extrapolating that to the following year. But that’s just not correct to do, because probably their malnutrition will go away and the risk will decrease? Elie Hassenfeld: Yeah. I think what was surprising to us was the original meta-analysis that we read didn’t highlight this fact. Now, maybe if we were more embedded in this particular field, we would be more aware of the norms of description there, but it presented a high-level number and we took that at face value, and obviously made this other error. Often GiveWell goes deep into things, and many times we go deep and it doesn’t change our mind. It’s like we know more about it; we’ve learned more. But this was a case where we went really, really deep and spent a tonne of time collectively, and it made a big difference to what we ended up doing. Rob Wiblin: I suppose I could ask this question about almost any of these programmes, but do you think GiveWell might benefit from having a dedicated malnutrition expert on staff, someone who’d worked on malnutrition in the past and was really familiar with it? Because, as I understand it, you mostly hire generalists rather than specialists. Elie Hassenfeld: I’m not sure about the answer. Mostly we get subject matter expertise by talking to outsiders. I mean, there’s so many different types of expertise that we rely on that I think it would be hard for us to have experts in every domain. So we spend a lot of time talking to people who don’t work at GiveWell for help. We are hiring currently for a malaria researcher. That’s to find someone who is going to be dedicated to malaria, and has more experience in malaria, and can bring that to bear on our work — because malaria is the disease to which we direct the most funding. And that’s an experiment. If that goes well, I could easily imagine bringing on more dedicated researchers in areas where we do a lot of ongoing work. Rob Wiblin: OK, let’s move on to the last of these six interventions, another nutrition-related one: vitamin A deficiency and supplementation. In the interest of time, we’ll keep this one brief, but what’s the story with vitamin A? Elie Hassenfeld: There were a series of trials conducted 30, 40 years ago that found that delivering vitamin A supplements twice a year to children between the ages of six months and five years reduces all-cause child mortality by about 20–25%. We are supporting two organisations — Helen Keller International as a top charity, and Nutrition International as an organisation that’s somewhat newer to us — to deliver this programme globally, where there are high rates of underlying childhood mortality and, we believe, relatively high rates of vitamin A deficiency. Rob Wiblin: Was there a reason you didn’t fund vitamin A earlier, or did you learn anything interesting in the process of looking into this one? Elie Hassenfeld: We’ve been funding vitamin A for a while, so I’d say it was still relatively early in GiveWell’s growth. For a long time, we were a teeny organisation, and so had very little capacity to do very much. What made vitamin A complicated, maybe most importantly, is that a lot of the trials are from a long time ago, and it can be difficult to know what the results of programmes today will be, given those trials from a long time ago. So we did spend some time sitting with that. There have also been some trials, including notably, the very largest randomised trial, that found a near-zero effect in India of vitamin A supplementation. So it was a more complicated case of evidence to sort through and arrive at a conclusion. Rob Wiblin: You’re saying a lot of the trials on vitamin A are from a different era — the 60’s, 70’s, 80’s. I guess maybe the trials were done back then because people were still learning more about vitamin A and nutrition, and maybe it was a bigger problem then. How do you go about extrapolating that to today, and figuring out whether anything relevant has really changed? Elie Hassenfeld: Without a great deal of certainty, the process that we take is we think that the key factors in a vitamin A programme are going to be underlying vitamin A deficiency — where people don’t have the vitamin A they need — and then high rates of underlying childhood mortality. Those are the two factors that are going to lead to the most cost-effective programme, and so that’s what we rely on. I think there’s certainly a question that maybe the world has changed so much over the last 30 years that… One way of putting this might be that maybe vitamin A deficiency is not the key characteristic: it was vitamin A deficiency and very high rates of infectious disease and childhood mortality. Those rates are all much lower today, and you might expect a lower effect. The way we’ve tried to approach this is we are mostly anchoring on the idea that vitamin A deficiency is the key driver of what’s happening here. But we have been exploring the possibility of running a larger-scale randomised trial to try and assess the impact of this programme in the world today. I think the biggest challenge in running a trial like that is that, with mortality rates being what they are today — which is great; they’re much lower — it makes effectively powering a trial much harder and much more expensive. So it’s not clear to us that we could actually pull that off in a way that would be broadly cost effective. By “broadly cost effective,” I mean, in expectation, we would guess that this trial could lead us to either put more or less into vitamin A by an amount that is worth the cost of the randomised trial. That’s actually something that has been a fairly active investigation, and it’s a good example of something that I think we would be more holistically considering if we had more research capacity. We’ve looked into this randomised trial. We said this could be good, but hard to figure out. We don’t really have enough capacity. It’s one of the reasons we’re trying to hire more people, because it can help us have a better answer to this question about what else we could be doing to update. Rob Wiblin: Totally. We’re going to push onto something pretty different now. But maybe a final question is: What’s an intervention that GiveWell hasn’t written about that much that you might write about in coming years that you’re kind of excited by? Elie Hassenfeld: Right now we talked about water, and I think water is the intervention that — I don’t know; it doesn’t matter — but I’m personally most excited about. I think that’s because the need is huge globally. There are so many contexts that have different characteristics that will lead it to be more cost effective or less cost effective in certain places. We have these three interventions that we’re looking at, never mind the opportunity to potentially support the implementation of those three interventions — dispensers, in-line coordination, or vouchers — alongside a government via technical assistance. So the scope of the opportunity here seems quite large. It’s also an area that, notwithstanding its concreteness and intuitive appeal, is relatively underfunded by major donors. You know, HIV/AIDS, malaria, TB, immunisations — these have large funding bodies, both from the US government and then from other governments, and the US government and the Gates Foundation that collectively pool their funds in institutions like Gavi or The Global Fund to deliver programming. This doesn’t exist for water quality, so I think we’re also very cognizant of that, and hopefully can not only support these programmes as they’re scaling up and support research so we make better decisions, but hopefully support institutions in trying to advocate to governments and funders to bring more money into this space, because it does seem relatively underfunded compared to its need. Rob Wiblin: OK, so that’s the six new intervention areas that GiveWell has started funding and some information about them. It’s all reasonably in the weeds, and now I want to push on to much higher-level ideas for how GiveWell could or possibly should do things differently. One alternative worldview or general take on how to think about effectively helping people comes from this outfit called the Happier Lives Institute. They’re a lot newer and smaller than GiveWell, and I guess their progression maybe puts them somewhere where GiveWell was at around, say, 2010 or 2011. Their aim is also to make recommendations about which charities do the most good, and they think a lot about the developing world as well. But the distinctive part of their take is that they want to cash out all of the impact of these projects in terms of subjective wellbeing itself — self-assessed subjective wellbeing. So to compare across charities, they use this metric called the WELLBY, which I guess is kind of a play on the DALY — I’m not sure whether they came up with that — the DALY being a disability-adjusted life year measure of health. The WELLBY is a measure of wellbeing, and one WELLBY is the value of raising someone’s self-assessed subjective wellbeing by one point on a 10-point scale for one year. If an intervention made me go from rating my life a 5 out of 10 to a 6 out of 10 — and that impact lasted for 12 months — then that would be one WELLBY that had been generated. Their hope is to evaluate charities on that basis, trying in every case to say how many WELLBYs are generated per dollar spent. GiveWell does, of course, care about subjective wellbeing of the people that it helps, but it’s more likely to cash out the effect of its grants in terms of doublings of income for a year, or lives saved, or increases in children’s weight, or benefits other than reported subjective wellbeing. First off, what do you think of the pros and cons of trying to use improvements in subjective wellbeing per dollar as a measure of cost effectiveness? Elie Hassenfeld: First I think it would be helpful for me to just explain what GiveWell is doing today, which is we cash everything out either in terms of increased ability to consume (i.e. people have more money) or reductions in disability-adjusted life years — some of which are health-related and some are mortality-related. But I very much take the point that subjective wellbeing is an important consideration. We don’t view the two outcomes we use today as the only outcomes that make sense. They’re just the two outcomes that we’ve been able to use to date. I do think over time, as we continue to grow and increase the size of our team, we’ll be in a position to include more factors explicitly in that analysis. I think the pro of subjective wellbeing measures is that it’s one more angle to use to look at the effectiveness of a programme. It seems to me it’s an important one, and I would like us to take it into consideration. I think the downside, or the reasons not to, might be that on one level, I think it can just be harder to measure. A death is very straightforward: we know what has happened. And the measures of subjective wellbeing are squishier in ways that it makes it harder to really know what it is. Also, I think some people might say, “I really value reducing suffering and therefore I choose subjective wellbeing.” I think other people might say, “I think these measures are telling me something that is not part of my ‘view of the good,’ and I don’t want to support that.” That would cause someone to want to leave it out of their calculus and the donations they’re making. In some ideal world, I would love for GiveWell to be able to offer options for donors who have different philosophical perspectives about what they want to achieve. Obviously, GiveWell institutionally also needs to have a view, because there’s funds that come to us directly. But ideally, in the future vision of GiveWell, for people who have subjective wellbeing as their core focus, other moral values, or maybe even a very different tradeoff between increasing income and reducing disability-adjusted life years (or increasing DALYs, maybe, depending on how you think about it), those are programmes we’d like to be able to bring to donors and let them choose. Because we’re not trying to add value by being particularly good philosophically. That’s not part of GiveWell’s comparative advantage. It would be better if we could, where donors want it, allow them to use their own judgements to make decisions. Rob Wiblin: Yeah, you’ve got to leave it to the philosophers to not have answers to those questions. Does GiveWell view its goal, or its primary goal, as being to increase subjective wellbeing of the people they’re helping? Or is it some more pluralistic or vague goal maybe, where you value saving someone’s life above and beyond or differently than just because it has allowed them to have more subjective wellbeing? Elie Hassenfeld: I’d say it’s overall a broader — maybe not vaguer, but a broader conception of the good. But it is vaguer in the sense that I can’t describe exactly what that is. Instead it’s aiming to, in some idealised sense, on one hand aggregate different conceptions of the good to enable no one to carry the day; or in another way, offer donors the opportunity to give in a way that’s consistent with their conception of the good. And one of those conceptions would be subjective wellbeing as the primary measure. Rob Wiblin: It seems like when making tradeoffs between the good done by doubling someone’s income for a year as against improving their health in one way or another or saving their life, you have to have kind of tradeoff ratios between these things in order to evaluate programmes that have these different economic and health impacts. It seems like even if subjective wellbeing isn’t the only thing that you value, probably in the background somewhere, there has to be some idea of the subjective wellbeing generated by these different interventions, and then you’ve got to do some rate of exchange between them. So yeah, is GiveWell kind of implicitly using WELLBYs to a point as some aspect of its intervention, even if it doesn’t call them that specifically? Elie Hassenfeld: I’m going to answer your question directly, but first I want to just give a point of context on this whole line of discussion around the philosophical judgements that go into these decisions, which is: There are some parts of GiveWell’s work that I would characterise as “mature” — meaning we have put a lot of time and energy into it. Obviously we might be making a mistake. We probably are in some ways, but we’re doing that work at a very high-quality level, because we’ve invested in it significantly over the years. I think this is an area — moral weights — where I don’t feel the same way. I don’t think this is a mature part of GiveWell. Instead, this is a part of GiveWell that has a huge amount of room for improvement — and quite frankly, one of the areas where I think we will improve as we increase the size of our staff, as we’re successful with recruiting, as new people come on board — is in this area. This is an area where it’s obviously quite hard and different than the rest of what we do. I’ll give you answers, but also want to make sure that when we’re having this conversation about GiveWell, it’s easy to assume that either we’re highly confident about everything we do or something, or that everything is at a similar level of maturity. And this is a part of our work that I think is… I don’t know. To put it differently, I feel much better about, say, the quality of the analysis, the depth, the number of times it’s been argued about and turned over, and just the attention that has been paid to an estimate of mortality for malnutrition — even though I know it may be very wrong — than I do about moral weights, where we haven’t reached that same level. So everything I say is at a shallower level of depth than maybe the other things we’ve been talking about so far. Rob Wiblin: Sure, yeah. So there’s that moral weights aspect. I guess there’s also an empirical question to some extent, about how much do the beneficiaries of these different charities think that these different charities are improving their wellbeing. I guess if you don’t have access to that data and subjective wellbeing is one thing that you care about — it’s one way of improving the good — then you might kind of implicitly have in mind, when you’re doing these tradeoffs in the spreadsheet, how much wellbeing is being generated by deworming or by malaria prevention. Elie Hassenfeld: Right. So then what’s actually happening? When we’ve tried to put our recommendations into a WELLBY framework, we literally have used the estimate from the Happier Lives Institute for cash as a way of getting a WELLBY associated with income improvements, which is a component of the programmes that we support. I also think that wellbeing is definitely in people’s minds when they’re trying to come up with the ratio of income to health. People are — whether it’s us or donors or the programme participants. We funded this survey by IDinsight to go around and ask people in Ghana and Kenya how they would trade off between different things. I think wellbeing is probably in their mind, but not elicited explicitly so far in the work that we’ve done. Rob Wiblin: One thing that occurred to me, reading through some of the Happier Lives Institute work this week, was just that I guess GiveWell is in a difficult empirical situation — where so many things that are very important for your cost-effectiveness analysis are hard to pin down; you don’t really know. I feel like the Happier Lives folks are in a much more difficult situation, because so many interventions that might be really effective at improving subjective wellbeing, even the studies of them don’t take surveys of subjective wellbeing. As you’re saying, even if they did, it’s harder to quantify properly using surveys like that than compared to measuring deaths, for example, which are far more concrete and less likely to be affected by exactly how you measure it. So just the number of studies that you can draw on if you’re strictly only going to consider subjective wellbeing is much lower. I think another thing that really bites is that subjective wellbeing outcomes are really at the end of the chain of all of these different factors about someone’s life — their income, their health, their education, their relationships, all of these different factors. If you improve one aspect of their life, like if you improve one component of their health by preventing them from getting malaria, if you were trying to detect the effect of that on some malaria-specific thing — like their iron levels, or their probability of dying of malaria, for example, or their probability of having long-term effects of having been infected with malaria — because malaria is the main thing, or a super important thing that affects those, you can see a much larger effect relative to the background variation in how much those things are changing over time and varying between people. But subjective wellbeing, being affected by so many other things, if you help someone with malaria, that effect could easily be really washed out by the time you’re looking at someone’s subjective wellbeing a month later or two months later — because there’s so many other things affecting that, creating much more variation. So in order to be able to pick up the effect of an intervention on subjective wellbeing, you need a much larger study than if you’re detecting some far more proximate outcome, like the effect of clean water on diarrhoea. Is there anything you want to add to that? Elie Hassenfeld: I think that’s exactly right. I mean, I think that it plays into some of how we’ve thought about some programmes that are explicitly focused on subjective wellbeing and thinking about their effects, and we can talk about that. But I broadly agree with you. I think that in some ways it’s a challenging metric to use. I think there is, on one hand, reason to notice that challenge and maybe take that into consideration as a point in favour that you’re just not going to find the same evidence. At the same time, you might say, gosh, it’s really hard to help people in low-income countries. Maybe there’s reason to give credence to the measures that are easier to deal with and easier to know that you’ve done something good and made someone’s life better. Rob Wiblin: Yeah, I’m expressing a lot of sympathy with the researchers at Happier Lives Institute because they have their work cut out for them. I think it’s a good challenge. Inasmuch as there are contexts where there are studies being done that are powered to detect an effect on subjective wellbeing that could start asking questions like this, maybe it would be worth getting more of them to do that and then see whether the development community as a whole can make meaningful use of them. Happier Lives Institute, HLI, it’s still young, and they haven’t looked into so many charities — I guess partly for the reasons probably we’ve been mentioning. But having looked at a few, HLI suggests giving to a group called StrongMinds, which tries to treat depression among women in Uganda, possibly some other countries as well. The intervention they use is a sort of peer therapy: building a community of people who are all suffering from similar mental health problems, who meet regularly and try to figure out how to solve their problems together. I think the recommendation rests on a couple of underlying ideas that their research has led them to conclude. The first one is that suffering from depression is extremely bad for subjective wellbeing. Another one is that group talk therapy can make a really big dent in people’s levels of depression: it can help a significant fraction of people recover sooner. Also, it seems like peer therapy is pretty cheap to provide per person, because one facilitator can support many patients in these sessions helping one another. Also, they think that reducing depression among women — in particular mothers, probably — has really big positive spillover effects to other people in their household, in particular their children and partners, I imagine. With that little pitch out of the way, is it possible to identify and break down the underlying reasons why GiveWell doesn’t recommend StrongMinds, or any other charity delivering that kind of intervention? Elie Hassenfeld: Yeah, totally. And I think we should do that. I think it’s really interesting, and I think it’s also illustrative of what’s really going on in the background, often, in what we are doing. I think it helps people understand GiveWell’s work, because often you might start and say, “Maybe the underlying difference is GiveWell doesn’t care about subjective wellbeing.” And it’s really not. I think ultimately what it comes down to is we have a different interpretation of the empirical data — meaning we look at the same empirical data and reach different conclusions about what it means for the likely impact of the programme in the real world. And maybe we could go through a couple of the most salient examples of those differences of opinion? Rob Wiblin: Yeah, definitely, go for it. Elie Hassenfeld: I do want to say, I’m not sure if I can be devil’s advocate on GiveWell, but you should certainly be, because obviously we’re just going to sit here and spout off on our take. And I think HLI has done a huge amount of work on this. This is a case where it’s not just that there’s uncertainty, but I think reasonable people can really disagree. Anyhow, I don’t want to be here kind of spouting off on GiveWell’s take with any notion that I think that we’re definitely right or something like that. Rob Wiblin: Totally. Elie Hassenfeld: So look, there’s these papers, and there’s a lot of different variables that go into coming up with a cost-effectiveness estimate. James Snowden wrote a post on the EA Forum recently about spillover effects and his disagreement. I think there’s a lot to go into, but I just wanted to focus on two, and I think these two factors can start to explain a lot of the difference. Factor number one is that the way the measure in the trial is is to, after the fact, go back to the treatment and the control group and ask them about their depression symptoms. There’s something that often is referred to as social-desirability bias, which is people responding to the surveyors by telling them what they want to hear. In this case, you can’t do a placebo controlled trial: like, you know whether you participated in group therapy or not. I think this creates a real challenge in interpreting the measured result in the study, in terms of its meaning for what you actually believe is happening in the real world. Meaning that we know what the treatment group and the control group responded to the depression questionnaire. What we don’t know is how much that has been biased by an expectation that they should respond in a certain way. I’m looking at our cost-effectiveness model, and we apply an 80% adjustment to that factor. We say, well, let’s just assume that we’ll downgrade the result by 20%. That’s what we assume. But I mean, honestly, I don’t know. Should it be 90%, should it be 10%? I really have very little sense. But I think HLI has it at 100%, meaning they just don’t adjust for that. They don’t discount for this factor. This is just one difference, and I can go through the other one, but before I do that at length (which is not necessary): if you have two factors, each of which you adjust for with a multiply by 0.7, you’re halving the impact right there. I think this is a big part of the reason why in general, GiveWell often has cost-effectiveness estimates that differ from maybe the organisations that come to us. Because we’re looking at a very wide range of interventions, something that we’re always trying to do is adjust for offsetting impacts, or how the results of the studies generalise to the real world, or ways in which the measured results in the study might not be representative of what you would get if you ran the study today. That has this series of adjustments, and we’re trying to make those adjustments consistently over time, but they have fairly big effects on the bottom-line numbers for the programme. Rob Wiblin: OK, so I suppose there’s various sources or various different studies of this kind of peer therapy. I think if you’re being really thorough at the cutting edge of analysing these sorts of things, there’s ways of trying to address this social-desirability bias to make it much smaller and get results that you maybe trust a bit more. Is it the case that maybe the studies that have looked at this haven’t been quite at that level? Or maybe you think they weren’t able to fully account for that, and try to have an equal amount of social-desirability bias between the treatment group and the control group? Elie Hassenfeld: Yeah, I’m not sure. My understanding is no, but I think it’s a good question. Rob Wiblin: Yeah, I totally understand. You can’t come across every single detail of everything. Do you want to describe the second issue? Elie Hassenfeld: Yeah. The second issue is how you would expect this programme to operate at scale, and how its effects would exist at scale relative to its effects in a trial context. This issue seems intuitive to me. If I’m a researcher and I want to study the effects of group therapy on depression, I’m very likely to ensure that the way the programme is conducted in the trial is maintaining a high fidelity to my model of how the programme works. As this programme reaches scale, that’s just no longer possible. Organisations grow and it’s harder to maintain a level of quality as they do that. We try to adjust for this. One of my favourite examples is a malaria net study, just one of many, where researchers were checking whether or not people were sleeping under nets. Every morning, at like four in the morning, they were literally going to people’s houses and checking. That was not common, but that’s the type of activity you might expect in a randomised trial that would not be possible at all. There are obviously other stories. The best GiveWell example is No Lean Season, which was an organisation we recommended several years ago. Not going to go into the details now; I think it’s been discussed. But it was a case where there were several randomised trials that showed a big effect, and when it was scaled up, it didn’t have the same effect. And this wasn’t because of anything weird or pernicious particularly happening, but when you tried to go from small context to large context, it changed the way the programme was implemented and it wasn’t as effective. I think it’s an interesting story — but that’s another one where we adjust: our adjustment is multiplying by 0.75, HLI doesn’t multiply by something, and so therefore you end up with another reduction. Again, I don’t have a great idea of how much to adjust. So those are two factors. Happy to talk about those factors, but I think another thing that would be interesting to talk about, just to flag it, would be: How do you start to think about where you want to be on this spectrum? Maybe near 100% adjustment, or multiply by one, which would be no adjustment — all the way down to multiplying by some very low number. As you probably know, in the case of deworming, we’re multiplying in one case by 0.1 — 10% — because we’re trying to adjust for a huge amount of uncertainty and strangeness in that result to arrive at what we think is a better, more accurate estimate. Rob Wiblin: It sounds like the fact that these kinds of adjustments — multiplying by 0.8 or 0.7, and then doing that a few times — that that is decision-relevant here suggests that it’s at least kind of in the ballpark of some of the charities that you’re recommending. You think it’s an interesting option, but it just hasn’t quite made it? Elie Hassenfeld: I think it’s an interesting option. It hasn’t quite made it. Like, if you asked my best guess right now, I don’t know. My personal best guess would be about one to two times as cost effective as cash transfers. But also there’s a lot of disagreement internally. There’s some people who would probably double that number. So yeah, it’s in an area where additional research could change our mind. I believe there is additional research coming out. There’s a study that’s being conducted that I think is registered in the registry of clinical trials that may even be out by the time this show airs. So that would be fun; that should give us some updated information. But yeah, certainly potentially in the range of possibilities, and it comes down more to empirical questions than philosophical ones. Rob Wiblin: Yeah, that’s really cool. I’m interested to see what comes of that trial. I know HLI has had various other posts where they’ve gone deep on some of your cost-effectiveness analyses and suggested some improvements that they think that you could make. Maybe could you point to one of those that you agree with that’s been helpful? Elie Hassenfeld: Yeah, they went extremely deep on our deworming cost-effectiveness analysis and pointed out an issue that we had glossed over, where the effect of the deworming treatment degrades over time. We had seen that degrading, and the way we had treated it, I should say, was that that’s just a noisy estimate, and we just took the average estimate persisting over the long run. Their critique convinced us that we should at least incorporate some probability that the effect is degrading into our overall model, and that shifted our overall assessment of deworming down by a small amount. Had we taken their correction on board in the past, it would have meant a few million dollars that we would have given elsewhere instead of deworming. Their published critique, I think we didn’t agree with the headline result that they reached, but we were really grateful for that critique, and I thought it catalysed us to launch this Change Our Mind Contest. And also it was a great example of the engagement that we’re getting from being transparent. That we can say, “Here’s our decisions, here’s why they could point to an error, and it changes our mind.” That was really cool, and we were really grateful for it. Rob Wiblin: That’s wonderful. Of course we’ll stick up links to all these articles if people are curious to learn more. I’m curious to hear your reaction to one other argument that the Happier Lives Institute has made about the goodness of saving or extending lives versus improving wellbeing while people are alive. I’ll have to explain their views here first, so bear with me. I’ve got to bring all the listeners along. I think it’s fair to say that a thread that runs through HLI’s thinking is that you and maybe other people are overweighting the value of preventing people from dying relative to the value of improving their wellbeing, holding constant how long they live for. One of the first reasons they offer for this, for why this might be true, is actually maybe it’s not so bad not to exist. Because it’s not bad to be dead while you’re alive, because you’re not dead. It’s not bad for you to be dead while you’re dead, because nothing can be bad for you while you’re dead. This is the so-called Epicurean argument about death. I’m going to skip over that one for time here, though. People who are interested can find some discussion of it in episode #86 of the show: Hilary Greaves on whether existing can be good for us. Another philosophical argument that one might make here — that I’m not sure whether HLI has made, but I’m more intuitively sympathetic to personally — is just the intuition that one has more reason to prevent someone from suffering than one has reason to enable them to have even more positive experiences than they’re already having. I think many people find the idea of giving some priority to preventing suffering over making people’s lives better to be an appealing notion. In some situations, that could potentially lead you to prioritising improving lives over extending them. I’ll also pass over that one in the interest of time, and also because it’s more philosophy, which is a trickier area. The third thing — which I do want to talk about — is the observation that saving someone’s life is less valuable if that person’s life is going to be unpleasant or very difficult. I think to most people, it’s intuitive that it’s more valuable to save the life of someone who feels that they’re really flourishing and is super glad to be alive than it is to save the life of someone who thinks their life is barely worth living, who maybe doesn’t even care that much whether they live or die. It could be useful to use some numbers to make it a bit clearer how this might end up affecting your relative priorities here. If you imagine people scoring their quality of life out of 10, that’s kind of the standard subjective wellbeing scale. Let’s say that we use the number 3 as the number at which someone is rating their existence as neutral, with the good and bad things in their life cancelling out: that’s kind of a typical answer for what people say would be the neutral point for them if they were scoring themselves. If someone is going to report a quality of life of 4 out of 10 for the rest of their lives, then from a wellbeing-adjusted life year, a WELLBY, point of view, then it’s equally valuable to them to prevent them from dying as it is to increase their wellbeing permanently by one point out of 10. That would be from 4 to 5 in this case. On the other hand, if someone reports a quality of life of 5 out of 10, then from a WELLBY point of view, it’s twice as valuable to save their life as to increase their wellbeing permanently by one point — in this case from 5 to 6 — because the difference from 3 to 5 is twice as great as from 5 to 6. HLI notes that many people in very poor countries — who otherwise might die of malaria in the absence of additional antimalarial bednets — have unsurprisingly pretty challenging lives with plenty of hardship in them. That, as I understand it, suggests that to them it’s more likely to be cost effective to make people’s lives better than to make them longer or less equal. So that’s a very long lead-in. What do you and GiveWell make of that line of argument? Elie Hassenfeld: I think the place I want to start is this is a case where I feel most strongly that I would want to hear from the people themselves in low-income countries about this topic. Because if you kind of draw out this line of reasoning, it leads you to the conclusion that there is a very high proportion of people living in low-income countries who would choose death over continued living, based on their self-reported life satisfaction. That’s a very uncomfortable conclusion, but maybe more importantly, one that is so counterintuitive that I feel the need to follow up on it before accepting it at face value. That may be a somewhat minor point about where you draw the line on the scale, but still, in this case, I think the maybe purely emotional urge I have is to say that doesn’t quite seem like it could be right. Intellectually, I know it could be right — therefore I need to follow up on it, because it’s so inconsistent with my starting point for what people would say. Rob Wiblin: Yeah, it definitely can get uncomfortable or weird. Or maybe if you were surveying people on their subjective wellbeing, and you really said, “If you score yourself a 2, we’re going to take it that you actually mean that you would rather not be alive right now,” then maybe people would reassess. Because an interesting thing is that when you survey people, almost everywhere in the world, even people in serious poverty almost always say that they think their life is better than not existing, and they want to continue surviving and so on. I’ve heard some philosophers say that that kind of intuition that we all have about how great it is to continue existing might be suspicious, because we might have evolved to have that attitude. We necessarily almost have to evolve to have that attitude, even if our lives are very unpleasant. That kind of bias might affect all of us. But I’m not really too keen to go there, and I feel extremely uncomfortable. If someone says that saving their life is really valuable, I’m inclined to take that at face value and to trust that over some subjective wellbeing survey. Elie Hassenfeld: Right. I think that discomfort is a good starting point, though not an ending point. Certainly something that we are very committed to internally — one of our company values or whatever you want to call it — is truth-seeking. What we mean by that is we’re going to have the hard conversations, and keep digging to try to get the answer that is correct, as far as we can see it. Therefore, in this case, I would say I am very suspicious of philosophising and reaching a conclusion that seems extremely counterintuitive and then running with it. But we’re a place that wants to go deeper and be open to strange conclusions. Or maybe I should say it differently, like: conclusions that seem strange to us today that will not seem strange to us in the future once we’ve spent more time with them and done more research on them. Anyhow, concretely, if we had more time and capacity, we’d be going further — because I think there’s important questions here. I think in our actual analysis, we’ve used what’s called the “deprivation” approach: We basically assume that if you’re alive, you’re getting the life satisfaction points of the representative person at your level. If you die, you lose those life satisfaction points. Not the philosophical view that death is 0, but death loses those points. That’s kind of how we’ve approached it. And then I think we — if I remember correctly, though this may be wrong — I believe we set the bar at an answer of 0.5 on life satisfaction points as where someone would choose to not be alive than to be alive. Rob Wiblin: This model can feel like maybe it’s going wrong at the extremes. At least for myself, let’s say that I have some preexisting sense of the tradeoff that I would intuitively think that one should have between saving lives versus improving them for people here in the UK. And then say that I read some new research suggesting that actually people don’t like their lives nearly as much as I thought, and the typical person is actually struggling much more than I appreciated. I think on the margin, that would shift me towards thinking that we should focus more on improving lives than we should on extending them. There’s maybe a question of like, is this empirically a large issue? There I’m not sure. But conceptually, I think it’s hard to get away from this making sense at some level. Elie Hassenfeld: Completely. I think that’s completely right, and I think one of the things that HLI has done effectively is just ensure that this is on people’s minds. I mean, without a doubt their work has caused us to engage with it more than we otherwise might have. Similar to some of the questions you were asking earlier, like, “Why doesn’t institution X see that it should do whatever?” Well, because it’s kind of hard, and sometimes you need another organisation to be pushing it in front of you. I think that’s really good that they’ve done that, because it’s clearly an important area that we want to learn more about, and I think could eventually be more supportive of in the future. Rob Wiblin: I thought that one possible way that you and Happier Lives Institute could end up making different recommendations might be that if they’re thinking about immediate-term impacts on people’s reported subjective wellbeing. I could imagine GiveWell thinking more in terms of economic development, and thinking we want to have interventions that are going to cause people to become wealthier and less likely to be in poverty over the long term. In part we care about improving people’s health because that enables them to be more productive and earn more and not be in such dire poverty later on when they’re adults, and to raise the next generation to be healthier and so on, and to just generally speed up the process of economic development. Is that an important factor in your recommendations that could cause you to recommend some things rather than others? Elie Hassenfeld: The way I think that is actually mechanically flowing through our analysis is via those moral weights. When people are thinking about the value of, say, doubling someone’s income for a single year, I think in the background they are thinking of the types of considerations that you just mentioned — that is really the way that that plays out. There is a part of thinking about the health interventions that we support where it doesn’t directly flow through to the model. But the fact that some of these huge improvements in welfare over the last 50 years have come from large-scale global health programmes to reduce mortality, that sort of angle on the problem, I think, provides some additional credence to those interventions and their effectiveness — the type of funding that we’re most known for. All of this is like a very zoomed-out philosophical level. I think it’s important to frame the argument in this way. But then also I think you can boil it down into some of the very concrete differences about how do we literally interpret the evidence? How do we forecast or estimate the effect of StrongMinds’s particular programme? What are your assumptions and beliefs about how that programme would affect people’s lives? I think the main point I would argue is that I think that carries 80% of the weight in the judgement that one would make. The judgements that we’re trying to make, and much more of what we’re doing, is trying to ask the question: What are the state-of-the-art ways of reducing social-desirability bias, and should the estimate be more 95% or more 5%? That’s, I think, where a lot of our energy is going — and it’s one example of something that we’d be able to be tackling more effectively with more capacity. Rob Wiblin: OK, let’s move along from subjective wellbeing and instead talk about an entirely different critique, which is related to what you were just saying. This is a critique that argues that what people focused on helping the very poor should be trying to do is to raise economic growth rates specifically, and that the best way to do that probably looks more like shaping economic policy in poor countries by funding think tanks or economic research or something than it does like funding basic global health interventions. This one was, at least as far as I know, most clearly and forcibly presented in the post “Growth and the case against randomista development,” by Hauke Hillebrandt and John Halstead back in 2020. Of course we’ll link to that, and I think it’s a pretty interesting read. I don’t think we’ve ever presented this view at any length on the show before, so I feel a bit bad about my little monologues here, but I’ve got to explain it in a little bit of detail so people understand where it’s coming from. I think the key claims from Hauke and John, and economists who agree with this line of reasoning like Lant Pritchett, they would say if we look at where people are dying young, and not getting educated, and generally struggling, or maybe not flourishing nearly as much as they could, it seems like the single best predictor of that is low GDP per capita. It’s not everything, but it’s a super important predictor of life expectancy and reported wellbeing and so on. Basically, back in the 18th century, everywhere in the world was poor by today’s standards — almost everyone was living in poverty. Now there’s a whole lot of countries that are not poor, that are not struggling with these really easily solved preventable illnesses very much and so on. The countries that are no longer poor basically have gotten there by figuring out how to produce a lot of goods and services per person. As a result of that, in a country like the UK, very few children are severely malnourished, just because there’s a far greater abundance of resources. And how did this come about? Countries that are kind of now globally rich, like the UK, got to the level of richness that they’re at over decades and centuries by gradually becoming more economically productive. So GDP per capita was high and people could earn decent salaries and they’d be very unlikely to go severely malnourished. From a historical point of view, what actually kind of changed mechanically in so many countries that are now middle- or high-income so that they could have such radically higher living standards than people did hundreds of years ago? Basically, the people who buy into this critique will ask the question: Did it happen because of the sorts of health interventions that GiveWell funds — like getting rid of malaria or deworming kids or all of the things that we were talking about earlier? Their answer would be that while those things probably made some small amount of difference, it’s much more that they’re not probably the major cause of development. And in many cases, randomista aid or charity or some of these health interventions didn’t even exist during the primary development era, because countries like the UK were getting rich in the 19th century and in the first half of the 20th century, before many of these problems even had been solved, and before we had the methodological approach to run randomised trials, or we certainly weren’t doing it in this context very much. On top of that, while GiveWell’s charities probably do improve economic growth by improving health, they weren’t chosen on the basis that they were optimised for that. It would be kind of surprising if they really were the best way to do it. In their mind, something else is causing countries to get rich, and it should be super important to try to figure out what that is and try to stimulate it. And the people with this critique would, I think by and large, say that the likely top causes are more often related to changes in economic policy or the legal systems in a country, or the physical infrastructure that’s available, perhaps the culture, the kinds of things that people do with their time. Those fundamentals allowed productive businesses to be built in these countries over time, and thereby we found ways of dramatically improving how people are coordinated and the kinds of technologies that they were using. If that’s right, the most impactful interventions are likely to focus on that sort of thing — doing think tanks to advocate for removing barriers to economic growth in a given location, for example. They point out that if you can help raise the economic growth rate of a poor country for a couple of decades, that is going to be as useful as sending them enormous amounts of money in net present value terms. We can just look historically and see that we get very large decreases in poverty when countries go through serious economic growth spurts. Now, we might not know exactly the formula for economic growth that works everywhere, but people with this critique would say that we definitely have some ideas about approaches that are better than others, and policy in many poor countries has very obvious problems that are holding it back. Maybe we don’t have the perfect recipe for economic growth that’s going to be super successful everywhere, but we definitely have ways of preventing big unforced mistakes that are sending countries backwards or holding them back quite obviously. And that sort of mentality, by tackling the root causes of poor people’s misery — that is, their countries being really poor and unproductive — that could be way more cost effective than the sorts of person-by-person or hospital-by-hospital health interventions that GiveWell spends the most time thinking about. I apologise to any advocates of this view if I’ve oversimplified it, just to get it to cram into a couple of minutes. But with that out of the way, at a very high level, what do you make of this alternative lens on things? Elie Hassenfeld: I think I want to start with the parts of the critique that I take on board, and what I think we would ideally be doing differently, but then move into the critiques of the critique that I have and where I think it maybe is overstating its case. The part of this critique that I really like, and I’ve been thinking about recently, is that I don’t think that we at GiveWell have put enough time into finding ways to explore the space of possibilities in this area, given its potential importance. I think that is something that I don’t regret historically — I’ll tell you why — but I do think going forward, as we’ve grown and as we continue to grow, I’d like to be in a position where we’ve explored this enough to have a really great answer, which either is we’re doing this in this area or we’re not, because of this pretty compelling reason. I think one of the things that explains GiveWell’s history, largely, is that GiveWell did something very unique by going very deep on charitable interventions and understanding them very well. A lot of how we’ve grown is by sticking to that core pretty intensively over a long period of time, while we expand out in many of the ways that we’ve talked about today. I think in some ways that is our greatest institutional strength and maybe our greatest institutional weakness. We’ve been very focused on maintaining quality and rigour, and that has been very hard as we’ve grown a lot. I think we’ve been successful at it, and also it has made us more deliberate in the approach that we take to things — and I think that’s a fair characterisation of GiveWell. So when there have been ideas that are more outside of our bailiwick, I think we’ve been just less effective at engaging with them. Just looking at the trajectory we’ve been on in the last three years and how we’ve expanded, when I look out five more years with our growth, I think we will be in a much better position to be engaging more seriously with these ideas. Maybe that’s the institutional critique and what I think we could do differently. But I’m happy to move on and engage more substantively with the ideas. Rob Wiblin: Yeah, totally. Maybe first, have you looked into any organisations whose focus is directly improving economic policy or any organisations where you’ll cash out their impact in terms of increases in GDP growth rates? Elie Hassenfeld: Yeah, we’ve thought about this issue a little bit. We’ve spoken a number of times with Lant Pritchett, who is a leading academic proponent of these ideas. We’ve looked for specific organisations we could support that are focused on growth specifically. I don’t remember the details off the top of my head, but what I do remember is just not coming away from those cases feeling very optimistic about the results we’d get back, or even the information we would gain to be able to learn. I used this sort of a maturity spectrum before — where certain things are more mature, and certain things are less — and this was definitely fairly shallow, fairly immature work. But we looked at that level and we’re not super optimistic about the opportunities that we considered. Rob Wiblin: Interesting. Is it maybe the case that there’s just fewer organisations who perceive this is their goal, this is their direct mission in the developing world, relative to how many health-related organisations there are? Elie Hassenfeld: Maybe, but I think it’s also a question of how you would attack this philanthropically — like I also wonder how neglected this space truly is. There’s the World Bank, IMF, there’s other institutions. There are the Washington think tanks that are definitely focused on economic growth, and academics who focus on macroeconomics and how we can improve low-income country conditions. Now I’m going to step out of my lane and say my best impression of it, though I’m hoping that I’m not horribly wrong: I think many of these ideas obviously and almost certainly contributed to economic growth to some extent. Also my understanding is that some of the understanding of what led Southeast Asian countries to come out of poverty may be less clearly and directly tied to sort of standard broad principles of what makes free trade as like a pillar, or the Washington Consensus — which was I think the jury is out on how effective this consensus of policy was. I think that what makes this more complicated to me — and again, I take the critique on board; shallow investigation of an important idea is insufficient — but what makes the critique sting less substantively is that I think my story would be more that we do know some things that countries should really avoid. I think people are working on that. This is not a totally neglected space. Then there is some degree of disagreement among the different groups working on this about what the right approaches are. Some of the evidence for that is a disagreement about the extent to which past efforts have been effective and to what extent. I do think — again, this might be personally and temperamentally more than something else — but it strikes me that there’s more of a risk of doing harm here, by assuming that we do have the answer and pushing economic policy in a certain direction. There’s just a lot of opportunity for unintended consequences of pushing countries to do things that are different even if we knew how to do it. All that said, I do think the critique still stands, because ideal GiveWell would have said, “We spent a year on this, because it’s an important idea,” or we funded, I don’t know, John and Hauke to go look into this for a while to come back with better ideas.” I think the high level is insufficient for the weight of the argument. Rob Wiblin: Inasmuch as you do disagree, what do you think is the biggest weakness with this line of argument? Are there any other ones that you haven’t mentioned? Elie Hassenfeld: No, the three I mentioned are the ones that seem most salient to me. I think the most important one is that it’s unclear to me that there really is a consensus about what should be done. There are some basic principles that should be followed, but when you look at the actual track record, I think it’s more muddy. Number two is it’s not neglected, as far as I know: look at all the groups I mentioned before that are working on it. And then finally, the risk of harm seems high. I also think there are some more detailed things. I think Alexander Berger actually raises this in the comments, that when you’re modelling the effect of GDP, you might want to look at log GDP instead of straight GDP. That is a big effect as you look at the big benefits of the very big numbers. I suspect those are smaller, in terms of their effect on the overall argument, but important nonetheless. Rob Wiblin: Yeah, just to clarify that last point: in some of the work on this, the benefit is reported in terms of just net present dollars of GDP increased. But of course, $1 is far more valuable if it goes to someone in extreme poverty. And as a country develops, fewer people are really poor, and so the value of each extra dollar of GDP generated goes down. You have to use this logarithm of each person’s income in order to get a more realistic translation of how that affects wellbeing, and that’s complicated. That’s like one adjustment that you would want to keep in mind. I could imagine, as I was describing this worldview, some listeners both being like, “Yes, this is exactly it; finally someone saying it,” and also people being like, “What is this absolute tosh? As if we know the answer to how to cause economic development and we can just go in and tell people. It’s far more complicated than that.” I’ll try to represent what I’ve read from this view. I’ll do my best and see what you think. I should say that all of the things you mentioned are somewhat addressed in that blog post, although not to the full satisfaction of anyone, I don’t imagine. I think they would say, at least in the cases of countries going backwards massively, we know things that countries shouldn’t do that quite consistently lead them to have economic disasters — like causing hyperinflation is one of them. They might say, even if we don’t know what the very best policy is, we at least know some things that are clearly bad, and maybe more effort should be put into preventing those, given how catastrophic they are. Do you want to react to that one? Elie Hassenfeld: Off the cuff, they also seem like the countries that are hardest to influence. If it’s so well known, then why are they doing it? Well, they’re probably doing it because leadership in the country does not have their population’s best interests in mind. That seems like quite a challenge for philanthropy to address. Rob Wiblin: Yeah, I think that’s probably my biggest concern with this line of argument, which in general I’m quite sympathetic to. Like you, I think there’s a lot to it, but I feel that often it’s not appreciating that there’s reasons that countries have bad policy. Very often it’s not merely just a mistake; it’s because of the political settlement within a country and who has power. And coming in and telling people that they could be richer if they change their policy one way or another — the elites often don’t want to implement those policies because they think it would weaken their position one way or another, or at least they’re not suffering from the poverty. There’s this whole other angle of political economy, trying to understand how countries end up with the policies that they do, given how the political system works. Elie Hassenfeld: That’s why I think ultimately, where I think GiveWell has something to add to this conversation — many of the conversations we’ve had — is to say that we can look at it from the 10,000-foot view or the 50,000-foot view. That’s important because it can help us decide where to put our resources. It’s hard to figure out what’s true from such a high level. I think to some extent what makes me really excited about our work, why I think it’s really cool, is that we’re trying to be good about thinking at the 50,000-foot level, but then dig all the way in and ask: What can we do in this case about this problem? When I think about this specifically, I have absolutely no idea what to give money to to improve economic growth in country A, B, or C. But I can imagine a proximate step of finding people to spend time on this for a while and see what they come back with. Having watched a lot of different types of programmes over many years — from GiveWell, from Open Philanthropy — more research often leads to new ideas. And so we’re excited, I’m excited, about our opportunity to support work like that, because it can bridge this gap between very abstract arguments — where there’s good arguments on both sides — to find opportunities to actually move things forward. Rob Wiblin: Another line of argument that I think advocates for this view might put forward is: Yes, there is controversy within development economics about exactly which policies are best, but to some extent that’s maybe overstated among laypeople. In fact, probably there is a lot of agreement: among development economists at least you might get like 70% or 80% on board with various different structural changes that you could make to economies — and that’s kind of the level that you expect with social science of this type, where the questions are quite complicated. Yes, those folks could end up being mistaken, as surely people have advocated for mistaken policies in the past, but it’s better than chance. And if you’re willing to take a risk, then maybe it is worth trying to push at least some countries to try to go closer to the general consensus, or at least to adopt the best practices that are agreed among a reasonable majority of experts in the space. Elie Hassenfeld: I think that’s probably one of the ways that this would end up cashing out. You’d say: Here’s the policies that should be implemented, and here’s the countries that should use the policies. Then try to understand why they’re not or who could, and then see what comes of it. I just don’t know enough to really engage on that substantively. But I think it makes sense as an area for investigation. Rob Wiblin: They might also put forward the argument that, yes, there’s the IMF and there’s the World Bank and there’s other economic groups, but they’re all somewhat restricted, or at least the kinds of behaviour that they engage in is limited by the fact that these are multilateral international organisations run by technocrats. Maybe there’s space for a more advocacy-focused international organisation, that goes around to countries that maybe are about to do hyperinflation and campaigns against that, or tries to raise awareness about it. Or there could be alternative models here that aren’t getting funded. I’m not sure that they would make that argument, but that’s something that jumped into my head. Elie Hassenfeld: I think I largely would say similar things to what I’ve said in response to this, which I know is maybe somewhat unsatisfying. Maybe the only additional point is that on the flip side, you’re just choosing to put money to that relative to all these other really great things that we were talking about earlier. That’s not a sufficient argument to say malnutrition or water or Kangaroo Mother Care is better. But certainly these programmes are doing and have done a huge amount of good. I think the pressure that we face, the stress, the job is to ultimately be making that choice where there’s limited resources — trying to increase the resources, but there’s limited resources — and then try to direct them where they can do a lot of good. There’s maybe largely, unfortunately, still too many places where it’s easy to do a lot of good. Rob Wiblin: Totally. Yeah. OK, we’ll wrap up this section in a moment, but before we move on, there was a particular point in that article that I think really helped to explain what was motivating people to really prioritise thinking in this way, which I didn’t want to let go. The quote is: The American Economics Association has 20,000 members. Assume there are twice as many economists globally costing around $150,000 each – at a total of $6bn. […] China’s growth acceleration from 1977 onwards produced $14 trillion NPV in cumulative economic output. Thus, if the only thing the economics profession achieved in 50 years was to increase by 4 percentage points the probability that the Chinese government shifted to this new economic strategy, then it would have had greater economic benefits than the Graduation approach. [Which is this alternative, non-policy-focused comparison that they use throughout and explain] It is implausible that the economics profession had an influence this small, and there is in fact a lot of evidence for substantial development economics influence on Chinese economic thinking at this time. — and they give some other examples where they think the economics profession has influenced countries to have big growth accelerations through policy. Yeah, that just really stood to me as I was reading this article again this week, and made me think that I would love to have more researchers looking at this kind of thing, even if it might be quite hard to find a specific project to fund. Elie Hassenfeld: Yeah, I guess I’m not totally sure what to make of that comparison. Yes, the discovery of basic principles of economics and their acceptance has had a huge effect. I mean, you could say something similar about, I don’t know, basic discoveries in medical science. And then I don’t quite know what to do with that in terms of making a decision about what deserves further funding. Rob Wiblin: Totally. OK, so we’re nearly at the end here. I think people who have been really interested in this conversation and have some flexibility in their career should think about working at GiveWell if they find these issues really interesting. By the time this interview comes out, the specific vacancies that you’re advertising now might be gone, but what roles are you trying to fill in general this year and next year? Elie Hassenfeld: Yeah. GiveWell is broken up into three teams: researchers who are trying to figure out where to direct money, outreach people who are trying to increase the amount of money we can direct, and then folks in operations who are supporting the organisation’s functioning. We’re trying to build all three areas. In research, we’re hiring researchers across the board. Senior researchers are intellectual leaders; they would be someone who I would go to and say, “We really need to look into this growth case. What could we do? And what do we think?” A senior researcher would just take on that project and try to explore it, understand it. The conclusion could be, “This is not promising, and here’s why.” Or the conclusion could be, “We should give money here, and here’s why.” Those senior researchers do everything from broad exploration to going deeper on questions, like the evidence for malnutrition, scoping studies to help us learn more, and everything in between. Those senior researchers tend to be fairly far into their careers, I guess maybe eight or 10 years or more. But we are also hiring more entry-level researchers and people earlier in their career who can come in and support that work, and then our hope is to develop over time to take on more responsibility. On the outreach side, the goal is just to raise more money. That has various components, but one of the largest ones is understanding GiveWell’s research well enough that you can be an effective advocate in conversation with donors. A large proportion of the funds we raise come from people giving a million dollars or more every year. Therefore, a lot of what we’re doing is giving them confidence that we’ve done a good job. Often that comes from direct conversation with us, even though there is so much information on our website. So we’re aiming to hire a larger team that can go even further in those conversations with very large donors. There’s other things we’re doing too, but that’s kind of the top goal. And then our operations team is very small for an organisation of our size, and there’s a lot that we want to do there. I think earlier I mentioned doing a better job investing our assets. We’re not looking for people who have investment expertise; that’s not what we need. It’s generalists who can bring common sense and an attitude of getting things done to the team, because we expect to grow a fair amount into the future and we want to support our staff, and support the organisation and its growth, so we can continue to be effective going forward. Rob Wiblin: What sort of distinctive skills or attitudes might make someone a good fit for each of those different categories? Elie Hassenfeld: I’ll mention research first, but then talk holistically about broad categories or broad qualities. In research, GiveWell sits at some midpoint between very practically minded, practically oriented organisations — sometimes I think of the prototypical management consulting firm or something, who are trying to advise clients directly on what to do in their business — and on the other end of the spectrum, maybe an academic doing deep abstract research. We’re trying to find people who combine those two skills: both an inclination towards academic-style research with a practical orientation that wants to put that research to work. This is unusually hard to find. I think that’s because people who really like getting all the way into the details often want to keep going, and people who like getting things done often have no patience for the details. The people who’ve been most successful for us, who we are really looking for, bring those two abilities together to help us make great decisions. And then I think there’s some other basic characteristics that are somewhat unusual but make someone successful at GiveWell. Probably the two most important ones are, first, maybe in a very conventional way, someone who is really great at moving things ahead. I think that often people can get stuck doing too much analysis or trying to find the perfect answer, and often an imperfect answer is actually good enough for our needs in a domain. People who have that inclination towards “I check this off, I want to do the next thing” while having everything else are really great. I think also, importantly, people who are just excited to share their work and get feedback. A lot of folks, you say, “Answer this question,” they want to sit in a corner for three months and bring back the A+ paper — and that does not work. But people who are excited to say, “I think this, and this is how I could be wrong, and please give me feedback so I can improve,” I mean, wow, those people are hard to find but amazing. If this sounds like you, then you should definitely come to GiveWell. Rob Wiblin: I’m tempted to ask what’s distinctive culturally about GiveWell, but I think it kind of shines through the entire conversation. I suppose it’s all of these things of wanting to quantify things. Do people in general have conversations at GiveWell that sound like this interview? Elie Hassenfeld: Yeah, I think a fair amount. I mean, the things that are obvious are we’re into transparency, we’re into trying to maximise the impact of our work, and quantifying things: we’re trying to get to the bottom of it. Something that is maybe not so evident from the outside, that people say when they arrive — I don’t know how this will sound — but that people at GiveWell are very kind and considerate. And I think that’s really important. I’ve worked in other places and seen a lot of places where a focus on quantification and transparency and truth-seeking can lead to a willingness to be mean. And I just don’t think that’s good. We fight back against that — both with trying to be considerate of others internally, considerate of the organisations that we are considering for funding, and the people that they’re ultimately benefiting. I think that that is a very important counterweight to this collection of characteristics that often go together, which is maximise, get the truth, and just be open about what you think. So we fight back against that a little bit, with that focus on consideration. Rob Wiblin: Has that been a very conscious thing from the beginning, maybe, to try to push for kindness and make that important hiring criteria? Or is it something else? Elie Hassenfeld: It’s something else. It happened organically. I noticed it when we would ask… When people join, we check in with them after six weeks, and someone sits down and asks them a series of questions. One of them is like, “What were you surprised about?” or “What do you like most?” And they would often say the kindness and the consideration that they received from their colleagues. I don’t know exactly why that happened, but I’m very glad that it did. Now, it’s not a hiring criterion, because there’s absolutely no way I think we could realistically… No one wants to hire a jerk, and there’s no way we could assess that. If we take referrals and someone’s a jerk, we won’t hire them. But it doesn’t come up. Instead, it’s just a big part of the message that we’re trying to send along the way, and I think we’ve done a good job maintaining our commitment to that value as we’ve grown. Rob Wiblin: Yeah. Only half the audience of the show is in the US. Are you able to sponsor visas for people from outside the US? Elie Hassenfeld: Yes, we are. For many roles, not for all roles. We also have folks who are based outside of the US. We have a couple of people in England outside of London, and generally I think we can employ people outside of the US as long as they’re within maybe the US Pacific to the European time zones. As it gets further afield, we’re not sure we can actually make the time zone shift work. Rob Wiblin: I see. The time zones for Australia are challenging. I should say you’re based in San Francisco. We didn’t mention that earlier, but it’s great you’re hiring remote workers in these other places now. I suppose that involves visiting the US periodically, but day to day, you work from Europe, say? Elie Hassenfeld: Our formal headquarters are in Oakland, California, but only about a third of staff are now based in the Bay Area, because we hired aggressively throughout the pandemic, and we were hiring remotely. So yeah, we’re very happy to hire folks outside of the Bay. Many people are based outside the Bay, and then we encourage people to come visit us quarterly during a week that everyone comes to the office — but that’s not mandatory, and some folks have chosen not to come, and that’s OK too. We feel like we can successfully employ people pretty much anywhere, though the further you get outside of the US time zones, I think the more difficult the experience is, because you spend more time without as many coworkers around virtually, I guess. Rob Wiblin: Yeah. You can work from Australia so long as you’re also a vampire. That’s amazing. I had no idea that GiveWell had transitioned, because I think years ago, almost everyone was local, but now it sounds like it’s almost remote first, or a majority of people are remote. Elie Hassenfeld: It’s basically remote first, yeah. It’s funny, I remember as late as 2019, right before COVID hit, having folks who joined remotely — and really pushing them hard to move, and talking about all the downsides of them being remote. I’m sure there are some costs, but I think that it was just a mistake. In some ways a silver lining of the last few years is that we’ve just leaned into being remote first, or remote mostly. I don’t know what to call it. It’s worked really well, and I’m glad that we’ve been able to hire people from all over because it’s massively increased the pool of candidates that are open to working for us. Rob Wiblin: Yeah, that’s fantastic. It’s one of the positive effects COVID has had. I’ve got to say, I think I was sceptical early on the pandemic that the remote work thing would stick. I figured we probably had good reasons to require everyone to be present. I mean, certainly there are big benefits to being local as well, but there’s also big benefits to allowing remote work, and it’s fantastic that it’s working for you, because it means such a larger fraction of people can potentially apply. Elie Hassenfeld: Yeah, exactly. Rob Wiblin: Pushing on from careers, if people in the audience want to support any of the programmes you’ve talked about today financially, can they generally do that? Or do these programmes typically not accept individual donors? Elie Hassenfeld: The recommendation that we make to donors is you can either give to our Top Charities Fund or our All Grants Fund. Top Charities is going to the very small set of organisations that meet high confidence in high impact. The All Grants Fund will support opportunities like the ones we discussed today, and you can see a list of all the things we’ve supported in the past. You certainly could support these organisations directly if you wanted to. We are happy with the level of support we have provided those specific organisations to date and we’ll reassess further support in the future. Supporting the All Grants Fund would be the way to support that ongoing work and new opportunities that are analogous to those ones that we discussed today. Rob Wiblin: Yeah. Do you want to just make a quick pitch to people to check out the GiveWell website and consider donating money to your recommendations? Elie Hassenfeld: Yeah, we are finding far more opportunities to help people across all of these programmes — health, livelihood improvement, things like malnutrition and water — than we ever have before. That means that more funding means more people will be helped. You can, on our website, see the things we’ve supported, the things we currently recommend, and all the underlying research behind it — so you can decide whether or not, hopefully, that you do agree with the conclusions that we’re reaching. Rob Wiblin: Fantastic. All right, we should wrap up and let you get back to normal work. A final question I have is: What do you do for fun? Who’s the man behind the cost-benefit analysis spreadsheet? Elie Hassenfeld: What do I do for fun? Well, one thing I like doing is learning new sports, or physical activities that I haven’t done in a long time. I learned how to ski a few months ago, but unfortunately, a couple of weeks ago, I fell and got hurt, and so I’m recovering from that. That whole activity of athletics is on pause for a bit. I have four young kids, and sometimes, often, that’s not unwinding. That requires a lot of attention, but they are a lot of fun. I think that not only are they great to talk to and hang out with, but they often hold up a great mirror to me. I see myself in them, and I’ve learned more from them than I think I have in a long time. It’s been not only enjoyable, but very fulfilling in a certain way. But also a very informative experience, because I’m like, “Oh my god.” My kids right now are doing something called a read-a-thon, where there’s a competition to see who can read the most minutes at their school, and they’re so anxious about winning. And when you see someone else being anxious about some ridiculous goal like that, I’m like, “Oh, I do that sometimes. Oh, man. Thank you for helping me see how wrong I was.” Rob Wiblin: Yeah. I’m hoping to have kids as well, before too long. Elie Hassenfeld: That’s great, Rob. Rob Wiblin: I spent a little while asking people were they happy when they had kids or not? People mostly don’t say that they regretted it. I guess it sounds like you haven’t regretted it. Elie Hassenfeld: I think it would be a hard thing to say. We have four kids, so we just kept going. We did that very intentionally, and are very excited about it, and yeah, I’m very happy. I actually remember after my first daughter was born, I was hanging out with a couple of coworkers. When I think back on those times, I remember them as being incredibly hard: the first three months of a new baby, and we weren’t sleeping, and she was crying. I also remember just being out with them and them saying, like, “Elie, you seem so much happier than you used to be.” That’s so strange. It’s weird because I can’t access that memory of my past subjective wellbeing. I’m glad that I have this objective data point from outside of me to remind me what it was actually like. Rob Wiblin: Yeah. Did you have a sense of what changed in your mentality? Is it just like adding more balance to your life, perhaps? Elie Hassenfeld: I think it probably has added more balance to my life. I find myself often, as some people might, stressed about work. And I don’t feel that same way about my family. I don’t know exactly why, but it’s this very enjoyable, very fun opportunity to do a lot of things that are creative, learn new things, learn about myself, learn about them. There are downsides to always having people around that you like, but also it’s nice to have people around that I like, and they’re fun. My wife and my kids are just always here, and I think that’s really nice. It has added and I hope it continues to add something really meaningful to my life. Rob Wiblin: Random question: Did having kids affect how much hearing about the horrible conditions that many people are growing up in — the people having clubfoot and not being treated, or suffering from malnutrition ongoing — did it make it feel at all more visceral, perhaps because you could imagine this happening to your own kid? Elie Hassenfeld: I think it did. Also it didn’t really change the way I relate to my work on a day-to-day basis. I definitely had the experience of going to the pharmacy and getting amoxicillin when one of my kids was sick, and then knowing that there are countries where you couldn’t get amoxicillin for your child. When I take a moment to pause, which I don’t do enough, and say, “Wow, that’s what I’m working on” — and I think in many ways, not exactly, but that’s what you’re working on; that’s what we’re trying to do — it does help me connect to the work more, to imagine it as being helpful to someone like my children, whom I feel obviously a great deal of love for. And to think about it has made it in some ways easier to feel strong emotion when I pause and try to make myself do that, which I think is good and helpful. Rob Wiblin: My guest today has been Elie Hassenfeld. Thanks so much for coming on The 80,000 Hours Podcast, Elie. Elie Hassenfeld: Thank you, Rob. This was awesome. Rob Wiblin: If you head to givewell.org/jobs you’ll find they’re hiring for all sorts of positions, including Research Associates, Donor Relations Associates, and Project Manager to name a few. If you liked that episode, you might want to go back and listen to: #129 – Dr James Tibenderana on the state of the art in malaria control and elimination #124 – Karen Levy on fads and misaligned incentives in global development, and scaling deworming to reach hundreds of millions All right, The 80,000 Hours Podcast is produced and edited by Keiran Harris. Audio mastering and technical editing by Simon Monsour and Ben Cordell. Full transcripts and an extensive collection of links to learn more are available on our site and put together by Katy Moore. Thanks for joining, talk to you again soon.