Latest / 80,000 Hours Podcast / #113 – Varsha Venugopal on using gossip to help vaccinate every child in India

Transcript

#113 – Varsha Venugopal on using gossip to help vaccinate every child in India · 80,000 Hours Podcast

• Rob Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where we have unusually in-depth conversations about the world’s most pressing problems, what you can do to solve them, and how a little bit of gossip can save a life. I’m Rob Wiblin, Head of Research at 80,000 Hours. Most of us have at some point read about a randomised trial showing that some product or service really helps people out — exercise programs for mental health, free vapes in order to help people quit smoking, electric cookstoves to reduce indoor air pollution and early death. Sometimes those studies suggest the intervention is incredibly cost-effective, offering huge benefits relative to the cost, and other times they might look too expensive to be high priority. But either way, we look at a summary of the study, think ‘huh that’s cool’ and close our browser tab, never to think about it again. I mean, what are you going to do, start an electric cookstove business? Well today’s guest Varsha Venugopal decided to quit her job, roll up her sleeves and do the equivalent of starting an electric cookstove business. Except it isn’t air pollution she’s been trying to fix, it’s young children going unvaccinated just because their parents didn’t make it a top priority to get them back to the doctor’s office on time. We talk about a lot of big, high-level ideas on this show, and of course I totally love that stuff. But none of those ideas matter if nobody figures out how to convert those ideas into something that someone can actually do — a project that a real person has to implement in this messy world of actual people, and then figure out how to scale it so they can go on and make a meaningful difference. We haven’t had so many episodes on that lately so I decided to go out looking for people on the sharp end of taking ideas about how to have a big impact and then executing on them. Varsha stood out for the process she used to figure out what to do, their success building a presence in India, and ability to explain this all. Varsha and her colleagues are still in the thick of growing, so fortunately they haven’t forgotten what things are really like. I think everyone can get something out of this episode, but particularly so if you might want to start your own project or work in global health and development. Alright, without further ado, here’s Varsha Venugopal. Today I’m speaking with Varsha Venugopal. Varsha started out her career studying urban and regional planning, then working as a local governance specialist at the World Bank for five years. She then did a Master’s in Development Management at LSE before working at the Natural Resource Governance Institute, the global health nonprofit called Options Consultancy Services and the International Governance and Risk Institute. But in 2019 though, she made a big career change, and co-founded the nonprofit Suvita, which aims to ensure that every child in India gets all of their basic vaccinations, which is what we’re going to talk about today. So thanks for coming on the podcast, Varsha. Varsha Venugopal: Thanks for having me. Rob Wiblin: I hope we’ll get to talk about the challenges of scaling up a new health nonprofit, and what you’ve learned from being a nonprofit founder. But first, as always, what are you working on at the moment and why do you think it’s important? Varsha Venugopal: So we are a two-year-old startup nonprofit working on improving uptake of routine immunization in India. Two things are top priorities for us right now. One is we are in the middle of an iterative cycle for one of our programs, the ambassador program, to figure out the best ways to scale it up. And second, we are planning an upgrade of our monitoring and evaluation system, which will allow us to make a more compelling case for the impact of our programs. I think it’s a particularly important moment now compared to any other moment, because we know for the last couple of years there has been millions of kids who’ve dropped off in India as well as abroad as a result of COVID-related lockdowns, and we have this huge sense of urgency to get these children back on track. Rob Wiblin: Yeah. Nice. Okay, well, we’ll come back to Suvita in just a second, but first off, maybe let’s quickly zoom out and consider the kind of broader context of the problem that you’re working on. So basically how many kids are going unvaccinated around the world? And I guess how many end up dying as a result? Varsha Venugopal: We have 19 million children who fail to receive their basic vaccinations by the time of their first birthdays each year. We know as a result, one child dies every minute because of vaccine-preventable disease. Rob Wiblin: So at their first birthday they haven’t received any or most of the vaccinations, or is it just that 19 million haven’t done the full schedule that’s recommended? Varsha Venugopal: So 19 million children are undervaccinated, and that’s a really good question because there is something called the zero-dose, which means they haven’t received any vaccinations. But with these 19 million children, what we are specifically referring to is that they are undervaccinated, which means they haven’t completed their full dose by the time they turn one. Rob Wiblin: Yeah. Yeah. And I guess, so you’re saying a child dies every minute, so that’s where you get kind of the 1.7 or 1.8 million people dying a year. Are there harms other than death? I’m guessing getting lots of these diseases might have long-term health consequences for people who are… you know, babies that even survive? Varsha Venugopal: Yeah, absolutely. So we know we are talking about vaccines saving two to three million child deaths every year. We know there are huge social and economic benefits at both individual and societal levels. So we know that there are costs related to medical treatments and also related to income and productivity losses, as well as, I mean, one can imagine the whole suffering of family and friends. There is also some evidence of association between vaccination and childhood development and educational outcomes. I think broadly there is a fair degree of consensus between experts that for every $1 invested in expanding immunization, there are about $60 returns in social, economic, and environmental benefits. Rob Wiblin: Yeah. I mean, I think the cost-benefit analysis on vaccines almost always shows these wild ratios between the benefit and the cost, and I guess the main questions that I’ve seen asked are like, “Well, how do we get more people to actually get these vaccines?” Because I guess all of the really low-hanging fruit has been taken, where it’s straightforward to get vaccines to kids. A lot of work has been done to do that, but then the ones that are left, it’s a little bit more challenging. There’s something making it hard. So you see… I guess, what are the main countries where kids aren’t getting vaccinated? Varsha Venugopal: So the big countries that are there are Afghanistan, Pakistan, DRC, Ethiopia, Indonesia, Nigeria, and of course India. India, which of course we’ll come to later, has about half of these undervaccinated children, so of the 19 million, we have about 10 million of them in India. Rob Wiblin: It’s a little bit surprising that India accounts for such a large fraction because I think of… you know, India’s not a rich country, but it’s not one of the poorest or one of the least functional countries in the world. It seems like it’s kind of punching above its weight in terms of undervaccination, perhaps, relative to what people might expect. Varsha Venugopal: Exactly! And that’s where it’s really useful to get into some of the details. And what we see is while it may be a supply-side problem in some of the other countries, in the case of India, it’s very much a demand-side problem. And by that what I mean is, it’s not that kids aren’t being provided the vaccine or the facilities aren’t there. It’s more about finding ways to motivate parents to go and complete the immunization schedule. Rob Wiblin: Yeah, I guess that is one reason why you might think that there’s more low-hanging fruit in India, or perhaps one reason why you’re focusing on India, is that I imagine there’s many children going undervaccinated or maybe very undervaccinated in war-torn areas or very remote areas where it’s very hard to get people in. And yet there’s an enormous number of children out there who you probably could access relatively easily, and it’s a question of getting the parents motivated to go and get the vaccinations. Varsha Venugopal: Yes. So in other countries, in several regions, it may be something called this last mile problem, which is more about supply chains and getting those vaccinations to these hard-to-reach areas. Most of that is fairly well covered in India as a result of government efforts and of other supporters such as Gavi, the Vaccine Alliance and WHO and UNICEF. So yes, absolutely, it seems very much a problem of the demand side that we are trying to address through some of our nudges. Rob Wiblin: Yeah. Yeah. So just to be concrete, what kind of diseases are we talking about? What are the main kind of vaccine-preventable diseases that end up killing people? Varsha Venugopal: So the big ones these vaccines protect against are measles, diphtheria, tetanus, whooping cough, polio, hepatitis, and others. Rob Wiblin: Yeah, who are the big players who are trying to fix this problem? I’m guessing kind of, you know, Indian government, lots of other major foundations and things like that? Varsha Venugopal: So one big player in this sector in the last couple of decades is Gavi, which is the Vaccine Alliance. And it’s an interesting model where it encourages manufacturers to lower the vaccine prices for the poorest countries in return for long-term, high-volume, and predictable demand from those countries. So it has a five-year funding cycle. So what that means is that it can negotiate with these manufacturers for a longer term, and instead of countries directly negotiating with these manufacturers, it allows it to buy in bulk for a longer period of time. And this partnership with other organizations such as Bill and Melinda Gates Foundation and WHO and UNICEF and others has allowed it to vaccinate half the world’s children at this point. Rob Wiblin: Hmm, okay. So have we been making a lot of progress against the undervaccination issue? Varsha Venugopal: Yes, absolutely. I think you had some figures there, right? Rob Wiblin: Yeah, I think… Yeah, I looked this one up on Our World in Data because I was curious to know how things were going, and it seems like we’ve been making massive progress. So even just in… Okay, so in the seven years from 2010 to 2017, I think we went from 2.8 million people dying from vaccine-preventable diseases down to 1.7, just like a one-third drop in seven years, so it seems like we’re kind of killing it. Varsha Venugopal: Yeah. But if you see in there, I think more recently, it has also been stagnating. And I think that’s where these problems of last mile and undervaccination and demand-side challenges come in. Rob Wiblin: So that’s probably COVID, right? Of people being stuck at home and… or is it more than that? Varsha Venugopal: The last couple of years we know the trend has to do with Covid related lockdowns with millions of additional kids dropping off. But the broader trend of stagnation seems more to do with undervaccination. For instance you know we have 3 doses of measles vaccines recommended and even though 85% of kids worldwide got their first vaccine, it dropped to 64% for the second dose. Rob Wiblin: Yeah. So you’re saying a much larger fraction of kids are getting the first dose, but then it drops off. I guess, well, it suggests that the parents don’t object to vaccines. You might think of, oh, it’s anti-vaxxers who aren’t getting their kids because I suppose that’s what we’re more familiar with in the UK or in the US, but in India it’s more just that life’s very busy or it’s more difficult, you got to go to a specific place at a specific time to get the kids the vaccines. Yeah, what’s the practical situation? Varsha Venugopal: Yeah, no, that’s a great point. Especially these days, there is so much more conversation on vaccine hesitancy and it’s really useful to, I guess, emphasize that that’s not the kind of challenge we are facing here. There are several reasons for this and we can go into that, but broadly people seem to trust vaccines, which is why they get the birth dose and they also get at least one other dose and then drop off for a variety of reasons, which have nothing to do with the supply-side challenges. Rob Wiblin: Yeah, so is it possible to go out and just survey Indian parents and find out at least among those who didn’t bring their kids in for the later vaccinations, what’s the deal? What was the reason that they didn’t do it? Varsha Venugopal: Yes, there have been at least two surveys of this kind that have been done in India. So one was a recent survey that was conducted in 2018, of nearly 40,000 caregivers of undervaccinated children in the lowest-vaccine-coverage districts where they found that the most common reason for missed appointments was an awareness gap. Then there was another nationwide survey which is slightly older, it was conducted by UNICEF in 2009, where respondents were four times more likely to describe demand side issues than supply side ones when giving reasons for a child being partially or not immunised. So it seems largely at least in India, the challenge is on the demand side. And there is further breakdown and hypothesis around why these parents choose not to complete the vaccination. The common barriers are around taking time off work, low uptake of routine immunization in rural areas where education rates are low or information can be confusing, or just difficult to access. Rob Wiblin: Hmm, so it seems like there’s other countries at a similar income level that are managing to reach a larger fraction of kids with these later-stage vaccinations. So is something perhaps going wrong with the state-level health departments’ approaches to vaccinations in India? Is there something peculiar about the India situation that’s making this a bigger issue? Varsha Venugopal: I don’t think so. In fact, to India’s credit, it is one of the top manufacturers of vaccines. It has an extensive supply of vaccination and it has an excellent health system that’s rolled out to deliver these vaccines. We also know they have had successes in the past with vaccines such as polio, where there was a huge government and community and other supporter effort, where they managed to eradicate the polio from the whole country. So I think we are fairly confident, the challenge is very much on the demand side and there isn’t anything else that the government could be doing on the supply side to make this better. Rob Wiblin: Yeah, interesting. Okay, let’s push on and discuss the organization that you’ve made to try to fix this issue in India. I guess, in a nutshell, what does Suvita do to try to get more jabs in arms? Varsha Venugopal: So Suvita improves uptake of immunization. That’s getting more parents to complete the immunization schedule through two models. So one is sending SMS reminders directly to parents and carers. The other is something called gossip. So what’s gossip? It is a very deliberate, systematic way of identifying the local community influencers and sending them regular messages and reminders, and it turns out it increased effect sizes in terms of parents taking their kids for immunization versus sending reminders directly to parents. Rob Wiblin: Yeah, I was super surprised to hear that that was the second approach that you were taking. So, we’ll come back to that because I think it’s really interesting. But maybe first, tell us a little bit about the history of how Suvita got started. Varsha Venugopal: Sure. So I was introduced to effective altruism in around 2018 through an LSE mate, Kellie, who had founded Effective Giving in the Netherlands and had called me over for this EA Global Conference in London in 2018. So it was my first time hearing about effective altruism. I went and read a bunch of papers and looked up articles and found it very interesting. And this… broadly, this idea of, we all have limited time, energy, resources, money, and we need to have the most impact, and how do we go about doing it and focusing on this impact, neglectedness, tractability framework was very compelling. So I went to the conference, met a bunch of people, did some more research and as a result, went through the strategy Incubation Program in 2019. And following that, I founded this nonprofit along with my co-founder Fiona. So I can run you through the steps that Charity Entrepreneurship took, if that’s useful at this point, to get to the point of identifying some of the top intervention areas, which we then took on to further research and zone in on the intervention we focused on. Rob Wiblin: Yeah. Okay, so you’ve kind of heard about these effective altruist ideas and you’re thinking, “Maybe I want to start a project to tackle a neglected problem in an effective way.” How did you find Fiona, and also how did you end up kind of settling on vaccinations and this approach? Varsha Venugopal: Yeah, so I was already working in development for more than a decade when I came across effective altruism. So I guess for me, the pivot was more about maybe slightly getting disillusioned with the areas of work I was engaging in, which was fiscal decentralization and oil, gas, mining, and development, which had complex theories of change and it was harder to focus on attribution and impact. And having come across effective altruism, and some of its precepts, this idea of simpler theories of change and focus on impact was very compelling. Varsha Venugopal: So I went through Charity Entrepreneurship incubation, which brings together others similarly interested and has a rigorous process of identifying potential entrepreneurs. What Charity Entrepreneurship does is, it starts by looking at some of the recommendations of GiveWell and the World Health Organization and the Copenhagen Consensus. And then does some shallow research based on several criteria, including cost effectiveness, scalability, et cetera, and additionally looks into the ITN framework — impact, tractability, and neglectedness — and where they had landed was unconditional cash transfers. So Fiona happened to also be in the same incubation program. They also go through a process of trying to match different entrepreneurs to each other, and it turned out we were a good match. So both of us started by looking at conditional cash transfers for immunization. Varsha Venugopal: Conditional cash transfers was something they had already looked at and we selected immunization based on, of course, the quality of evidence — there was external and internal validity of research and validation from experts through interviews. And in terms of India, I guess we were veering towards India for various reasons. There was a case to be made around scalability. There was also my own personal and professional networks there. So we started by looking at interventions that could potentially beat conditional cash transfers in terms of cost effectiveness. Varsha Venugopal: And that’s when we came across this large-scale randomized trial run by the Poverty Action Lab, which was doing exactly that. So quickly about this trial, it was run in the state of Haryana, and they were looking at across 7 districts. It was led by Abhijit Banerjee, Esther Duflo, and others. And what they were specifically looking into was different combinations of policy tools that could have the most impact on improving uptake of routine immunization. So the three policy tools they were looking at were: mobile credit directly to parents and carers and again, different amounts and increasing at different levels; SMS reminders directly to parents and carers; and this gossip, this idea of finding influencers. And what they found through this study was that the combination of SMS reminders and ambassadors was the most cost effective. Adding local ambassadors and text messages to the government’s routine immunization program increased the number of fully immunized children per dollar spent by 9.1%. So we found that extremely exciting. Rob Wiblin: Is it 9.1 percentage points? Varsha Venugopal: This is percent. Rob Wiblin: Percent okay, yeah. Varsha Venugopal: So in terms of the actual increase in percentage points, we know for the ambassador program, it is 10 percentage point increase. Rob Wiblin: Okay, yeah. Varsha Venugopal: And then we did additional research on the SMS reminders and looked into some meta-analysis to reach our own conclusions on that separately. Rob Wiblin: Yeah. So, some of these studies that look at the effect size, they don’t think that much about cost. Did that paper kind of do a per-dollar analysis? Varsha Venugopal: Yeah, exactly, which was quite exciting and interesting for us. This wasn’t there in the draft papers we were looking at initially, but then the actual paper just came out a few months ago and in the subsequent versions, they have been looking at cost effectiveness. And especially because they were looking to advise the Haryana state government on how to go about this, they were very keen to look at not just impact but cost effectiveness. So if you just looked at impact, the combination of conditional cash transfers plus SMS plus ambassadors is the most impactful. It has the highest effect sizes. But when you look at it from the cost-effectiveness perspective, it is the combination of only SMS and ambassadors. And one can see the rationale for that because setting up and running conditional cash transfers can be a fairly expensive operation. Rob Wiblin: Yeah, interesting. So the idea… yeah, even though you could have more impact in one particular village doing all three, it makes more sense with the given budget to do the two that are best and then try to reach more areas because I guess that’s easier to scale and — Varsha Venugopal: Yes. Yes. So for us, the plan is very much to start with these SMS reminders plus ambassadors, but maybe at a later date there may be certain pockets which lend themselves better to conditional cash transfers. It’s not something we’re completely ruling out but it’s not something in our portfolio of programs at the moment. Rob Wiblin: Yeah, so I didn’t realize that Charity Entrepreneurship was quite so central to getting this started as it was. Yeah, for those who don’t know, Charity Entrepreneurship is this project, I guess an incubator, run by people who are very sympathetic to effective altruist ideas. And they do a bunch of research where they try to kind of shortlist interventions that they would love to see people going and actually trying to implement in the real world. And then they try to match those with people like you, who are thinking, “I want to start something, but I’m not sure exactly what, and I’m not sure who to do it with.” And I guess, yeah, pair up founders and then pair them up with an idea that’s a good fit for them and then help them actually get moving. Rob Wiblin: Yeah, maybe we’ll come back and talk about that later, because it could be an excellent resource or excellent thing to go through for some people in the audience. I saw on your website that you have this kind of table summarizing the evidence on SMS reminders, and I took a quick look over and it seemed like there were five trials that had kind of positive effects and then five where it didn’t really seem like the SMS reminders had moved the needle very much. Did you go away and kind of, either yourself or get someone else to do this kind of literature review to learn other lessons about under what circumstances these things work, and what are things you can vary about the intervention to make them have a bigger impact? Varsha Venugopal: Mmm, yeah, no, that’s a great question. So our colleagues at Charity Science Health [merged with Suvita in 2020] did undertake this meta-evaluation in 2019 with the nine existing randomized trials, and they found a 7.4 percentage point increase in full immunization due to SMS reminders. So then to be conservative, they also ran the meta-analysis after subjectively making various substantial discounts in the reported effect sizes, and this reduced estimated effect size is around 3.2 percentage points. So our sense is the range of effect size is somewhere between 3.2 and 7.4, but in all these cases, the program is still cost effective. Varsha Venugopal: I think one point we are aware of is we were quite clear on focusing on SMS versus, say, WhatsApp because based on our research, we could see that the smartphone penetration was still not as high as regular phones. So we were quite keen focusing on text messages, but we are also quite aware of the high correlation between mothers who haven’t got high literacy and the kids dropping off vaccinations. So the next iteration we are quite keen to introduce is voice recordings to see how they could further reinforce the messaging that’s going through the text messages. Rob Wiblin: Yeah. Yeah, I was really happy to see the table both had studies with positive results and studies with null results where it couldn’t find an effect. Because an intervention, even one that’s effective, you should find some studies that by chance have found that it didn’t work in this particular case or perhaps were underpowered, there weren’t enough people in the sample to detect even a real effect. And if people aren’t aware of that, when they’re setting out, it suggests that they haven’t really been quite comprehensive enough in looking at all of the evidence out there to get a proper average kind of effect size across the full range of results, rather than just cherry-picking a few of them. Varsha Venugopal: Yeah, no, we absolutely believe in making sure we look at the broad gamut of evidence that’s available. This is also the additional reason why we are quite keen on introducing the ambassador program along with the SMS reminders, because this seems like the effect sizes with the ambassadors is higher and we want to make sure we can capture some of that impact on the table. Rob Wiblin: Yeah. So were there any studies that didn’t find an effect, where it was possible to kind of identify that some particular thing had been important. You were suggesting, for example, literacy might mediate the effect of the messages. If people can’t read them, then that’s going to reduce the effect size pretty substantially. Yeah, are there other factors that you had to think about, about whether that’d work in your specific context? Varsha Venugopal: Yeah, there is quite a fair degree of variation. I mean, they’re obviously in different regions to begin with, these text reminders. Also in terms of what the exact messaging itself is. Is there a feedback loop on whether the child actually got the vaccine, and then is the next reminder that’s going out in any way connected to that? Et cetera. So there was… for each of these studies we had to apply some of these discounts based on how similar they were to what we were planning to undertake and what that could mean in terms of the effect sizes of our intervention. Rob Wiblin: Yeah. This is a bit of a demanding question to know off the top of your head, but maybe to give listeners a sense of kind of the funnel here, if you were to spend an extra $1,000 scaling up the program, how many SMS messages might you be able to send, and maybe how many parents would you be able to hassle through this gossip mechanism? Varsha Venugopal: Right. So right now at the margin we estimate we can cover an additional village — that’s both SMS reminders and ambassadors for eligible parents — at around $150 to $200. And given each village has about 100 children due for vaccines, what that would mean is for $1,000, we would cover roughly six villages or 600 children. So now taking into account the J-PAL study — which in the original paper suggested a 14 percentage point increase in uptake as a result of the ambassador program — we discount it to 10 percentage point increase. We think by spending the additional $1,000 and reaching 600 children, we would be reaching an additional 60 children that will be immunized. Rob Wiblin: Okay, so something like $20 per additional child immunized or reaching the full immunization schedule? Varsha Venugopal: That sounds right. Rob Wiblin: And is it possible to give a rough approximation of how many lives that will save, or how many children you have to get extra vaccinations in order to save a life? Varsha Venugopal: That’s the bit that gets complicated. Rob Wiblin: Yeah. Varsha Venugopal: So the last time we looked into this, what we did was plug our numbers into the GiveWell model for New Incentives, which is a GiveWell top charity working in North West Nigeria using conditional cash transfers to improve uptake of routine immunizations. So we borrowed some of their figures and adapted it by plugging in the relevant data for our own context. By doing that, we estimate we can save one life for roughly every 100 extra children vaccinated. So vaccinating 60 extra children, we save 0.6 statistical lives. Rob Wiblin: Okay, so something like — Varsha Venugopal: Having said that though, I realized we plugged into a previous model, which has recently changed in the GiveWell page. So at that time, I think it’s a 2017 model, we could save a child’s life for maybe something in the region of $1,700. But I see the GiveWell top charities now are somewhere between $3,000 to $5,000 for every additional child’s life saved, so we need to go back and question that. Rob Wiblin: Yeah, their numbers are always changing. We’ll stick up a link to some of the GiveWell resources. I guess the back-of-the-envelope way that I would estimate this is… You’ve got these numbers, I guess from, I don’t know, the World Bank or the World Health Organization, of how many kids in India are not being fully vaccinated, and then also how many children in… estimates of how many children in India are dying of vaccination preventable diseases. And maybe you could say, you just look at the ratio there and you say, well, if there’s 10 million children that are going through each cohort not fully vaccinated, and then 100,000 of them are dying, then you’d say, well, it’s 1 in 100, so it’s 100 to 1 ratio. So if you could vaccinate 100 extra children, you might save one life. Varsha Venugopal: Mmm. Hm. Rob Wiblin: Is that like a non-idiotic way of doing it? Varsha Venugopal: That sounds reasonable. Though, now having the GiveWell model, and we see that as the gold standard, we would rather follow that. On how the money gets spent on the program, given the early stage we are at, the funds that we raise now will play an important role in bringing down this cost per life saved for future work. Varsha Venugopal: And we see three main ways of doing that. So one is the straightforward economies of scale: you add an additional district, it brings down the overall cost. If we are not adding more office size, for instance. The other is process improvements. So for instance, we are currently hiring for a tech role, which we think, for instance, can bring through some exponential increases in our efficiencies. There are also so many ways we are iterating our nomination survey and recruitment survey, which would allow for increased efficiencies. And the final is this process innovation, which is, I guess still an area we need to explore further. There may be some huge exponential gains we could make by reducing cost by say 40%. For instance, if we could find the right influencer without having to go through the whole nomination process. So this third piece is higher impact but lower likelihood. Varsha Venugopal: So I think there is something unusual about the impact of our funding as a startup nonprofit compared to some of the more established GiveWell programs, because our model is still quite malleable and developing. And some of the lessons we can learn from this funding can help us get more future funding, which could be used more effectively. Rob Wiblin: Yeah, that makes sense. You’re still kind of improving and developing and reducing your per-child reach costs. So yeah, you hope to kind of increase that cost effectiveness over time. And I guess, the people who get in early help to fund that development of the product basically, and yeah, could maybe have a bigger contribution. Varsha Venugopal: Yep. Rob Wiblin: So SMS reminders is pretty intuitive. I guess you get numbers of parents, maybe from the hospital where a child was born, and then you message them about the vaccination schedule. What’s this gossip intervention then? Varsha Venugopal: So the gossip intervention is, in the original J-PAL study, the surveyors were sent to a random set of 17 households. And they asked several questions, but the main question is, “If there was a fair in town, who is most likely to tell you about it?” We then run an algorithm to identify the top gossips, or community influencers, and the surveyors then go back to these top influencers and recruit them as immunization ambassadors. So that is the ambassador program in the J-PAL study. Varsha Venugopal: What we did as a result of COVID was pivot to a remote model, which already brings down the cost significantly, in that we call up a random set of households to identify these influencers and then call these influencers to recruit them as ambassadors. So compared to the SMS model, this model at the moment is still not as mature, and we’re still iterating to find the most cost-effective way of doing this. Rob Wiblin: Yeah. Okay. So you identify a village. I guess, it seems like a significant fraction of India’s population is living in villages of 500 to 1,000 people or so. Is that kind of where you’re targeting? Varsha Venugopal: That’s the standard ratio we are using. Yes. Rob Wiblin: Yeah, okay. And so you call up a random bunch of households and you say, “Who would tell you about an event?” And then you pick maybe the person who is most often mentioned and get their phone number and then call them up. And then you say to them… what do you say on the phone? It seems like an odd call to get. Varsha Venugopal: We have some standard language we use. And in fact, we are iterating with some of this, which also seems to increase the success rates. But broadly, it is saying that, “You have been identified as a community influencer, and we would like you to be an immunization ambassador. There is no monetary cost involved in this. It is a voluntary exercise. We would be sending you regular reminders on immunization camps. Are you willing to be an ambassador?” Rob Wiblin: Yeah. And what do people say to that? Varsha Venugopal: So we conducted a pilot study earlier this year for two purposes: one was to see if a remote version works, and the other was to see if ambassadors act. And the answer to both questions was broadly yes. Varsha Venugopal: So in more than 90% of the cases, when people were contacted randomly, they were able to give us not only the name of an influencer, but also their phone number. And then in more than 95% of the cases, the influencers agreed to be ambassadors. The challenge we had was reaching these influencers. Possibly because they are influencers, they’re not around most of the time. It’s unclear. That’s just a supposition. But that’s the bit we are now iterating, to try and call at different times of the day or different days of the week to get to them. Varsha Venugopal: The other bit was, are these ambassadors then sharing the information? So one thing we did early on was give them a phone number that they could then share with parents and ask the parents to give a missed call to enroll into the SMS reminders program. And I think we had more than 30 people sign up in the first week, which again, gives us some confidence that the ambassadors are sharing this information and people are receiving it and somehow responding to it. Rob Wiblin: Yeah. So just trying to build a better model of exactly what this looks like in my head. So do you know, are these like cool people, or very extroverted people, or high-status people in the village, like maybe someone who’s involved in local politics? Do you know what kind of person we’re talking about? Varsha Venugopal: That’s a great question. So in the J-PAL study, they looked at both trusted people as well as community influencers and see the interaction effects, and it turns out it is the community influencer bit which seems to be causing the biggest impact. Other than the fact that we know about 18% of our influencers are women, which is kind of similar to the J-PAL study, we are very curious to better understand the motivations of the ambassadors and why they take it on. And that’s the kind of thing you don’t get in an RCT, so that’s something we want to go there and interrogate further. Rob Wiblin: Yeah. That makes sense. And do you know what conversations they’re having? Are they… I was wondering, are they going around and kind of shaming parents for being lousy parents because they are, I don’t know, not vaccinating their kids? I know there’s this whole thing of people policing other parents’ behavior and things like that, or is it more like they’re reminding people of this fun vaccination event that they can go to? What’s the conversation I should picture in my head? Varsha Venugopal: That’s a good question, and honestly, we do not know. In the study itself, there were some most enthusiastic people in the J-PAL research who put up posters. And they deliberately removed them from their study to ensure they could focus on the — Rob Wiblin: Gossip element. Varsha Venugopal: — intrinsic gossip element. Exactly. So it seems something about this intrinsic motivation, ambassadors by just being influencers are able to take on that role. How they actually share that information is not something we have figured out yet, except that they seem to be able to do it. Rob Wiblin: Yeah. Okay. So just to have a picture in the head of the magnitude of the benefit, you’re saying it increases vaccination rates by 10 percentage points and that’s about 27%, so huge magnitude. How large is the SMS reminder effect again, separately? Varsha Venugopal: Somewhere between three and seven. Rob Wiblin: Three and seven percent. Varsha Venugopal: Yeah. Rob Wiblin: Okay. So I’m guessing the SMS reminders are much cheaper because you’re just sending texts, and I guess they’re probably easier to scale as well, because you just have to get a bunch of phone numbers and then stick them in some piece of software that sends out text messages. On the other hand, it seems like the ambassadors have a larger effect size, maybe a more robust result in the RCT. How do you kind of trade off this easiness of scalability and low cost? Varsha Venugopal: Yeah, so we know SMS reminders, you’re absolutely right, are easier to scale up. In fact, in one of the two states we work in, Maharashtra, we already have a memorandum of understanding with the state government which allows us to rapidly scale up across the state. Having said that, we are quite aware that it’s the ambassador program which seems to be having some of these large effect sizes, and we’re really excited to overlay it on top of the SMS reminders program. Varsha Venugopal: With that in mind, we are now introducing an ambassador program in Maharashtra, we hope. And in the other state, Bihar, we have already completed this pilot and we just this week have also started exploring an additional district, Patna. I think for us, the fact that we’ve already moved to a remote version makes it much cheaper than the original version, so that’s one. Rob Wiblin: In the original version, people would go to the village and… Oh I see, yeah, yeah. Varsha Venugopal: Exactly, so the fact that we are now calling up these people makes the overall cost effectiveness higher, we would think, depending on what the actual impact is. So the big focus for us is to make the ambassadors as scalable as possible. Luckily, we know it already starts off scalable. Maybe not as much as SMS reminders. It’s fairly light touch. It’s just about identifying these right influencers. There is no component of training involved. It’s more about tapping into these existing social networks and influencers who seem to have a natural tendency to spread this information. So yes, while SMS reminders are more mature and faster to scale up, we are waiting to get to that same point with ambassadors so we could scale them up together. Rob Wiblin: Maybe I’ve got the wrong picture in my head here. Well I guess… so the SMSs are pretty easy to scale, but I guess you have to get the numbers, you’ve got to get the permission to do it. And you’ve got to put them through the software, clean up the data, and so on. Rob Wiblin: But it sounds like, in fact, getting some ambassadors from these villages is not that hard at all, because you’re making a bunch of phone calls, getting some names. And then you call them up and you’re not putting them through some difficult training process. You’re kind of just asking them, could they do this, and it might be a relatively fast conversation. And then potentially you’ve covered a whole village, right? Varsha Venugopal: Exactly. So it is not that far from a scalable model, but still compared to SMS… So let me just, I guess, quickly go over the process involved with SMS, because even with that, it’s a fairly different model in the two states we work in, and has obviously different impacts on cost effectiveness. Rob Wiblin: Yeah, go for it. Varsha Venugopal: So the first step is data collection. So in the case of Bihar, we have a couple of team members, Ram and Ravi, who actually visit the primary health centers and, under this government partnership, collect the relevant data from birth registers. These have already been recorded on paper by the clinician with consent of family and the data is encrypted at this point of collection. So then once they have this data, they digitize it from paper birth records for our database. And then the final step is this data processing, which is this downstream process where the data is sent out as personalized reminders and we use a third-party scheduling software, which connects through an API to our bulk SMS provider. Varsha Venugopal: So what this means is, in the case of Bihar, while it sounds like it’s an easily scalable model, there is an element of surveyors actually having to visit these primary health centers and collecting the data. Compared to that, in Maharashtra state, because we have this MOU with the state government, we pretty much miss the first two steps and directly go to the third step. We just directly get this reproductive child health data from the state government and can roll out the messages across the district. So obviously that has implications on the cost effectiveness of both models. Varsha Venugopal: Having said that, something I remember from your podcast with Max Roser, just the importance of getting good quality data. I think… I’d like to think we are doing some additional service with just getting better data from these primary health centers in Bihar, and we know the state immunization officer is pleased by the fact that there is higher-quality data now available which wasn’t available before. So there is a difference in the quality of this reproductive and child health data in both these states, and I guess there is an additional benefit we are providing by just allowing for better data collection for the government as well. Rob Wiblin: I was kind of asking this earlier, but I want to ask again, because I think it’s just kind of important. It seems like parents don’t want their kids to die, and the benefits of vaccinations are really big, and they don’t, the parents don’t object to vaccinations. What’s going on that’s causing the parents not to really prioritize, as a really high priority, making sure that their kids get the later vaccinations? Maybe they don’t appreciate that one in every 20 or something of the children who don’t get vaccinated end up getting these horrible illnesses and suffering severe consequences? Varsha Venugopal: So, to put this in context for India, we have about 26 million children born every year and we have 16 million children that are completing the vaccination schedule, so we are talking about the remaining 10 million. Varsha Venugopal: Now of these 10 million, we have two million who do not touch the hospital system at all, the ones that are called the zero-dose kids. We have one million that get the first dose — that’s BCG and tuberculosis, Hep B and OPV, that’s polio — but then disappear. So just the birth dose. And then we have these seven million kids who are getting the birth dose and at least one other vaccination and are dropping off. And that’s our primary end user we have in mind, though we do think we can probably make some dent with the other three million kids as well. Varsha Venugopal: So one of the big reasons we think that they drop off, and there has been some research, is to do with time inconsistency argument. I think it’s something referred to by Banerjee and Duflo in their book Poor Economics as well. This whole idea that I value my present very differently from the way I value my future, right? So even when I’m making decisions on exercising or gym, it’s all something I’d rather postpone for all these other myriad intrusions on my time in the present. And I think some of that is what’s at play for these parents as well. Varsha Venugopal: So caregivers may miss their appointments for various reasons. They could just be forgetting about them. They may not have the right information to accurately understand the benefits. We do know sometimes they don’t know how many appointments they need to come for, or they just don’t want to take a day off work. They don’t want to deal with a crying child. They don’t want to take the bus to go somewhere for vaccinations. So all reasons which possibly by small nudges in the margins could be addressed. Rob Wiblin: So their best guess is, it’s just the basic thing that we can all relate to of, you put something off and you put something off because it’s kind of a pain in the ass to do, and no particular day feels like the day that you want to go and do the vaccination when it requires crossing town, or dealing with your baby not wanting to get an injection. Varsha Venugopal: Yes, and this is exactly where it’s really important to reinforce that it’s very different from COVID vaccine hesitancy or other kinds of vaccine hesitancy that may exist somewhere, which is possibly far more complicated and involves issues of trust. I mean, here we know, because they’ve come in for the birth dose and at least one other dose, that they broadly trust the vaccines and believe it’s a public good, but for a myriad of reasons are then dropping off because of just being overwhelmed with their daily lives. Rob Wiblin: So this is… It’s maybe a little bit more like, how do you get people to stop getting late fees on paying their bills or something like that. Some other unpleasant ‘ugh’ task that people put off and then they end up suffering because of it. Varsha Venugopal: Yeah, and that’s the kind of areas where some of these nudge mechanisms seem to have the most impact, right? And I think Duflo and Banerjee of course have talked more about it, and Cass Sunstein in your podcast has done the whole research on it, and the different ways one can nudge people in the margin for something like this, where they broadly trust the intervention, but have not gotten around to completing it. Rob Wiblin: Actually, maybe it’s more like the retirement savings thing, where people do not set up their retirement savings properly, or they’ll continue paying this 1% fee on their savings for decades because they could never be bothered going and doing the paperwork to change it. And then it ends up costing them often $100,000 or hundreds of thousands of dollars over the course of their life. Well, that’s like a guess where the consequences are very big, but… Varsha Venugopal: Yeah, and the UK government, I think, has been quite good at embracing some of this, and they have a whole team set up. BIT, I think. Rob Wiblin: Yeah, Behavioural Insights Team. Varsha Venugopal: Exactly. That’s also been quite influential in how we should be responding to COVID. And I remember early on, there was this research on whether organ donation should be a default on your license, and I think putting that on as a default dramatically increased the number of people who then chose organ donation just because it wasn’t something you had to check out versus checking in. Rob Wiblin: Okay. Makes a lot of sense. When you were looking at the studies on the SMS reminders and the ambassadors from Banerjee and the others at J-PAL, were there any kind of weaknesses to the research or perhaps missing components that worried you at all? Varsha Venugopal: I think the questions you ask when you’re trying to scale up an intervention are quite different from those in a research study. So, while we are quite excited with the intervention and its impact, we do have some questions which are all around how can we scale this rapidly, right? So we already pivoted to a remote version, but we still have some open questions. Varsha Venugopal: I think one big question that an RCT can’t answer is about the mechanism: what is fundamentally changing at the community level as a result of these interventions? So SMS reminders we think are increasing salience; ambassadors we hypothesize are somehow changing trust and norms in the community. But this is something that’s still an open question that we would need some additional clarity on. I think another element of this is, any sense on where most of the effects come from would be really useful to know. Could we have a few super ambassadors for instance, so we don’t have to go through the whole nomination process. So again, that’s a question we hope to get answers to through our own M&E processes. Varsha Venugopal: I think other features of the study design… A big one is does the impact change over time? Now if these ambassadors, as we suspect, are actually changing norms and trust, then we don’t need to be hiring or finding ambassadors every year. Maybe it’s a one-off thing. But obviously again, our cost effectiveness calculations and our ideas on impact changes if we have to go back in and find ambassadors on a routine, say, annual basis. Varsha Venugopal: The other bit is the SMS. In the J-PAL study, they only enrolled children who had already shown up for at least one appointment. And they only sent out SMSs to either 33% or 66% of those people. In our case, we enroll all children who have been born in a public facility, and we send out messages to all of them. So intuitively we think we have higher impact on that one, but again, open to studying further. We are talking to J-PAL colleagues on all of this and hope to get closer to the answers soon. Rob Wiblin: You mentioned super ambassadors. What are super ambassadors? Varsha Venugopal: So just the idea that if a small number of ambassadors are responsible for the highest effect, are there ways we can directly reach them versus going through this whole nomination process? Again, if we get closer to understanding the motivations of these ambassadors, maybe there are ways… Maybe like you were asking, are they always leaders, maybe they’re of a certain education, of a certain caste? Anything else it could correlate to would help us then go, “Right. Let’s just go for — ” Rob Wiblin: Narrow in on that. Varsha Venugopal: Narrow in on that. Exactly. Rob Wiblin: Yeah, yeah. That makes sense. That’s where some of this qualitative information about like, how are they having the effect, what kinds of people are they, and what are they actually saying to other people is really useful for then having theories about what things to test next. So yeah, let’s come back to Suvita. How many people have you managed to reach so far? You’re only two years in, so it’s kind of early stage, still potentially finding product market fit to some degree. Varsha Venugopal: So through our SMS reminders program, we currently reach 350,000 parents across several states, but the main new states we are enrolling from are Bihar and Maharashtra. Varsha Venugopal: I mentioned this MOU we recently signed with the Maharashtra state government which allows us to access and scale SMS reminders across the state. We’re also in discussions with the state immunization officer in Bihar to scale SMS reminders across two districts. Varsha Venugopal: So, while we reached 350,000 parents so far, we plan to reach another 150,000 over the next year. With the ambassador program, the remote pilot that we conducted, we reached 5,000 eligible children and we are hoping to double it by the end of this year. Rob Wiblin: Okay. So you’re reaching quite a bit more people with the SMS stuff, I suppose, because it is slightly more work to… You have to train people to find the ambassadors and so on. Yeah. What’s the vision for a fully mature Suvita? How many people might you reach in the fullness of time? Varsha Venugopal: We want to solve this problem of underimmunization and drop-off rate in India. So we have six priority states which have more than 70% of the underimmunized kids, and we have a few plans on how to get there. I think the default idea is that we would be running certain call centers and be able to reach at least across those six states and have a footprint. Varsha Venugopal: But having said that, I think there may be some other endgame options we want to actively explore over the next few months. One is we advise the governments themselves and they take on some of this work, or we partner with other institutions working closely with the government, like WHO or UNICEF, who are already based in state government offices. Varsha Venugopal: Another idea is potentially identifying other nonprofits within local contexts who understand the local area and can possibly quickly hire people or send out reminders, and we train them. I think our focus is very much on developing an operations manual over the next few months to then explore some quick exponential wins we can have in terms of scaling up through other ways. In addition to, of course, replicating and scaling up ourselves. Rob Wiblin: Yeah. So what’s the biggest impediment or the biggest challenge that you foresee getting up to that much larger scale that you want to achieve? Varsha Venugopal: So I think the first is the fundamental challenge with designing and implementing a program which has proven evidence at a much smaller scale. So any general challenge of scaling up and ensuring the effect sizes remain the same. How do we make sure there isn’t a reduction in the fidelity of the design? Can we have the same degree of oversight of these surveyors, is the big one. Varsha Venugopal: I think the other big one for us is walking the line of innovating versus scaling. Something I mentioned earlier, there are certain funders who are quite keen on us growing in certain ways, which may or may not perfectly align with our strategic priorities. And especially at this stage for us, should we be reaching one additional district or a few more additional districts versus making some of those investments which may not see that direct impact on the ground in terms of additional lives saved, but may then build up to that exponential increase at a future date. So making some of those decisions is, I guess, high leverage but also high stress. Rob Wiblin: Yeah. So what’s the situation with the funders? There’s some that just want to scale up the existing thing exactly the same, and maybe there’s others who would rather try something new and try to improve the model. Is that it? Varsha Venugopal: Yeah. I think there is… This is an observation and I’m not sure where I align myself, but I do see how within the EA funders, there is much more focus on us delivering by ourselves, right? So the kind of organizations we see, AMF or New Incentives, they are rolling out the programs by themselves. Varsha Venugopal: Outside EA, I think one observation I have is a lot of these venture philanthropy kind of funders are very keen for nonprofits to have an endgame from day one. And their idea on endgame or sustainability is that, highly likely, the government needs to take it on. So what is that pathway for the government to take it on needs to be kind of embedded in your pilot stage itself. Varsha Venugopal: For us, we are somewhere in the middle. We have done the pilot. We are working closely with our government partners and supporting them. We are also replicating ourselves. And our next stage is to try and figure out, are we continuing to grow and become that possibly call center model? Or are there ways we could be training state governments or approaching other institutions to take on some of that role? Varsha Venugopal: So I think there is some fundamental difference in how this is viewed by funders. I think, even within EA, there are different ways I see that organizations have progressed. You have the AMF model, which is 10 staff, and I see their key role or key capabilities around raising funds and identifying local partners on the ground who are highly credible and then setting up extensive rigorous M&E systems to monitor how they’re performing. But by themselves, they’re a fairly small team. And then on the other end of the spectrum, you have an organization like New Incentives, which is a 100+ team across North West Nigeria, operationally rolling out conditional cash transfers by themselves. Varsha Venugopal: So a big question for us is, where do we land in that spectrum and how should we be progressing? And some of that takes a lot of experimentation and iteration over time. And what that means is… It’s a term I’ve heard some of our colleagues mention, this ‘valley of death’ time frame, where we are not exactly the new shiny, innovative startup on the block, but we are not the organization that has it all figured out and are ready to scale. We’re kind of somewhere in the middle, so trying to find funders who are okay with that level of uncertainty and buy into the idea of experimentation for a while longer. Rob Wiblin: Yeah. What do you think of the idea that the endgame ought to be that the state-level departments of health in India basically absorb your staff and then your project and just start doing it, because I suppose they have the revenue-raising capacity and maybe the economies of scale more broadly to do this in a sustainable way? Varsha Venugopal: It makes perfect sense in an ideal world. The governments should be taking this on. The challenge is that the reason nonprofits such as ours exist is because of some kind of market failure or government failure. So at the moment, the government is not taking it on to a level of rigor we would like, and therefore we exist. Varsha Venugopal: Having said that, in an ideal scenario, we would want the governments to take it on in the long run. Which is why we are working very closely with them, also so we can identify other opportunities to support them and find ways to build their technical capacity, to take this on in the long run. Rob Wiblin: It’s interesting that you say, it seems like the more effective altruist-minded donors are more interested in just delivering, just like scaling the project that you’re doing. Whereas it’s maybe the less effective altruist-minded donors who are more interested in this handoff to the government. Rob Wiblin: You might think that it could easily be the reverse, where the people with the effective altruist mindset are like, “Well, wouldn’t it be much better if we could just fund this for a little while and then have the government take it over, and they could deliver it to more people and we wouldn’t have to pay for it anymore?” Do you know what’s going on there? Varsha Venugopal: Possibly. I think it just comes down to risk tolerance. And I think even within the EA community, there are certain individuals and organizations, for instance, Founders Pledge has been involved in funding us because they think we could be as cost effective or more cost effective than the top GiveWell charities, even though we may not be there yet. Varsha Venugopal: Having said that, yes, I think there is possibly less risk tolerance for some of the larger funders who would rather have us there already before they choose to fund us. Rob Wiblin: Yeah. Okay, so the motives and interests of the funders is one issue. Are there other challenges to reaching your full scale? Varsha Venugopal: Well, another challenge is our continued focus on M&E. So when I worked in my previous organizations, the World Bank and Soros Foundation, we had strong M&E processes in place. Rob Wiblin: That’s monitoring and evaluation. Varsha Venugopal: Yes, monitoring and evaluation. And focus on better understanding how our programs are running and what’s the impact. Having said that, when I came across effective altruism and its tenets, the whole idea of using rigorous evidence as a starting point to determine which interventions to focus on was quite novel and exciting, and was part of the rationale for convincing me to move into this area. So RCTs as a starting point for choosing interventions makes a lot of sense and clearly something we bought into. Varsha Venugopal: The other bit is then figuring out when we should roll out a randomized trial for our program itself and what that could mean. So at the moment, given we have this large-scale, high-quality randomized trial that’s just being released, we think we can kind of ride on its coattails for a while longer. But the next challenge for us would be, within the next five years, figuring out how we can set up and implement a randomized trial. Varsha Venugopal: I guess an ongoing debate for us and just something to be aware of is within the broader community, we have seen organizations… funders backing off because the randomized trials produced mixed results. And some of it is a lot to do with just communicating the results and understanding the nuance of the results, which sometimes gets lost. And it’s seen as, right, the program had null effect, so you know, whatever they’re doing is not impactful. And so we are very cognizant that at an early stage, we don’t want to get kind of stuck in that trajectory and want to have additional confidence in our own dimensions before we choose to run our own randomized trial. Rob Wiblin: Yeah. So you’re saying a potential trap or a potential problem would be if someone tries to replicate these RCTs and finds that, in the second case, the SMS messages or the ambassadors don’t work or the effect size is a lot smaller. Then what would you say to your funders? Maybe you want to be able to point to work that you’ve done assessing your own impact in exactly your context, because you are measuring it, and be able to say, “Well, we do have a good sense of what impact we’re having at least.” Varsha Venugopal: Yeah. So we have some monitoring and evaluation we are conducting already, which gives us increased confidence in our work, both on the SMS and the ambassador side. And yes, if somebody else were to run a randomized trial on some intervention that broadly sounds like ours and comes out with some results which are not aligned with ours, it’s then harder to control some of that communications around it. So at some point, we would want to run a randomized trial on our own programs to of course increase our confidence. We are fairly confident it’s working, but also to ensure that we are on the right track before we scale it up to a much larger scale. Rob Wiblin: I guess, a non-crazy view might be, this has been tested quite a lot already. There’s been a big study done by the very best people who do this kind of work, and they found that it works, the effect was pretty big. It also is very intuitive that it would have this effect of getting more parents to vaccinate their kids. Yeah, it’s not a wacky intervention that you would have guessed wouldn’t work. And so maybe it’s a waste of your attention or a waste of money to be going and doing all that much monitoring and evaluation. Maybe you should kick that can down the road and wait for another couple of years before going into that. Varsha Venugopal: Yes. That is exactly our thinking for the moment. Rob Wiblin: Interesting. Okay, so the idea is you’ll do the monitoring and evaluation later on, once you’re trying to grow more. Varsha Venugopal: We definitely have a lot of monitoring and evaluation systems in place now and are putting in more frameworks, but they’re not of the quality of a large-scale randomized trial. So that’s something we want to do much later, maybe around the three- to five-year mark. Varsha Venugopal: Right now, it’s more about, are you getting the messages? We had more than 8% of our 500 people we asked saying that they probably or definitely wouldn’t have taken their kids for immunization if they hadn’t got our messages, so that gives us some degree of confidence. Varsha Venugopal: So I think it’s more about iterations to make a program more effective, but also to get some direct user feedback that’s useful at this stage. But to roll out a whole large-scale randomized trial is something we would be looking at at a later date. Rob Wiblin: Yeah. I guess there’s a certainly cheap way for you to potentially measure impact, which is a little bit harder at the testing phase when you’re only reaching a small number of people. Because if you’re actually growing a lot and you say there’s like a thousand villages on your list that you eventually want to reach, but you can’t reach them all right away, then you can do a before-and-after comparison. Where you say, well, we got to this one first and then we looked at what happened the month after and the month before. And then with the next one, we reached them a month later after and before. Rob Wiblin: Normally this before-and-after comparison isn’t completely persuasive, isn’t super persuasive because there could have been some broader trends, some other thing that happened around that same time that might’ve changed vaccination rates that wasn’t you. But if you have enough villages, if you have a large enough sample, then it’s very likely that any such broader changes are canceling out. And you also get this diversification across different time periods where you released the program. So simply doing this before-and-after comparison, if you’re large enough, could give you a very good measure of your impact. Varsha Venugopal: So that’s what we’re actually looking at now. Are there ways we can get to having higher confidence in our impact without rolling out a large-scale randomized trial? And exactly, is there some kind of before-and-after methodology or a difference-in-differences method where we look at a neighboring set of villages which didn’t have ambassadors or SMS reminders, and what does that mean? Varsha Venugopal: The challenge is we are relying on administrative data in a state where it may not be as high quality as elsewhere. And in addition, we’ve had COVID over the last year and a half. So at the moment, we’re looking into the administrative data to check the quality of it. And the next step is to try and find ways we can get closer to the impact, either through some kind of before-and-after, or a difference-in-differences method. Rob Wiblin: Fantastic. Okay. Yeah, what’s another potential roadblock to achieving your full potential? Varsha Venugopal: I think one challenge we’ve had so far, and hopefully it’s in the past, has been COVID. The first wave and definitely the second wave in India hit fairly hard. We moved all our team to remote fairly soon, and that’s part of the reason as well we conducted the remote pilot. Varsha Venugopal: With the second wave, we also tried other ways to respond to COVID. We sent out messages to do with proning. We knew there was an identifiable gap around this. I don’t know if you know what that is. It’s about lying the patient on their front to improve their oxygen saturation level. And this is something that doesn’t require a hospital and that’s something that can be done at home. That’s something, we focused on designing messages out and sending reminders on, and we did see some percentage of people clicking, and we did some followup surveys to see their overall information and knowledge on this increased. Varsha Venugopal: I think for us, COVID was also… In terms of Fiona and me being on the ground, we still managed to grow our team and communicate with our government partners. But we also realized just in terms of being closer to our end users and working closely with the government partners, it’s been a challenge over the last year and a half. But again, hopefully we are at the tail end of that. Varsha Venugopal: I think the final challenge is very much around this fundraising. As I mentioned, we have about 70% of our funds through venture philanthropy and EA-aligned funders. The challenge for us is something I mentioned earlier on this ‘valley of death,’ finding the right kind of funders who feel sufficiently convinced about our focus. There is also the whole challenge of my time. So at the moment I spend about 50% of my time looking at fundraising, and there are obviously better ways I could be spending time, focusing on M&E and program and staff hiring, et cetera. So I think from that perspective as well, if we were able to raise larger sums of funds that could cover us, then overall our organization could be more effective. Rob Wiblin: Yeah. Is it likely that COVID is going to present longer-term challenges? Or is it more, it’s hopefully a temporary thing for just a few years? And actually, what was the main problem that COVID presented? I guess you had to switch to remote work, but was it that people were unwilling to leave their house to go and get vaccinated? So it wasn’t clear what you were asking them to do? Varsha Venugopal: There was a short period of time when we had to pause our work because there wasn’t clarity if routine immunization should continue. And at some point we also had to send a different set of messages because vaccinations were only happening in certain areas and not others. So we wanted to make sure, again, from a ‘do no harm’ perspective, we weren’t sending people out without clarity on where they needed to go for what. Broadly, governments have been overwhelmed as well and focused elsewhere, and they haven’t managed to focus as much as they would’ve liked on routine immunization. Varsha Venugopal: But again, that has changed more recently when there has been more clarity that, A, routine immunization is overall net positive and needs to continue even during COVID-related lockdowns, and B, that there has been these drop offs over the last year and a half, and we need to catch up on that. Rob Wiblin: So I guess to reach 10 to 100 times as many parents, you might need something like 10 to 100 times as much funding. Do you think if we can hire the right people, if we can demonstrate that we’re roughly having the effect that we are expecting to, that amount of funding is in theory available from foundations or different places that you can apply? Or could that potentially just… Could you eventually reach some level where you’re like, “Well, we’ve gotten as much money as we can from all of the funders who are interested in funding this kind of thing, and now we’re stuck.”? Varsha Venugopal: That’s a good question. I don’t think I have it worked out. I do think, given that we can make these advancements in terms of developing an operations manual and finding pathways to scale, our overall cost effectiveness may make it so easy for us to roll out fairly quickly and get governments to buy in and send out these reminders that it would be solvable. I mean, the plan right now is to find a way to solve it, whether it’s through ourselves or through the government or others. Rob Wiblin: Yeah. To reach all these people, you, I suppose, need a team in the thousands, possibly tens of thousands. I guess, conceivably… I guess probably tens of thousands seems maybe right, if you’re going to cover all of India. How difficult is hiring as a scaling challenge? Varsha Venugopal: We have been experimenting with different rigorous hiring processes and they seem to be working well so far. If any listener has applied for any role in our organization, you would know you have an upfront testing process, which allows us to distill a large number of applicants to a fairly small, manageable number. We have tried this recently for our tech and business development roles, so we would be curious to see where we land. But more on the junior data officers, our colleagues are fairly well networked in this space and we have been fairly successful in hiring at that level, which would be the biggest hire as we expand, and we don’t see that as a major bottleneck. Rob Wiblin: Okay. So I suppose the thing that you need the most of is people to be calling up the villages, finding the ambassadors, calling them and asking them. And I guess also people collecting numbers and then sending SMSs, and that’s something that I guess can be a reasonably standardized role where potentially it’s not so hard to find people who are able to do that. Varsha Venugopal: Exactly. If we were to go with this call center model, the idea is we have this operations manual built out and we can roll that out fairly rapidly and hire people fairly quickly, and they’ll all be of a certain caliber and trained a certain way. Rob Wiblin: Yeah. I guess, to get a lot bigger and cover as many states in India as possible, I suppose in each one you’ll want to go in and try to coordinate with the department of health or coordinate with the hospitals to get all of the numbers of all of the parents of the newborns. Because that would be just so much more effective than trying to, I guess, do it doctor by doctor. Are you likely to have any issues getting coordination or collaboration from them, or are they just very happy for someone to take these numbers and run with it? Varsha Venugopal: In India one can’t work without collaboration with the government partners, so that’s a given. It’s quite hard, probably impossible, to set something up in the charity space without having their buy-in. In terms of different ways to go about it, we have already experimented with a couple of different ways. In both cases, we work closely with the government partners, but in one case we have people directly going into primary health centers with the assent of the state immunization officer. In the other case, we have managed to get the data directly from the state government through the State Family Welfare Bureau. Varsha Venugopal: So there are different pathways, and so far, as long as they see us as credible and supporting their work and their priorities, it’s been straightforward. Having said that, even with this MOU model in Maharashtra, it took several months of back and forth for us to get that MOU signed. So again, going back to that point on replicating to another district versus investing in potential exponential pathways to scale, but lack of clarity on how long that could take. You know, there are different ways to think about it, and obviously that has implications on the cost effectiveness of the model. Rob Wiblin: Yeah. Okay. Different topic. Did you consider early on potentially moving to India permanently and just running the entire organization and all of the staff from India? Varsha Venugopal: At the moment, most of the team is based in India, and that was always the plan. I am based in London, and the plan is that I go back and forth. One thing we have heard from Charity Entrepreneurship, and I think it’s fairly standard advice by Y Combinator and others, is the role of a co-founder. I think I was quite lucky early on to find Fiona, and the idea is that she would very soon be moving to India and be based there. So most of our team will be based there with me going back and forth as and when needed. Rob Wiblin: Yeah. Is there a benefit to having you in London? Because I guess potentially for fundraising, it might be better, or there’s other people you want to coordinate with, where it’s perhaps easier to do it in Europe? Varsha Venugopal: Is there benefit for me being in London? Yes and no. Yes from the perspective of potentially linking to funders in this part of the world and maybe also in the US. It’s more a personal decision because I’m on a certain trajectory and I already was before I came into this. It’s much harder for me to move. If I didn’t have those restrictions… I mean, I have a family here and if I could convince all of them, I would probably move to India. Rob Wiblin: Yeah. Yeah, yeah. You’ve put down some roots in London, so you can’t just uproot them all. Yeah. I’m guessing you might have some family in India or perhaps have spent some time in India before you started this organization? Varsha Venugopal: Yes, I have family in India and I was born and I grew up there. Rob Wiblin: All right, yeah. Was that a substantial advantage as a co-founder? Do you think maybe… Would Fiona have found it more difficult without someone who was more familiar with operating in India? Varsha Venugopal: Possibly, yeah. So India is of course a big country and my middle class experience may not be the same as people in other regions and different income levels. But having said that, because I grew up there, I’d like to think I can tap into my personal and professional networks. Maybe I also understand some of the local context and politics better, and I’d like to think I can also relate better to my team and the end users in different ways. Varsha Venugopal: I’m also a mother, which maybe gives me a personal understanding of the challenges that parents face and thought processes they may use to make decisions. I think something I mentioned earlier, being aware of one’s strengths and limitations and actively seeking a co-founder to complement them could be valuable for any startup. So Fiona, she’s much younger and she’s closer to our team’s median age, I think, and she manages our team on the ground. Varsha Venugopal: So I think the fact that we bring complementary skills and experience allows us to run the organization in a way that we may not be able to do by ourselves. For instance, one of our values is around ‘think and act.’ And I find it so interesting how Fiona’s standard is always, “Let’s sleep over it.” And I tend to kind of jump the hoops and reach a decision sooner. And so we somehow seem to land in a better place, we think, in terms of decision-making as a result of some of that difference in how we approach some of these big decisions especially. Rob Wiblin: Yeah. I think, both having the desire to mull things over and the desire to make a decision and move on is always very useful to have both of those tendencies competing, I guess, when you’re trying to manage something. I’ve been told that it can be useful having me in a meeting because I always want to end meetings and just make decisions and end them. And obviously that would be terrible if it was the only way that you’re doing things, but it’s like you want to have a flavor of different dispositions towards that kind of thing. Rob Wiblin: Has anyone been annoyed by what you’re doing and made your life difficult over the last couple of years? Varsha Venugopal: I think our goal is pretty universally supported by everyone, which helps to make our work far more tractable, and we are not pushing against an opposing force. In fact, if anything, over the last year and a half I think there’s been an increased understanding of routine immunization within government and also some of our peers and networks. And there is an overall support and appetite for the work we are doing. There may be some other lines of entrepreneurship, like tobacco or alcohol regulation, where there may be more organized opposition, which may reduce your odds for success. So far we haven’t encountered that. Rob Wiblin: Yeah. Yeah. I was trying to think who might be annoyed. I suppose anti-vaxxers perhaps, but maybe they’re operating more in other countries. What about, has anyone been annoyed that you’re kind of spamming them with these messages or perhaps the immunization ambassadors annoyed someone by hassling them too much? Or I guess it just hasn’t been an issue so far? Varsha Venugopal: We take consent quite seriously, and India is also currently planning to pass laws similar to the EU GDPR laws. We do have an opt-out process at the start of our text messages. Very few parents have chosen to opt out. I think around 0.1%. I think parents overwhelmingly say the messages are useful. In our surveys, amongst those who remembered receiving the message, more than 99% said they found it useful and wanted to continue. So, so far we haven’t seen — Rob Wiblin: Any meaningful backlash. Varsha Venugopal: — extreme backlash on spamming. Rob Wiblin: Okay, that’s great. I guess we’ve talked about the possibility of this being done by government or possibly even absorbed by government at some point in the future. What about going the opposite way? I was imagining an alternative model for trying to get vaccines to everyone might be that the Indian government starts effectively paying a bounty for anyone who’s able to immunize a child who otherwise would have gone unimmunized. Rob Wiblin: And so you could imagine, I guess, semi trying to privatize this by just making it the case that any company in India could potentially try to make a profit by using their marketing ability or their logistics ability to immunize as many people as possible, and then go to the government and get paid for it. Does that sound like a crazy idea that only an economist like me would come up with, or might there be an actual useful nugget in there? Varsha Venugopal: What you’re describing seems like something we call an impact bond, it’s a social impact bond and in development spaces it’s called a development impact bond. So no, it’s a great idea, and there have been some innovations around this. The challenge is getting the government to pay for it. So in the impact bond model, you would have an investor who puts in the money and then you would have the government or some philanthropist who would be paying the investor the returns, if we meet certain milestones. Varsha Venugopal: It is something we may want to explore ourselves, even from a philanthropic perspective, if a philanthropist were to say, “Right, I would pay you a certain amount, but the next tranche would only be released if you meet certain milestones around number of people reached, et cetera.” So, it’s absolutely plausible. The challenge is, in the past with impact bonds, is getting governments to fund or getting… The grantor bit is usually the challenge. Rob Wiblin: Yeah. Yeah. Interesting. I guess I thought of that, because I was like, this seems like a sales issue. It’s like, who’s really good at marketing, who like, annoys me all the time, trying to get me to use their services? It’s like businesses are the experts in that potentially. Although, I guess of course you could hire people who have marketing expertise to try to figure out what should the SMS messages be. Maybe it’s a slightly odd fit for the immunization ambassador, one that perhaps is better as a nonprofit. Varsha Venugopal: Yeah. I think the big question is who’s paying them, or any company, and that’s where it comes to naught. We have been, especially me, when I think about my time in fundraising, I am quite keen to see, are there any ways we could make this revenue-generating? Sadly, the kind of people we want to reach are not the income-paying community, and there’s no way we could figure out a finance model there. So yeah, impact bond is the only one I have actually come up with as well, where at least there is some clear incentives tied to certain milestones and some funding attached to it. But beyond that, yes, it seems very clearly in the charity nonprofit space. Rob Wiblin: All right. Let’s go a bit back into the history and talk about Charity Entrepreneurship, because I think potentially quite a lot of listeners who are thinking about how they might be able to have more impact by changing what they’re doing could be interested in potentially going through this program. It describes itself as, “…an effective altruist organization that helps start multiple high-impact charities annually based on extensive research. […] Our mission is to cause more effective charities to exist by connecting talented individuals with high-impact ideas.” Rob Wiblin: So yeah, some of the things they’ve started other than Suvita: people who are working on policy change to improve the welfare of fish in aquaculture, people who are going on working on policy around cigarettes in developing countries, and also groups that are trying to encourage reductions in lead exposure, I think in particular, in developing countries again. So what sort of services did you get by going through this program? Because I suppose, well, both actually you and Charity Entrepreneurship were very much in the early stages back in 2019. Varsha Venugopal: Yes, I was part of their first cohort. I think they just had their third cohort go through this summer. Each year I think they have hundreds of research hours that they go through to identify these promising interventions, which is how they came up with conditional cash transfers as a starting point for us. So as part of this program, they provide these two months of free training to selected entrepreneurs, and you go through quite a rigorous process to get selected. In addition to that, once you are through, they also offer some free legal training, a workspace, and I think the biggest thing for me was connection to funders, peers, and advisors who continue to motivate you and ensure that you remain focused. Rob Wiblin: What did you find most useful about the program? I suppose it was Charity Entrepreneurship that shortlisted this intervention, is that right? Varsha Venugopal: Yes. So the starting point was Charity Entrepreneurship had looked into several interventions, and within the global health space, the intervention they had looked into was conditional cash transfers. I think some of this research they had already done in the past when they had looked into setting up Charity Science Health, which was focused on SMS reminders for routine immunization. So, that’s where we came in, saying, “Right, SMS reminders or conditional cash transfers for routine immunization. Is there something else that could beat that?” What they required from us was full-time commitment. They had lectures, activities, and presentations by others in the EA development space. Varsha Venugopal: So in my cohort, some of the organizations you mentioned, we had Happier Lives Institute that was focused on mental health. We had colleagues who then went on to start Fish Welfare Initiative, Animal Ask, and Animal Advocacy Careers. As part of this, going through this incubation, we also interacted closely with colleagues at Fortify Health and Charity Science Health. Varsha Venugopal: And I think the CE support was catalytic; I wouldn’t have founded this charity in their absence. And something I mentioned earlier, this support of funders, advisors, and peers was quite instrumental in continuing to motivate us and keep us focused. I understand CE has incubated 18 new charities. I think they’ve already fundraised $3.5 million. So they seem to be doing something right so far. Rob Wiblin: Yeah, yeah. Did they help with fundraising in the early stages? Or do you get a grant if you go through the program? Varsha Venugopal: Exactly. So in addition to this training and the network of mentors and advisors, one additional thing which I know is quite promising and useful for a lot of people considering this option is they provide seed grants for up to $100,000. I think in our program, there was some clear amount that was dedicated to all the charities that were founded. Varsha Venugopal: Moving on, I think the last couple of years it has moved to an online version, which I think they may be continuing, but also the amount may not be available for all entrepreneurs. I think there is some degree of competition maybe at the end, where they have to send in their proposals based on which certain ones are selected. But broadly they do have a pot of funding that does go out to the most promising charities. Rob Wiblin: To what extent are they helping by legitimizing the decision to quit the other things that you’re doing? I imagine there’s plenty of people who could imagine going and starting something new, but then perhaps it feels very risky to do it, or they don’t quite feel like they have the confidence to strike out on their own. Maybe having people select you from a competition or a competitive pool of applicants, saying, “Yeah, you should absolutely do this and we’re going to help you do it” gets people over the line. Varsha Venugopal: Possibly. I think the way I would see it is… You’re right. There is an element of legitimizing the process. I think they also kind of carry you through the whole process, which you otherwise wouldn’t take on. So I think two of the key bottlenecks that they are responding to, one is there are a lot of outstanding intervention ideas that aren’t being implemented, and the other is founders starting out without a support network. And creating avenues to respond to both of that allows I think a lot of potential entrepreneurs to take that first risky step. Varsha Venugopal: I think also very soon they plan to have a quiz on their website, which allows you to then make a call on whether this is a good fit for you before you apply. But broadly, anybody who’s interested, I would say just go and apply, and just going through the process may help you come closer to the decision. Varsha Venugopal: I think the other bit that I have observed is that, even after going through the process, there may be different pathways one could take. I have seen people start charities as a part-time founder. Somebody has started a foundation to find other impactful charities. I know people who have gone through the program who have then been absorbed by other impactful charities, that they’ve gone on to be employed by other organizations. So, I think even once you’ve gone through the process, you may have several avenues to explore. Rob Wiblin: Yeah. Nice. Have you stayed in touch with the other founders who went through the program? Varsha Venugopal: Yes. And that’s, I think, been a big part of the plus, feeling that you are not alone and doing this all by yourself, which can be quite common within entrepreneurs. They offer not just mentors that I regularly talk to, such as Patrick who’s director of their communications, but we also have these Slack channels where we discuss operational challenges and other areas. For instance, registration in the UK, which is a standard challenge for several nonprofits. So they have found an organization to provide legal advice pro bono for all of us. So there is some economy of scale in operation, which helps all of us by being in touch and by continuing to be in touch with CE. Rob Wiblin: Yeah. For listeners who are thinking they might want to apply, do you know what kind of selection criteria they’re using to decide who to admit to the program? Varsha Venugopal: I would say if you were to take the quiz and you were already inclined to consider this as an option, you should just go ahead and apply. Rob Wiblin: As I understand it, Charity Entrepreneurship had started this previous thing, Charity Science Health, a couple of years earlier that was already doing the SMS message reminder thing, and then you kind of merged with them. What’s the story there? Varsha Venugopal: So while going through the charity incubation process, we got close to Charity Science Health, which was focused on SMS reminders to improve uptake of routine immunization in India. And while we were following this evidence base for immunization uptake, I think both of our organizations independently concluded that a combination of SMS reminders and ambassadors was the most cost effective and impactful. So as a result, it seemed fairly logical for us thinking of scaling up to join forces, so we could build on the infrastructure CSH already had, including some of these government partnerships, and we could also jumpstart the ambassador program. Varsha Venugopal: I think while we were considering the promisingness of the program, in addition to the evidence and the outstanding cost effectiveness from the attractability perspective, the authors had just won the Nobel Prize, which made it all quite exciting. So I think we decided that it made most rational sense for us to merge as an organization, and Kat and Erik who were leading the organization at that time are still very involved and are advising us, and they have moved on to other impactful things: Kat’s now with the World Bank and Erik is working with Michael Kremer at the University of Chicago. Varsha Venugopal: So, maybe another data point for people considering the entrepreneurship track, that one could then move on to other impactful areas. And there seems to be an openness, I guess it’s all seen as positive having been an entrepreneur in the past. Rob Wiblin: Yeah. Yeah. Makes sense. So you found your co-founder Fiona through this program. How essential do you think it is to have a co-founder if you are starting a project like this? To me intuitively, I imagine doing something like this on my own, and I’m like, “That would never work because I would get demoralized and then there’d be no one to pick me up when I was feeling down or facing challenges.” Do you have the same sense? Varsha Venugopal: Yeah. I think for me, given where I was with my life trajectory and knowing I couldn’t possibly move to India, it made so much sense to have a co-founder who could actually be based in India. But even beyond that, the advice we got from CE and others is that having a co-founder is immensely useful in complementing some of the strengths and weaknesses and limitations you may have. Varsha Venugopal: And from that perspective, it’s definitely been hugely useful to have someone who is just as stressed about carrying through with your day’s work. And yeah, just in terms of entrepreneurship being a lonely journey, having somebody whom you can constantly throw ideas back and forth and debate on how to take some of our decisions forward is immensely useful. Rob Wiblin: It’s interesting that you’ve done this co-founder split where Fiona’s doing most of the management and I guess you’re on fundraising and strategy and planning and that kind of thing? Varsha Venugopal: Yeah. It was broadly from this idea that she would be based in India and most of our team would be based there, so it would be easier for her to take on that role. And it’s worked out mostly fine, so I think we would be continuing with that model. It would be much harder for me sitting here to be managing a team on the ground. There is also this element of people being mostly in charge of their own selves. So the management is fairly light touch and more in terms of, “How can we support you to do your work better?” Rob Wiblin: Yeah. That makes sense. I suppose the relationship between co-founders is so incredibly important for an organization’s success. Do you and Fiona spend time socially together to ensure that any potential frictions kind of get smoothed over? Varsha Venugopal: Yes. Before our first trip to India, we spent quite a lot of time together, but even otherwise we are on call and chat pretty much all the time. Just again, I think just the nature of startup in early stage and lots of moving bits and pieces… I think we pretty much clicked fairly early and continue to be clicked. Yeah, I think we bring very different perspectives and skills and somehow are complementary and seem to agree on most things, and where we disagree we manage to work through in a fairly nonviolent way. Rob Wiblin: Did you do a lot of screening to figure out whether you were a good match before you started the project? Or was it kind of, “Well, we’ll try getting the project going and that will test the relationship”? Varsha Venugopal: I think CE has gotten pretty good at this. So they did some kind of… I think they even put us in some cat and dog category. They never told us which one of us was a cat or a dog, but I think they did a lot of behind-the-scenes calculations, I guess, based on several exercises we did with different people within the incubation. So I think there was some of just trusting their judgment. But also I think it is such a fundamentally important decision to get right. That it’s okay to, I guess, try out early on and see if it works or not, like probably with a lot of these decisions. And if not, being, I guess, brave to — Rob Wiblin: Go find something else? Varsha Venugopal: — to walk away. Yeah. Rob Wiblin: Yeah. Are there any ideas that Charity Entrepreneurship has shortlisted as promising interventions that people should scale up that you are particularly excited by, that you’d like to see listeners maybe jump on? Varsha Venugopal: I know for next year CE plans to focus on international policy as a priority course area, and I know they plan to incubate a total of five charities that are significantly more cost effective than EA-recommended direct delivery interventions. So I let them, I guess, focus on the policy interventions. The one area I am keen to see grow is this effective giving movement in India that has been led and supported by The Life You Can Save and others. Varsha Venugopal: I get approached fairly regularly by university students and others in India, interested in learning more about starting effective charities in India, but also in effective giving. And my understanding of the key bottleneck there is that most Indians are most interested in giving to charities in India, and there isn’t a clear credible list of potential charities that are highly evidence-based and cost effective in India. Varsha Venugopal: There has been some attempt in the past to figure out which of the top GiveWell charities are also functioning in India, but the challenge has been that once you donate funds to them, they go into a central pool, so it’s hard to ring-fence the amount that’s going directly to impact lives in India. So I’m quite keen to make a call out there for some kind of individual or institution to take on that role of exploring evidence-based, highly cost-effective charities that could make the cut for effective giving in India. Rob Wiblin: Yeah. Do you have a view on this question of how broad the program should be that charities should run? So to explain what I mean, I guess one vision for Suvita might be that it’s really good at delivering the SMS and the ambassadors, and then it should expand into all kinds of other health interventions in order to have more impact and use the expertise that it’s developed. Rob Wiblin: An alternative might be, no, Suvita should specialize in this one thing and absolutely kill it and just get as big as it can with that, rather than diluting itself into other areas where maybe it’s not going to be quite as strong. And if someone wants to deliver a different intervention, they should start a new organization that just nails that other thing. Yeah, do you have a view on that? Varsha Venugopal: That is a great question. We haven’t figured out the answer yet. What we have agreed on, Fiona and I, is our focus is on this problem of underimmunization and that’s what we want to resolve first, before we move on to other things. Having said that, there have been, maybe we can call them distractions throughout the process from funders or otherwise. For instance, in India, we had a foundation that was interested in using our SMS reminders for students in schools on a whole host of behaviors around taking the folic acid or being clean, et cetera. And we went some way in there before realizing, okay, this was distracting and probably taking us away from our core focus. Varsha Venugopal: Another example, which is probably more, I guess, salient, is around COVID vaccines. Can our nudges, especially around ambassadors, be useful for getting more people to take their COVID vaccine? And this was something that came up in some earlier discussions with our colleagues at J-PAL and they have been looking further into it. We have also been following the research in India on the different reasons for hesitancy. We are aware it’s very different from routine immunization in terms of drop offs: A, we are talking about adults, B, it may not be high-trust area, and C, there may be other challenges around cost — with routine immunization, it’s provided free of cost, whereas for COVID vaccine, there may be a cost element. Varsha Venugopal: So, all of that makes us fairly hesitant to enter that area, but it’s not something we have completely ruled out. And that’s also an area where CE has been quite helpful and useful having mentors and advisors who keep nudging us back on track, so we don’t get distracted by all these other huge problems where we could potentially make a dent on, but would take us away from our core focus. Having said that, this whole immunization ambassador model and this idea of systematically identifying community nodes is really promising and exciting and possibly has other low-hanging fruits that it could have an impact on. We are keen to explore that at some later date, hopefully not too far out. Rob Wiblin: Yes. Speculating, what else might the ambassadors be useful for spreading? The first one that jumped into my head was getting people to quit smoking, but I suppose that’s maybe a much heavier lift. You’re shaking your head — Varsha Venugopal: Maybe cash transfers, there has been some research, but I don’t think ambassadors would be sufficient. The ones we have looked into in the past, and may be promising because we reached pregnant women already with SMS reminders, would be around taking folic acid or making sure they understand when there is any kind of complications on whom to reach, making sure they focus on institutional delivery. So… Breastfeeding, tetanus toxoid injections for the pregnant women, even before the child is born. Varsha Venugopal: So I think those are the most, I guess, low-hanging fruits. Maybe there’s something around vitamin A supplements as well. But beyond that, we haven’t actually looked into the range of possibilities that could exist. I think it’s also useful to note, this is an open question for us with the J-PAL paper, on how the ambassadors work versus SMS reminders. It’s something we still need to investigate further. So, we think SMS reminders are increasing salience. It’s something you already think is a good thing, and this serves as a reminder. Varsha Venugopal: Whereas with ambassadors, there is a likelihood they’re actually changing norms and trust in the community. Which is why beyond the seven million kids in the country, we do think maybe there is a way even those other three million kids who are either just getting birth dose or not even being born in primary health centers. If there was a way the norm in the community were to change, and immunization was seen as a cool thing to do, they would come in for completing their routine immunization as well. Rob Wiblin: Do you know what was done in countries like the UK or the US to get really high levels of childhood immunization in the past? Varsha Venugopal: Well, I know as a mom, I got a red book when I delivered at the NHS, and that meant I got text messages and constant reminders on when to get vaccines. And again, I do know that served a huge purpose, when you feel so overwhelmed. It does help to keep you on track. Rob Wiblin: So it’s actually not so different, this intervention, than what the UK is already doing, for example? Varsha Venugopal: Exactly. It’s just that when you think about low-income parents and carers, there is a whole host of additional ways your life is complicated and overwhelming that just makes it harder to remember to go get those vaccines. Rob Wiblin: All right. We’ll stick up a link to the Charity Entrepreneurship website, where I guess you can find out about the process for applying and take a look through all of these really interesting white papers that they’ve written, where they look at different interventions where no one’s particularly trying to scale that intervention at the moment, and evaluate them on a whole bunch of different criteria. Like how practical is it? How cost effective might it be? Rob Wiblin: Yeah, it seems like a really cool program. I’ve enjoyed looking at some of the other organizations that have gone through Charity Entrepreneurship as part of prepping for this interview. It’s pretty inspiring to see people out there doing things on the ground and not just releasing podcasts. Let’s talk maybe for a minute about current opportunities to work at Suvita. For people who might like to get involved, what roles are you currently trying to fill? Varsha Venugopal: We had a business development role that I think just closed this week. In the future we plan to have some monitoring and evaluation roles open up. I would say broadly, because we are a startup and move fairly quickly, if you are interested in this line of work, please take a look at our website and sign up for our newsletter. Rob Wiblin: Yeah. Do the jobs mean moving to India or is there going to be a London office consistently as well? Varsha Venugopal: I am quite keen to move as much of our team to India as possible, given there is a surplus of talent in there. Even within this space of evidence-based, cost-effective decision-making and nonprofit sector, there are fairly well-developed organizations there and potential staff members. So, I think the business development person, for instance, may be based anywhere, but for most of our operational team moving forward, the likelihood is they’d be based in India. Rob Wiblin: Yeah. Are there any particular skills that are most limiting your ability to grow? Like technology, or management expertise, or familiarity with logistics and so on? Varsha Venugopal: I think based on CE’s advice and even otherwise we’ve realized operations was one of our crucial roles and I know that’s something EA and CEA have been promoting as well. So I think that’s something we have mostly figured out, I’d like to think. So we have a head of operations now. At the moment, we do think there are some exponential leaps we could have with a head of tech, which is why we are going for this role. I can tell you more about — Rob Wiblin: Yeah, yeah. Go for it. Varsha Venugopal: — the tech role. So as we move from this pilot to replication and scale-up phase, we want to find ways to leverage technology optimally and expand our reach and effectiveness. And to do that, we think a head of technology can be quite useful. So the main aims of this role is to identify priority opportunities and bottlenecks to design solutions, both by talking to us team members, but also possibly talking to the end users that we are trying to reach. Also, possibly liaise with external tech providers and tech-focused funders as needed. And finally, there is the whole data security and broader regulatory standards and norms around transparency, and we need someone on the ball who can ensure we are meeting those requirements. Varsha Venugopal: What we think we need from this person is that absolutely you are motivated by the satisfaction of building something great and are passionate about this opportunity to see these solutions immediately have an impact. The bit about years of experience we think is not that essential. I think experience in the design and implementation of software solutions is key. You may have done it in a large organization or not, but we do think if you have built something in a startup environment, or have single-handedly completed some impressive side project, or you have some project lead experience in a software team, all of that could be quite useful and relevant for our role. Rob Wiblin: Yeah. To what extent is it useful to have a background in global health and development versus just having really developed some particular strong skill in whatever area? Varsha Venugopal: Yeah, I think it quite often depends on the role. I think for some of these technical roles, you’ll obviously need those particular hard skills, whether it’s on technology or M&E, but for some of these other ones like this business development role we are hiring, the key skill is someone who can do scientific writing. So just someone who is sharp and can get on with it and is basically not somebody with strong technical skills in any area or development and health background. Varsha Venugopal: I think for some of these roles like M&E that we may be hiring, there would be a requirement that they do have a development background and experience, because it just makes it so much easier to understand the context on the ground. And if they already have run a randomized trial or they understand how some of this works, that’s super useful. Varsha Venugopal: I think one big thing for us is, we work quite hard to be evidence-based in our hiring systems. It’s still a process we are trying to learn from. We try to minimize our implicit bias and focus carefully on candidate skill and fit for specific roles. For instance, focusing on our test task outcomes rather than too much evidence on years of experience. It’s still a work in progress and we need to see where we land with some of this. Rob Wiblin: Yeah, hiring is difficult. What fraction of your total work hours go towards hiring? Varsha Venugopal: I would say more than we would like at the moment, but the way we also justify it is we are learning so much from this process because it’s all a lot of experimentation for us. So this idea of, especially with India, to go for large quantities and then figure out a way to have questions upfront to funnel that, I think it seems to be working out. Varsha Venugopal: It’s also, even though we try to do some kind of persona mapping upfront, we realized when we go through this interview process, we ourselves change our minds on what we prioritize and how much and how much weight we should give it. So I think we are investing more time than we would like because we see this as a learning process and an experimentation. But because we are also carefully codifying some of our reflections and lessons, hopefully the next iteration would be much faster. Rob Wiblin: Yeah. I wonder… maybe it’s not so bad if you’re spending a huge amount of time on hiring. At least I’ve heard stories that many very successful organizations, early on, an astonishing fraction of staff time was spent on getting exactly the right hires and figuring out exactly what roles you actually need to hire for. Rob Wiblin: It makes some intuitive sense because the cost of a bad hire could be pretty famously quite large, and the benefits of getting exactly the right person into exactly the right role can be huge in terms of then unlocking this potential to grow more. So maybe it’s not like… You could spend half of your time on hiring and actually that’s getting you a lot of leverage maybe. Varsha Venugopal: Maybe. Rob Wiblin: Okay. Yeah, someone told me a story recently that in the early days of Google, that senior staff were spending 40% of their time on hiring, which I think they also thought was extremely high, but then I guess it seemed to work out for Google. Rob Wiblin: So do you need people who have experience in India, perhaps working with health authorities, in order to build and develop those relationships with health departments who seem really essential potentially to scaling up at least the SMS part of things? Varsha Venugopal: Yeah. For both programs, and in fact for any nonprofit in India, one needs to have close relationships with the government. In our case, we recently hired program managers, one for Maharashtra and one for Bihar. And one of their primary roles is maintaining that relationship with the government and also looking out for additional opportunities where we could support them, which helps us in the longer run, but also finding additional avenues where we can grow our immunization programs faster. Rob Wiblin: Yeah. Yeah. And I guess for those who can’t see themselves moving to India or working for Suvita, but are earning to give and want to fund this sort of thing, there’s a whole bunch of information on your website. You link off to the J-PAL papers and you’ve got your literature review up there. I think there might also be a review of the evidence on this topic on the Charity Entrepreneurship website (cash transfers, SMS reminders) if people are interested to learn about the intervention. Varsha Venugopal: I would also encourage you to sign up for our newsletter, we hope to send out fairly soon. We are also planning to publish our pilot results and a few additional research papers that we’ve been working on. Rob Wiblin: Nice. It sounds like you’ve dealt with a bunch of different funders as you’re trying to get this off the ground and then scale it. Do you have any advice for funders on how they could approach projects like this better, or how they might be able to improve their thinking in order to have more impact? Varsha Venugopal: I think having a clear sense of your own values and also your risk tolerance would help you make a decision on whether Suvita broadly fits in that category. At the moment, we do think we could be somewhere in the top GiveWell charity range. We are funded by individual EA funders as well as Founders Pledge, which has done its own cost effectiveness analysis. And I believe they think we are about four times GiveDirectly in terms of cost effectiveness range. Rob Wiblin: Okay. Let’s push on and talk about things that you’ve potentially learned about management and founder careers more generally, maybe over the last two years or over the whole of your career. Which past experience or training before you started Suvita do you think was most valuable in preparing you to be able to actually do what you’re doing now? Varsha Venugopal: So I had most experience on applied research at the World Bank and then the Soros Foundation, that’s the NRGI. It was more recently at Options, which is a subsidiary of Marie Stopes International, that I was doing much more work on operations, so looking at hiring, where is our money going? What are our margins? And I found all of that around managing large programs and budgets and hiring staff quite useful and very relevant to a lot of the work I’m doing now. Varsha Venugopal: I mentioned earlier, I know 80,000 Hours and the Centre for Effective Altruism have been quite focused on the role of operations in organizations, and I think CE also recommends ops as one of the first hires, and I completely agree with that. I think there’s a lot of learning there. Varsha Venugopal: I think… I was asking my co-founder this question earlier. I think my knowledge of the development space and the key players involved has been quite useful for us to navigate our conversations. And I think what I hear from her is that I’m able to get the big picture fairly quickly and get a clearer understanding of how we could fit in with other peer organizations or even with the government, and what their expectations could be. And I think some of that comes from the experience of having worked in that space. Rob Wiblin: Yeah. Operations is a pretty huge class of activity. Are there any particular sub-parts of operations that feel particularly important to have expertise in or hire people into early? Varsha Venugopal: So I think for Suvita and probably the same for a lot of nonprofits, the kind of operations role we’ve been looking into, it can take on a lot of bandwidth that can otherwise be spent on other areas, such as programming and hiring itself. So for us, operations has been a lot to do around administration. Registering in the UK, for instance, it’s not a one-off process, there’s a fair degree of back and forth with our legal advisors and getting it to that point. Varsha Venugopal: Also, developing some of those systems around safeguards and hiring. And a lot of it around… Right now we are working through the US and UK and India. So just in terms of transferring money, and what does that mean in terms of exchange rates, and how do we make sure we are within laws while we do some of this… It’s something I just want it to go away and not deal with, and having somebody who is on the ball with that is hugely valuable. Rob Wiblin: Yeah. Is there anything in particular you look for in a hire for a head of operations role? Varsha Venugopal: We again went through a fairly rigorous process in terms of test tasks to ensure they had that attention to detail and could also do some kind of whiteboard thinking. I think my one broad lesson coming out from our hires so far is that someone who’s worked in a startup environment and kind of that white space is able to take some of this on fairly quickly and run with it in a way that somebody who has been embedded in larger institutions or in a single kind of culture finds harder. Whether it’s a question of quick pivots or just being able to produce the first draft of something on how to take it forward is just something they have done before, and they’re able to wrap their head around it faster. Rob Wiblin: Yeah. I guess the challenge for a head of operations with a new organization is that there’s no manual, there’s no procedure for doing lots of things. They just have to be willing to dive in and grapple with an accounting system that doesn’t exist yet, and make it exist. Varsha Venugopal: Yeah. And also just ideas around risk, right? Because there is that, I mean… Again, having worked with larger institutions, there is that perfect accounting system one could have. So it’s more a question of what is that MVP, that minimum allowable product, which we can use to get to the next level and then possibly design more efficient and robust systems, but may not want to start out with the perfect product from day one. Rob Wiblin: Yeah. What do you think made you willing to quit a quite normal, secure, sensible development job in order to go and start this organization, when so many other people are not willing to take that kind of step? Varsha Venugopal: I think a lot of this has to do with real or perceived safety nets. In my case, because I have done some of these things that people may have a vague list in their heads on, makes it easier. So for instance, you know, having a family or wanting to travel or make money, I think those are all sensible aspirations, and it’s harder to decide not to pursue any of those in favor of something like entrepreneurship, in which case, you may need to make certain sacrifices. So I think for me, it was just easier because I had checked some of those boxes and maybe had some ideas around how much I value them and what I was willing to let go. Rob Wiblin: Yeah. So potentially you’d saved money, you’ve been working for a while at quite a lot of like, serious jobs, and I guess perhaps you’d saved some money and felt a bit more financially secure to jump off and do something, something different that may or may not pan out. Varsha Venugopal: Yes, that’s true. And also I have a partner who continues to make money, and I think maybe there is something there in terms of choosing partners who may be broadly value-aligned. That does help in making decisions such as this. Rob Wiblin: Yeah. This feels like a slightly odd thing to dwell on, but I think it is super important. Because for people who want to go into entrepreneurship, or just in general their plan for impact involves taking risk later in their career, we often suggest it’s quite valuable to save money. Because that just might be the thing that prevents you from doing it later on, if you don’t feel like you have a reasonable financial safety net. Rob Wiblin: And I guess with partners as well, it makes sense potentially for one partner in a relationship to have a more risky career and the other one to have a more solid career, and you can get a joint optimization perhaps between you that allows you to both have a good personal life and prudential considerations as well as have more impact. Varsha Venugopal: Yes. And ensure there is buy-in into each other’s careers, because in my case there is also me just dropping balls and going off to India, and I need to make sure my partner steps in with childcare. So, it’s not just financial. He has to buy into this model in more ways than one for us to make it work. Rob Wiblin: Yeah, so he’s supportive of having impact with your career, that’s an important priority for him as well? Varsha Venugopal: So far, yes. Rob Wiblin: Okay. Cool, cool. What’s been most difficult about founding Suvita that people might possibly not suspect? Varsha Venugopal: Well, this is fairly boring I guess, but five months in having to face COVID, which was quite world-altering, was quite huge in more ways than one. One was of course, like I mentioned, on government relations and staff motivation. But also we had plans to raise more funds in India and us not being physically there, it didn’t allow me to build on some of those networks, and the economy the way it is, I’m not really sure it’s an area we would be prioritizing anytime soon. Rob Wiblin: Yeah. What was your approach? I think some organizations, they froze their projects kind of, and tried to save money and they were like, “We’ll call you, staff, when we’re ready to kick things up again.” Did you consider doing that, or indeed, possibly do that? Varsha Venugopal: We definitely considered all alternatives. Because we managed to pivot to a remote version in response to COVID and we managed to hire teams on the ground, we had certain colleagues already who we could trust to then hire additional team members. We were able to progress despite the COVID-related lockdowns. I think the bigger one was just motivation of our team members and for us to ensure we did, you know, weekly some version of hanging out in the pub with all our team members to ensure we all continued to be motivated and focused. Rob Wiblin: Okay, so it was people being… It’s potentially hard to motivate a workforce that’s used to working together to do stuff at home. Or you have to start thinking a lot about people’s mental health and how are they doing. Varsha Venugopal: Yeah. And in India, the COVID phase was quite bad. So it was not just about working from home. Pretty much everyone was also personally affected with their family members. So it was quite hard for extended periods of time to be stuck in your room or your home and to continue being motivated and then get these negative messages about your friends and families and networks. Rob Wiblin: Was there anything you did to help maintain your mental health during COVID? Varsha Venugopal: So I think as a team, we had this chai pakora Zoom event every Friday or every second Friday, and that was quite useful for a lot of us to not talk work and to be able to talk to each other. I think more generally for me personally… I guess, meditation, exercise, all the common ones definitely helped with sanity. Rob Wiblin: Yeah. Makes sense. Are there any instructive mistakes that you’ve made at some point in your career that you’d be open to sharing? Varsha Venugopal: I think thankfully not any major ones yet. But the one we do recognize as a distraction is the mission drift, which was quite real with this potential COVID vaccine. We had a lot of interest from funders to see how we could respond to that, for instance. And again, having colleagues within CE and otherwise nudging us to continue remaining focused has been hugely helpful. I think that’s the biggest one for us so far in terms of potential mistakes or almost mistakes that have been distracting us from our core focus. Rob Wiblin: Yeah. In thinking about whether to do more COVID work, I suppose one issue is, you’ve specialized in doing this other thing, which is getting kids vaccinated, and maybe your programs work better for that. There’s also just the issue that COVID is quite deadly, and so are all of these other diseases that you’re trying to get kids vaccinated for so they don’t die. And doing the thing that’s most topical, like getting people vaccinated for COVID, might just seem straightforwardly worse on a cost-effectiveness point of view than trying to get more children vaccinated with even more effective vaccines against even more deadly illnesses. Varsha Venugopal: Yeah. So we know for instance, measles is six times as infectious as COVID and there have been real challenges with potential outbreaks of measles. So I think once we had more data coming in on how COVID was then having an impact on routine immunization, and also lack of clarity on how our programs could be most impactful on COVID, it made it easier for us to refocus on routine immunization. Having said that, it’s still an ongoing question for us on whether there are some low-hanging fruit areas we could impact in terms of COVID response. Rob Wiblin: Yeah. Yeah. Do you have any general life advice that you think is underrated? Varsha Venugopal: I think one experiment I have done over the last few years which has served me well is around putting myself in uncomfortable situations. It’s almost like building a muscle. I think there’s a whole classic book on this, which is called Feel the Fear…and Do It Anyway. And especially, I think as an entrepreneur, you get thrown in uncomfortable situations quite a lot, and I think consciously and deliberately choosing to do that in your daily life, whatever that is, buying something expensive you otherwise absolutely wouldn’t or talking loudly somewhere where you wouldn’t. It just, I think in a way, helps you increase your comfort zone and helps you face some of those uncomfortable situations better. Rob Wiblin: Yeah. I guess, well, talking more loudly in a public space isn’t super important itself, but I guess the idea is you’re training this general ability when it does actually matter at some point to be able to violate convention or do something that you haven’t done before. Varsha Venugopal: Yeah. Another one my colleagues talk about is the idea of collecting of failures, which I think is super useful for entrepreneurs to think about. I know in fundraising it’s almost a stat, right? You have 90% negative responses for every 10% positive. So it’s almost again, deliberately aiming to go get those negatives so you can increase the odds for some of the positives. Rob Wiblin: Yeah. Collect them in what way? Oh, I see. The idea is you should consider it a success every time you’re rejected for a job or a grant or something, because that makes it more motivating to do it. Varsha Venugopal: Or consider it a failure, but celebrate it anyway. So go out there and collect some of them. Maybe it doesn’t have to be huge failures, but just on a daily basis, you may be afraid to go talk to someone, but just do it because it’s another failure in your pot. Rob Wiblin: That’s super interesting. There’s some people on Twitter who I think they publish every time they get rejected for a paper from a journal or every time they get turned down for a job and things like that. It’s this funny sort of, what do you call it, temptation bundling. Or maybe it’s not quite temptation bundling, but I think it’s unpleasant to be rejected for a paper or for a job or a grant or whatever else. But then if you can make content out of it, where you get to share your misery with the public and people get to talk about how… Of course, everyone has been turned down for so many things before, it somehow makes it, I think, perhaps more enjoyable or motivating for folks because they get to share their frustration. Varsha Venugopal: Yeah. I think at some point it almost became a meme, right? There was some Princeton professor who posted something on all his failures and I think it took it to a different level. But I think fundamentally there is something there we could all aspire to do more of: fail. Rob Wiblin: Yeah. Yeah. I mean, I guess it’s the challenge with all sorts of discussion about how to be more successful. Or all lifestyle advice is that you only get this top 10% of things that people think… things that they’re willing to share, which is not… Most of our lives we prefer to keep it private rather than sharing on social media or putting it in a podcast, which can very much bias what kinds of things you hear. Varsha Venugopal: True. Rob Wiblin: Yeah. So obviously most people’s CVs are like all of the papers they did publish, all the jobs they did get. And if they went through and put into their CV every single time that they were rejected for anything, it would have made it incredibly long. But entertaining, and I think relatable. What does your day look like? Is there any way of summing up, broadly speaking, how much time you spend on different kinds of founder functions? Varsha Venugopal: So every Monday, Fiona and I try and have a conversation on, are we hitting our broad OKRs — objectives and key results — for the next three months? What is our plan for the week? And so a lot of I guess reflection, but also long-term thinking of, are we on track? On a more day-to-day basis, we also have Wednesdays, where we try and be good at not having any calls and doing more deep work and reflection and working on our papers. Varsha Venugopal: The rest of the time I would say is fairly chaotic, though I try and bring some semblance [of order] to it. Especially because a lot of my work is outward-facing and trying to reach funders or talk to peers and potential partners or maintain government relations, so there is an element of time not completely being in my control. I think the one good thing is because we work in India and the day starts early and I’m an early riser. So it’s also early on once the kids are out of home, I can get a lot of my work done in the first few hours. And then some of the calls and work that doesn’t require any kind of deep attention I can focus in the afternoon and in the evenings. Rob Wiblin: Yeah. We’ve had this interesting experience. Keiran, who’s listening to this at the moment, he used to be in a time zone five hours earlier than me, which meant that because he’s an early morning person and I’m a night person, we would basically be in sync with our working hours. And recently he’s been in the same time zone. This has created issues that at the start of the day, I’m not around, and at the end of the day, he’s not around. I guess it could have gone better, but I think it has actually been more challenging to be out of sync rather than in sync. Varsha Venugopal: That’s hilarious. Rob Wiblin: What sort of cyclical meeting structure do you have within the organization? Varsha Venugopal: So at the moment Fiona manages our team, which also makes sense because she’s moving to India. So she has one-on-one meetings with several staff colleagues. We are trying to get better at organizing them in a way where she and I talk and then we kind of roll down, and the meetings she has with others follows that. We also have a senior staff meeting on Monday afternoons — that’s with Alex (the head of operations), Fiona, and me — again, going through what is our reflection from last week? What are we aiming to get done this week? Especially the bits that need coordination. Varsha Venugopal: And also we have this gratitude section Fiona insists on, which is again about keeping us motivated. And I am really bad at that. I usually start ranting halfway through. So I think those are the big cycles. And then other than that, every Friday we either have this chai pakora where we meet, all of our team, India and here, and talk shop. And then every second week I know our field team meets by themselves without us because it just allows them to also kind of vent and talk shop without the senior team being on the call. Rob Wiblin: Yeah. We also have an appreciation section at the start of all of our team meetings. It makes them really fun actually. We have to do appreciation of things like within 80,000 Hours, and also appreciation for things in the broader world, which is important to keep in mind. Varsha Venugopal: Sure. Rob Wiblin: Do you have all-hands meetings? Varsha Venugopal: We have something called panchayat. This is based on the Indian local system of the village governance, which we try to organize every few months. And the idea there is that unlike chai pakora, which is where we all meet but it’s more talking shop, this is focused on ensuring everybody’s on board with a certain strategy. If we are rolling out an HR policy around insurance, for instance. The last time we did it in the middle of COVID, making sure everybody of our staff has health insurance, they all understand what that is, and what that could mean in terms of what they can avail of. It also allows them to question anything we are doing and offer feedback. Rob Wiblin: Yeah. So it’s more about getting people in sync than making decisions, because it would be too big a meeting to be making decisions. Do you have any kind of, I suppose perhaps a cycle where you set goals for like, hire a particular number of people, or send a particular number of messages, and then you evaluate that sometime later? Varsha Venugopal: Yeah. So we have these broader OKRs, which we set out for the year and then for every three months. And then within that… For instance, right now our key focus is on improving our cost effectiveness and setting up our M&E. And then within that we do have, for instance, what are the different iterations to improve our nomination process? So we have a whole dashboard of different ideas that everybody brainstorms on, and then we take up three and say… Right, so for instance, one of the OKRs is we need to increase the number of nominations by a certain time and we have a number there and we are holding ourselves accountable to it. Rob Wiblin: Are there any times that you’ve noticed that, say, the communication or the management structure is creating problems, and then you were able to reorganize things and fix things up systematically? Varsha Venugopal: Yes. All the time. And especially because of COVID, I am quite aware that there is a physical boundary, and especially when you’re working across different cultures and across different income levels. I think it’s a constant question for me. Are we all on the same page and any decisions we make, are they being rolled over? Varsha Venugopal: What we are doing right now is rolling out a staff survey to get a better sense of feedback from all our colleagues to make sure that… You know, what are the top areas we should be thinking more about? And we have some questions there on, “Do you feel sufficiently valued? Are you motivated when you wake up in the morning?” So at least to start giving us, I mean, they’re all anonymized, but to give us some ideas on areas we need to focus more on. In the past, health insurance was one where we hadn’t thought of it, and then we got that input and feedback from a couple of team members who were getting scared about COVID and then you’re like, “Right, we need to roll something out.” Varsha Venugopal: In terms of systems and processes, I think it’s not been our top priority intentionally on the HR side, because we want to get a product right first and then invest further. I think broadly our sense is, our team is satisfied and happy. There’s obviously a lot more we could do on that front in terms of staff motivation and HR systems and benchmarking and salaries. That’s all something we have deliberately, I think, deprioritized. Rob Wiblin: Yeah. Do it later on a bigger scale. Makes sense. Where do you guys stand on the delegation centralization thing? Are there areas that are just kind of handled down the line and you don’t worry too much about them? Varsha Venugopal: Yes. So, because we are focused so much on hiring the right people, the idea is these program managers, for instance, can then develop their own teams and take the work to the next level. There is the one-on-one calls they have with Fiona once a week to keep us updated and what the plans are. But we are very much working towards this decentralization model where they can then be responsible for their own budgets, especially because they have a clearer sense of the context on the ground, and the government asks. They have a clearer sense of how soon they can, for instance, hire the next few people and what the timelines around them should be. Varsha Venugopal: Because we are still a fairly early-stage startup, while we have these ambitions for decentralization, I think we still don’t have those systems. So for instance, a lot of things still come through us in terms of… for instance, even budgets, somebody can’t just go ahead and hire, it’s still a decision made at our level. But that is our aspiration. Rob Wiblin: All right. Well, yeah, you’ve exhausted all the questions that I had, so we could wrap up. Maybe just the last one is, do you have any book recommendations for someone in the audience who has enjoyed this interview and stuck with it through to the end? Varsha Venugopal: I think for anyone interested in global health and development, I would highly recommend Abhijit Banerjee and Esther Duflo’s book, Poor Economics. It’s 10 years old, but I still find it highly relevant and easy to understand. And at the time it came out, it overturned several myths around how people with less income make decisions, for instance. And I think it continues to be so highly relevant. So for anyone interested in evidence-based interventions, I think this book would be highly valuable. Rob Wiblin: Yeah, yeah. I can second that recommendation. I read that 10 years ago, and really loved it. I think we have a lot of social science-y and economics-y sort of people in the audience. So I think, if you’re one of them and you haven’t read this yet, go out and get it. Rob Wiblin: Well yeah, thanks so much for taking the time and thanks so much for being adventurous and being willing to go and start something new. Yeah, I really love seeing people go out into the world and actually try to make a difference on the ground. It’s very cool, and maybe something we haven’t covered enough on the show yet. So hopefully we’ll be able to possibly even speak to the Charity Entrepreneurship people, or perhaps some other organizations that go through it over the years and hear from their successes. Actually also the failures, maybe we should talk to some people who start charities and then decide to shut them down, let’s say. Varsha Venugopal: That’s fair. Yeah. Rob Wiblin: Cool. My guest today has been Varsha Venugopal. Thanks so much for coming on the 80,000 Hours podcast, Varsha. Varsha Venugopal: Thank you for having me. Rob Wiblin: If you’d like to support Varsha and Suvita you can of course donate at Suvita.org. And as of when I’m recording this they say they’ll soon be advertising for a Head of Programmes. If you could imagine switching job we’ve currently got 765 vacancies listed on our job board at 80000hours.org/jobs. You can filter them down according to where you live, what problem you want to help solve, and what stage of your career you’re at. That’s 80000hours.org/jobs. Alright, the 80,000 Hours Podcast is produced by Keiran Harris. Audio mastering by Ben Cordell. Full transcripts and an extensive collection of links to learn more are available on our site and put together by Katy Moore. Thanks for joining, talk to you again soon.