Latest / 80,000 Hours Podcast / #114 – Maha Rehman on working with governments to rapidly deliver masks to millions of people
Transcript
- Rob Wiblin: Hi listeners, this is the 80,000 Hours Podcast, where we have unusually in-depth conversations about the world’s most pressing problems, what you can do to solve them, and why green face masks are so last season. I’m Rob Wiblin, Head of Research at 80,000 Hours. We talk about a bunch of big, high-level ideas on this show, like whether this is the most important century, or whether tiny decisions will alter the identities of almost all future generations. But these conversations are only useful if they’ll eventually lead to a concrete project an actual person can go and do in the real world. So this is our second episode in a row which focuses on the day-to-day experiences of folks actually getting stuff done in the world. The first was with Varsha Venugopal, who’s trying to make sure every kid in India gets all of their basic vaccinations on time. And today’s is with Maha Rehman, who earlier this year led the rapid scale-up of the NORM mask-wearing program in Pakistan, and is now focusing on increasing vaccine uptake. Maha is in this category of really impressive people whose ideas and accomplishments haven’t gotten a huge amount of attention in the US, UK or Australia as yet, and I was delighted to talk with her about: The groundbreaking Bangladesh RCT which inspired her to take action What’s actually involved in taking a cool result and trying to apply it to millions of people as quickly as possible Here experiences and lessons learned from coordinating with governments And how to lower waste in government procurement If this isn’t your cup of tea and you’d rather hear about big ideas in economics, philosophy and global priorities research, don’t worry — we’ve got plenty of interviews on that stuff coming down the pipeline for you as well. Alright, without further ado, here’s Maha Rehman. Today I’m speaking with Maha Rehman, who is policy director at the Mahbub Ul Haq Research Centre at the Lahore University of Management Sciences. This year in June, she led the rapid scale-up of the mask-wearing promotion program known as NORM in Lahore, Pakistan, before the Delta variant hit the city. Months earlier, NORM had been successfully tested in Bangladesh, in the first large and really high-quality RCT on the effectiveness of mass promotion that has ever been conducted out there in the real world, in the field. Knowing that the results were going to be positive before they had actually become public, Maha and others coordinated to try to deliver NORM to as many people as possible, as soon as possible. Her background is in economics and data analytics, which she studied at the Lahore University of Management Sciences, and then Duke University. Since then, she has worked at the World Bank and with teams at the Harvard Kennedy School, and the Centre for Economic Research in Pakistan, before joining the faculty at the Lahore University of Management Sciences. Thanks so much for coming on the podcast, Maha. Maha Rehman: Thank you so much for having me. It’s a pleasure to be here. Rob Wiblin: All right. Today I hope we’ll get to talk about what’s involved in delivering a program to millions of people as quickly as possible, as well as some research that you were involved with about how to lower waste in government procurement. But first, as we usually ask, what are you working on at the moment and why do you think it’s really important? Maha Rehman: I really like that question because the project I’m working on right now is very close to my heart. I’m working with the Lahore Commissioner and his team to increase vaccine uptake in Lahore. Given my background in data analytics and the fact that we are still not out of the pandemic, it’s going to be quite a long journey still ahead of us in the developing countries. In order to do that, while we implemented and launched the mask-wearing project in June, now we are working on increasing vaccine uptake in Lahore. And for that again, we’ve adopted the NORM model that I’ll talk more about today. I’m also happy to share the details of how we’re increasing the vaccine uptake in Lahore, and what we are learning on the ground. Rob Wiblin: Nice. Yeah. Just before we started recording, you were telling me that Pakistan now has really enough vaccines available that anyone can go and get a vaccine now, if they want. It sounded like it was a lot better than some places in Europe even potentially, because there’s such a wide range of vaccines on offer. Maha Rehman: I think the strategy that NCOC — that’s the National Command and Operation Centre at the federal level — has followed has been quite effective. COVID in Pakistan has been more of an urban phenomenon. Even in terms of vaccine delivery, they focus more on the urban areas that were the hotspots in all four waves of COVID. With the focus on the urban areas, they have been able to ensure enough supply so that people are now being offered a choice of seven vaccines to choose from. There are no times in the day when the vaccine is not available. Vaccine is available at all centers that are there in the city. In addition to that, they’ve also increased the access. Talking about the intervention in Lahore, besides the 166 union councils that have vaccine available at different offices, you now have 206 temporary vaccination centers. So almost every neighborhood in Lahore has access to a vaccination center. Maha Rehman: In terms of the availability of vaccine, there is really no excuse that, “Oh, the vaccine was not available, or I did not get the vaccine of my choice, or I wanted Pfizer and I was offered Sinopharm and that’s why I did not get vaccinated.” All that has been effectively dealt with. Maha Rehman: Then comes the second part of it. That’s jostling people out of their complacency and getting them to get to a vaccination center, and get their jab. For that, we are now launching, working with the Commissioner of Lahore, a very rigorous social mobilization drive. We work with universities across Lahore, 25 universities, that we have now put together a team of 1,000 volunteers who will be trained to deliver a customized speech, go house to house, ask who’s vaccinated, who’s not vaccinated. Maha Rehman: For the people who are not vaccinated, why are they not vaccinated? Is it because of access? They did not find the time? Or is it just because of laziness, or just because of the fact that in a lot of cases, especially for women, it’s mobility issues. That we didn’t have anyone to go to a vaccination center with, or we didn’t have a car, or we are concerned that this affects fertility, or there’s maybe some other concerns. Maha Rehman: So they hear them out and then deliver a customized pitch based on the information that has been offered to them at that particular house. Then the data is very systemically recorded. That data will then be synced with the data from all vaccination centers to ensure completion in every neighborhood. Lahore is going by a very data-driven and a systemic vaccination drive to ensure that we fully vaccinate one union council after another. Rob Wiblin: Nice, yes. Super impressive. Yeah, we’ll come back to the NORM mask-wearing program and the vaccine campaign in a minute. But before we get to that, I wanted to talk about this randomized controlled trial that originally inspired you and your team to take action, because the results were announced a week or two ago, and they’ve been making a pretty big splash because they are cool and original. On top of that, I was amazed to find out that it was funded using grants made by our friends, friends of the show over at GiveWell and the Effective Altruism Funds. It turns out it’s a bit of a small world. Rob Wiblin: Yeah. You weren’t directly involved in making this RCT in Bangladesh happen last year, but some of your colleagues were, and I imagine you’re somewhat familiar with the results. You should be able to go over them all right. And the background is that, I think until this study in Bangladesh that was coordinated by J-PAL, there had never been a large field trial of how much mask promotion and mask wearing would actually reduce the spread of a respiratory disease in the broader community, out in the real world. Which is kind of hard to believe, but as will become apparent, it’s actually incredibly laborious and challenging to answer that question in a rigorous way and to do that research properly. Rob Wiblin: To set the scene, do you remember what we did and didn’t know, or what we didn’t believe about mask wearing before this experiment started? Maha Rehman: Yeah. Before I go into the details, let me credit the original project investigators. On this staff is Dr. Mushfiq Mobarak, [Dr.] Jason [Abaluck],[Dr.] Laura [Kwong], [Dr.] Ashley Styczynski, and [Dr.] Steve Luby. So Mushfiq and Jason are at Yale. Laura is now at Berkeley, and Ashley and Steve are at Stanford. They were the ones who designed and implemented this together with other researchers at [IPA] (https://www.poverty-action.org/), [IPA Bangladesh] (https://www.poverty-action.org/country/bangladesh), and several other partners. Maha Rehman: Until the study had been done, there was no real evidence of how masks would work in a community setting. There was a lot of evidence coming out of laboratory experiments, out of lab trials, but how does it actually impact or reduce, or slow down the COVID transmission, or how do you… So, the simple question, “Do masks work in a community setting?” — the first question. And the second question it was trying to address was, “How do you then get people to wear more masks in a community setting?” Because even with a mask mandate, you don’t see a lot of widespread adoption of masks. What is it that you can do to get people to wear masks? So, this very powerful RCT was based around these two simple, yet very important questions. And I think, the evidence that has now been put forth is very, very important. Rob Wiblin: Yeah. What was the actual intervention that they were delivering? Maha Rehman: Let me just talk a bit about the NORM model here. A lot of different interventions were tried in the field, but then the original research team found out that there were four things that work the best in a community setting to encourage people to wear masks. They’re now using the acronym of NORM: N stands for no-cost distribution, O stands for offering information, R stands for reinforcing the message and the information in the field, and M stands for modeling. You ensure enough supply of masks in a community setting, and you offer information as to why is it important to wear masks. But then, at the same time, you also reinforce the message periodically in markets, in mosques, and the reinforcement is done through role models, but also through promoters in the community setting. Maha Rehman: These four interventions together form a very powerful model that in Bangladesh, if you talk about the results, it increased mask wearing up to 30%. This was done across 600 villages, about 350,000 people. I think this is very strong evidence in terms of how can you initiate behavioral change in a community, how can you get people to start wearing more masks. Rob Wiblin: Yeah. Just before we get to the results, why did people decide to test this specific combination? Is this a very standard thing, or was it more that there was theory that these four elements in combination should work particularly well? Maha Rehman: A lot of other different things were also tested. They also tested incentives. So, different interventions were tested and then measured. And in the end there were several interventions that did not work, but the evidence showed that these four work in a very convincing way to increase mask wearing in a community. Other things that did not work was offering people incentives, signaling, getting the police to also be a part of the reinforcement drive. But those interventions did not exhibit very strong evidence in terms of increasing mask uptake. This was really a data-driven exercise where you were trying to get to the best mix that would then initiate, or stir, this change in the community. Rob Wiblin: Yeah. As you said, there’s two parts of the experiment, or two different things that were attempted to be tested. One is, “Does this intervention cause people to wear masks more often?” And there they found that mask wearing went from about 10% of the population to 40% of the population in public off the back of this. The second thing was, “Does that actually reduce the spread of COVID?” Yeah. What was the result there? Maha Rehman: In the 30% increase in the mask wearing, you were able to reduce the infections by 10%. This 10% I think is a huge number because it applies to this 30% increase in mask wearing. From there on, you can also extrapolate and say that if you were to ensure 100% mask wearing, or if everyone in the community was wearing masks, how much more would you see a decline in the COVID infections, in symptomatic COVID. So this was particular to symptomatic COVID in the community. The effect was pronounced for people 60 years and older. It was 35% for that group. Between the cloth mask and the surgical mask, it was more pronounced for surgical masks. I think these are very strong findings. Maha Rehman: Let me just put that in a context. When COVID came to these developing countries in March 2020, we did not have conclusive evidence on what would work and what would not work. And to now start having evidence like this RCT in a community setting that allows us to understand what will work, I think is a big step forward. COVID really brought us down on our knees in a very significant way in last March. There was a lot of chaos at the policy level. There was a lot of firefighting going on. Everyone was trying different measures as to what would work, what would not work. So given where we were then and where we are now in terms of such conclusive evidence being put forward, I think that’s a very, very important result that needs to be disseminated far and wide. Rob Wiblin: Yeah. Okay. Mask wearing went from 10% of the population to about 40% off the back of these. Then that reduced the number of COVID cases that they detected in the population over that time period by about 10%. That suggests, I guess, that if you manage to go from no mask wearing to full mask wearing, it would reduce the spread by basically a third, or at least it would reduce the number of cases in this kind of takeoff scenario by about a third. How does that compare maybe to what people were expecting? Because some people were arguing masks weren’t going to work at all. Other people have prioritized them incredibly hard. I suppose this is somewhere in the middle, that it’s quite effective but of course it doesn’t end the pandemic, which maybe is what we should have suspected. Maha Rehman: In order to actually stem the transmission of COVID in a community, you need to bring R naught to a number that is less than one. So if R naught is one, that means one person is at least affecting another one person. So you would need a complementary set of interventions that’s implemented in any community to fully stem the transmission of COVID. Maha Rehman: I think this result is just the beginning. There’s several other studies that are being planned; there are followup studies to this in Bangladesh and in several other cities. In Lahore, for example, we adapted this to an urban setting. There’s a followup study plan in Bangladesh, BRAC is scaling this up to 80 million people. So there’s several other aspects that are now also being investigated in greater detail. Rob Wiblin: Yeah. There was a couple of other really interesting nuggets in the results. One thing people might remember last year when we were discussing whether people should wear masks or not, at least in the UK, there was this big concern that if people started wearing masks, then it would give them a false sense of security, people would say. And so it would cause them to not engage in social distancing as much, so they would stand closer to one another, and that might undo all of the gains that you might get from the masks. And this experiment tried to measure that. Do you remember the results from that one? Maha Rehman: In Bangladesh, it was observed that mask wearing also led to an increase in social distancing. The fact that people would not be as careful anymore is a conclusion that was being put forward by a lot of international bodies, but they also quickly updated their stance on this. And this experiment also then gives us evidence that that’s not really the case. This message also became a key part of the messaging in Lahore, where anyone who is wearing a mask is also more likely to comply with other SOPs. Because then you’re getting into a state of mind where you’re more likely to be the first ones to get vaccinated. So we also have a lot of evidence coming up in which people who are more likely to wear masks are also the ones more likely to get vaccinated first. They’re also more likely to be the ones who are more mindful of the standard operating procedures, vis-a-vis COVID. I think this experiment just contributes to that. The international bodies that were advocating differently have also now updated their stance vis-a-vis social distancing and masks, and how does it operate in a community setting. Rob Wiblin: Yeah, I think this experience with masks has caused this kind of reasoning, which at least in economics is called risk compensation. So there’s often this concern that if you make something safer in one dimension, then people will offset it through their behavior in another dimension. For example, if you design a car so that it’s safer, then perhaps people will drive more recklessly because they feel more secure knowing that they have airbags, or whatever. I think this was always a little bit questionable, but I think this has made people realize that this might be an idea that’s a little bit too clever by half. In fact, most of the time, just doing the obvious thing that makes things safer is actually almost always good on net. Rob Wiblin: It’s interesting that it was always possible to make a roughly equally compelling argument in the other direction, which would be that if people are wearing masks, then that shows that they think that COVID is serious, and so they’re providing social proof to the people around them that they should be engaging in social distancing because there’s a pandemic happening. It seems about as sensible as the risk compensation argument that they’re going to stop engaging in social distancing because they’re wearing masks. I guess, ultimately in this experiment, it seemed like that effect or some similar effect was the dominant one. Maha Rehman: Yeah, absolutely. That’s correct. Rob Wiblin: Can we tell what were the most important parts of the program? Because of course there’s four different aspects to it. It would be lovely if we could cut three of them and just deliver one of them. That would make your job a whole lot easier. Maha Rehman: No, I think all four of them reinforce each other and all four of them are very critical when it comes to delivering this intervention. And so even when we were adapting this model for Lahore, all four of them were important. You need to ensure the mask supply. Information has to be offered, because otherwise people have different concerns as to why they’re not wearing a mask. Some are complacent, some have real concerns as to why is this important, so offering information is key. Once you have ensured access and you’ve offered information, then reinforcing that message so that it becomes a habit. So you would need to reinforce once, twice, thrice, and the fourth time you see the person, the person is more likely to be wearing a mask. Then also sharing this information through people they look up to. Maha Rehman: I think that’s also very important. The role models are also very different in a rural setting, as opposed to an urban setting. How people get their information in urban settings is also very different. And so just in adapting this, I think all four elements are very important. Where you are not reinforcing, you don’t see as much of an increase. So, the R in NORM is also critical. So is M. How you do it in different cities, in different settings, is obviously going to vary. We are learning a lot as we are now scaling this to different cities and through different partners in South Asia. Rob Wiblin: I guess some other details of the experiment… You mentioned that they found that surgical masks were a whole lot better, which I guess won’t shock anyone. But the design there was in half of the villages they distributed surgical masks, and then half of the villages they distributed cloth masks. They found that, I think, it was like plus 50% effectiveness to be using the surgical masks versus the cloth masks, which is nice data to have to try to persuade people to up their mask quality. Maha Rehman: Yeah. In a lot of rural settings in some South Asian countries, people tend to wear more cloth masks because that’s also the mask that is being produced more locally. A lot of women are sitting in their homes stitching cloth masks, the cloth is readily available. And so a big shift in rural Bangladesh is now trying to convince people to move to a mask that’s going to be more effective. Maha Rehman: You don’t see that as much of a trend in Pakistan. In Pakistan, you see people who were wearing more of a surgical mask wherever they are wearing a mask. In rural communities, not so much. But the way rural Bangladesh and even rural India are different, that a lot of people’s livelihoods depend on this cloth mask making it to the market and then being sold. And so yes, the experiment definitely showed a greater increase [in effectiveness] where people were wearing surgical masks. And we cannot say for sure of the impact on cloth masks, but the next trial will also investigate the type of masks further. That is now being planned for Bangladesh by the original research team. Rob Wiblin: Yeah. Another nugget from the results table was that, like you mentioned, there were various other things that they tested including incentives for, say, officials in a village to try to get people to wear masks. They found out that didn’t work. But actually the thing that most impacted mask wearing was the color of them. People are just very responsive to the appearance of things that they’re putting all over their face. I think people preferred purple to green or something like that. The color was a significant result. Maha Rehman: We haven’t tested this in Lahore, but that’s also something that’s more likely to increase uptake, if the masks conformed with certain fashion trends, or as you’re talking about, in terms of a color. So it’s also very interesting to see how people respond differently to something that’s considered more trendy or something that’s blending in well with their overall appearance. That’s definitely a very interesting result that’s coming out of Bangladesh. I think in the next trial, we would also try and look into that in Lahore. Rob Wiblin: Yeah. It makes a ton of sense, given what I know about human nature. Yeah, we’ve talked about the results a bit, but we should explain to people what the experimental design was. Because I think they might be impressed by just how many hours must’ve gone into getting these numbers. Broadly speaking, what method did they use? Maha Rehman: It was a randomized controlled trial that involved 600 villages and 350,000 adults. It was based in completely a rural setting in Bangladesh. You took 600 villages, for the sake of simplicity divided them into treatment and control. The treatment also had sub-arms, as you’ve already rightly pointed out. Then an RCT allows you to rigorously evaluate the difference in areas where you were making an extra push to getting people to wear masks, and where that extra push was not being administered in the control areas. For the sake of simplicity, I think this is one of the more rigorous ways to evaluate impact and to evaluate what works. This was a cluster randomized trial where each of the villages… the village itself was a cluster. That’s a specific type of a randomized controlled trial that we’re implementing on the ground and in the field. Rob Wiblin: Yeah. So they’re not only delivering this program to 350,000 people across 600 villages, where they are delivering masks, and running advertising campaigns to encourage people to wear the masks, and having like local figures who people respect wearing masks in order to try to encourage people to wear masks. But then they had to go back to all 600-something villages and have people stand, you know, look at the mosque, look at the street, look at the market, and see what fraction of the population that I can see is wearing a mask right now. Which is how they got to this thing of, in places where we didn’t do anything, there was 10% of people wearing masks, and the places that we did, it was 40%. Rob Wiblin: But just the sheer amount of legwork that’s going on to hire all of these people to go to all these villages. I think they managed to do all of the observations within about a week because they wanted it to be comparable across them. Just an enormous amount of data collection that’s going into this behind the scenes. Then they also had to survey everyone to find out whether they had COVID. Maha Rehman: Yeah. I think the data collection, and this is a very interesting component because the data was very systemically collected at the baseline, during the intervention, and also endline. And a very important result of this intervention was that the effect was also sustained 10 weeks after the intervention ended. So they kept collecting the data for another 10 weeks after the intervention ended. Maha Rehman: The enumerators are very carefully trained into observing people who are correctly wearing a mask, incorrectly wearing a mask, or not wearing a mask at all. And sometimes, women also cover their face using a face cloth, which is not considered proper mask wearing. And so this was very carefully recorded. The actual observation times during the first week, the second week, and so on, and then of course the people who were collecting data did not know whether it was a treatment area or control area. And so very high quality standards were maintained when the data enumerators were observing people in the mosques and the markets, in both the treatment and the control areas. Rob Wiblin: So, they were both checking the symptoms and then they were doing blood tests in order to check whether… I suppose there’s a risk, of course, that it might be stopping non-COVID respiratory diseases. It’s possible that people weren’t getting the flu as much, and that would show up in the symptom data. So then they also had to do blood assays in order to confirm that it was actually able to stop COVID as well, rather than just all diseases. Although I guess if it stops other diseases, that’s pretty good news as well. Okay. So let’s push on to talking about what it is like trying to take a result like this, and then applying it to a huge city on a tight deadline. I think you were saying Lahore has a population of about 13 million. I guess you got a heads up that the results of this RCT were going to be positive about masks before other people knew, and it was time to go out and actually make this happen in Pakistan. Rob Wiblin: I’m kind of someone who struggles to just stay on top of all of the tiny logistical things in life, and just constantly feeling stressed out that I’m kind of falling behind on my life admin. So I’m just amazed that this kind of program is even possible at all, that anyone is able to complete so many tasks so quickly in order to make this kind of thing happen. And this kind of discussion of practical deployment issues is something that perhaps we haven’t talked enough about on the show relative to higher theory. But first, when did you actually start working on delivering this program? Maha Rehman: So Dr. Mushfiq Mobarak reached out to me with the results in March. They had received some initial promising results on how mask wearing can be increased in a community, and were interested in sharing that with various stakeholders in various South Asian countries. And so the fact that I’d already been doing a lot of data work in COVID over the past one year allowed me to quickly reach out to the relevant stakeholders, get them interested and excited. Maha Rehman: And the fact that NORM itself was very actionable. There were four very actionable components, easily understood by a policymaker who could then go on and explain that to his team and implement that and take that up. Because one thing that’s very critical whenever you’re looking at implementing a research experiment with policymakers is… I mean, there’s a very simple ask that they have: “Don’t send us papers. Tell us what to do.” And so this was exactly telling them what to do in a very simple layman language. And the fact that the NORM team actually listens to what the policymakers had to say, takes their feedback and then offers sort of strategy planning and consulting advice as to how can this be planned, implemented, and then evaluated. So because they were given planning support at every component, it became very easy for them to take it up. Rob Wiblin: Yeah. Casting your mind back to the start of this project… I guess in March, and then you actually started delivering it in June. What was kind of the context you were operating in? Were case rates going up, was there kind of a race-against-time element to this? Maha Rehman: So in March and April, Lahore was going through a very deadly third wave. And so that is why the uptake of any intervention that was likely to stem the spread of COVID was very… policymakers were very willing to listen to it because they had seen the worst of COVID. They had been in the midst of it, lives were being lost. It was not only lives being lost; it was lives and livelihoods both lost. Because once you lock down, even if you’re smartly locking down certain areas, there will be livelihoods lost in the process. And so at that point, the policymakers were really looking for workable solutions and that is why they really jumped onto this solution very, very quickly. And because the third wave was the deadliest of the waves so far, another aspect I think, which the policymakers really value, is the planning support that came with it. Maha Rehman: And so together with the original research team — Dr. Mushfiq Mobarak, Dr. Jason, and others — myself and my team at LUMS provided very close planning support to the Commissioner Lahore Office. We collected baseline data for them that allowed us to map the mask-wearing rate in every part of Lahore, in both markets and mosques. And then based on that data, worked with them to adapt a rural model to an urban setting. Maha Rehman: And so there were a lot of ways in which this urban model is different, because an urban setting is different: mobility is high, Lahore is a very dynamic city, an urban setting is less cohesive. I mean, of course Lahore is a big city, so the intervention then needs to be designed keeping in mind the fact that you should be able to reach as many households as you can. And then information is consumed and processed in a very different way. You have people listening more to the information coming in from the cable network, social media, role models are different. And the reasons for which people are not wearing a mask are also different. Maha Rehman: And so keeping all that in mind, we worked very closely with the Lahore Commissioner’s Office to plan this drive. And we eventually then launched it towards the end of June with the health minister inaugurating the campaign. And then there were several elements that were different. Rob Wiblin: Yeah. What do you have to do differently in an urban area? Maha Rehman: So in a rural setting, you could hire a team of promoters that could go from house to house, hand out a mask, and you know, give a pitch as to why was this important. A similar intervention was very expensive in an urban setting like Lahore and so the first adaptation was that the masks were then sent out through the Lahore postal system. Maha Rehman: So one of the key partners to the campaign was Lahore Post. And they offered to send and post masks out to every household free of cost. And so then there was a challenge of where do you get everyone’s addresses from? What’s a credible data set? And so we got the addresses from Lahore Electric Supply Company as billing data because they have verified addresses. They deliver that bill every month. Maha Rehman: And then of course there’s a major cost element to it involved, that where do the masks come from? And the masks were then collected by the Lahore Administration from various corporates and philanthropists. So not a single mask was bought. They were all donated by different philanthropists and corporates in Lahore. I think that was also something that’s very distinctive about the adaptation in Lahore. Maha Rehman: Once that was set up, then of course you also wanted to offer information in person or reinforce it in person. And so reinforcement teams were stationed in markets and in mosques, which were a part of Lahore’s MCL [Metropolitan Corporation Lahore] Division. And in a rural setting, a person with a loudspeaker goes around a community dispensing information. In Lahore, you had trucks with mobile announcements going around the city. Announcements were also made through mosques. And one key element that is also distinctive of the Lahore intervention is flag marches. So the Lahore chief of police and the commissioner also led flag marches around the city. Rob Wiblin: What’s a flag march? Maha Rehman: So a flag march is when the Lahore police chief and the Lahore commissioner, the key administrators of a city, take their workforce with them. And they go around different markets in Lahore, dispensing information, talking to people, and ensuring compliance to an intervention. This was first implemented earlier, and then this aspect of the intervention was refined for this model. Rob Wiblin: Is this a common thing in Pakistan? I’ve never heard of this before, but maybe it’s a good way to get a lot of media? Maha Rehman: So flag marches, I think that’s very interesting because this is usually done in a war setting when there is more of a security threat to the city, that you would go around the city and ensure everyone’s safe. But that was adapted for COVID because you again need to save lives, and to save lives, you need to go into the markets and in areas that are more crowded and ensure that people are wearing masks. Maha Rehman: So this was more of a strategy that was adapted for the COVID intervention, and this campaign in Lahore that now goes by the hashtag #LahoreWearsMasks. And so that was the hashtag that the campaign was associated with. There was a lot of marketing collateral that was put up around the city, which was again sponsored. A lot of marketing collateral was also then aired online through different cable networks, as well as put on social media, because that’s the area where a lot of people take their information from. Rob Wiblin: Okay. So at the start of this, I guess you’re wanting to deliver a mask to every house in this city of 30 million people or so. And you want to have people monitoring, or you want to find some way to monitor mask wearing in different locations across this huge city. You also want to have an informational campaign that reaches as many people as possible. And I guess also have role models available across this entire place in mosques and like other public events where people talk about how it’s valuable to wear masks, and I guess, if they’re a credible person, them wearing a mask might move the needle and convince people to do it. Rob Wiblin: So as you sit down at your desk in April or something trying to make this happen, what does your day actually look like in order to try to get this to happen as soon as possible? Maha Rehman: I think there were two key challenges. So the Commissioner’s Office was then supported by a lot of partners. And so whenever you’re working with a large coalition like that, coordination failures are bound to happen. So one key challenge was while you want to plan everything to the detail, you also want to ensure smooth communication amongst the coalition. So we had regular meetings that were convened by the Commissioner’s Office where all stakeholders would come on board, discuss. Everyone was kept updated on every aspect of the campaign in order to ensure that the information completely flows through the coalition. Maha Rehman: The second thing I think that was very important is because we wanted to design this campaign well and base it on data. So getting the required data and collating that was a huge challenge, mapping the markets and mosques all over Lahore. And for that, we first relied on the administrative data that was available from the market committees, from the religious department, and used that to then map all these markets and mosques for baseline data collection. That baseline data collection then allowed us to choose the treatment and the control areas. Maha Rehman: And so I think coordination, getting the required data, and keeping all partners’ incentives aligned is very key because we didn’t want to spend a lot of resources on it. And so different partners had to come on board. There were a lot of aspects of this campaign that were sponsored, and we did not spend on marketing at all because we got the media interested and excited in what the key message of the campaign was. And so instead of putting ads on televisions, we got a lot of organic coverage for the campaign. So the message flows through the industry in a very seamless manner. Rob Wiblin: Yeah. So your particular background and strength is in data analytics. What kind of data were you trying to collect and how would that potentially change what you were prioritizing in terms of regions or different interventions that you needed to go harder on? Maha Rehman: Before we started the intervention, it was key to get credible addresses that we could then use to map them onto the administrative units of Lahore. So there was a lot of pre-planning that was involved that required that one data set should speak to another. So the LESCO [Lahore Electric Supply Company] addresses should be mapped onto the administrative units. And then we are able to also map the markets and the mosques. So a lot of existing administrative data had to be cleaned and mapped onto one another. I think that was a very critical baseline exercise. Maha Rehman: Based on that we then collected baseline data in markets and mosques in which we again had people observing at intervals of 10 minutes in a market and in a mosque, and then systemically recording that on the mobile app. So we use SurveyCTO to systemically record that: how people were wearing masks when they arrived in the market? In the next 10 minutes, how many more people were wearing a mask? How many of them were men? How many were women? We incidentally found that women are more likely to wear masks than men in Lahore — Rob Wiblin: I’m shocked that women would have better hygiene than men. Maha Rehman: — and what was roughly the age group. And so this data was very systemically collected. And then this data was also collected during the intervention, where then the data was weekly shared back to the Commissioner’s Office, that these are the areas in which the intervention is working well, it’s not working so well in these areas. So maybe we need an additional focus on these areas in terms of more workforce, in terms of tweaking the message… just figuring out why it is not working. Because in some areas we realized that the imams of the mosques were anti-government and so did not really want to comply with the message coming from a government office. It was easier to get the message across in high- to middle-income areas and not so easy in areas where there was a higher resistance to adopting a new practice. And so there was a lot that we learned when we then systemically communicated this data back to the government. Maha Rehman: And there was this quantitative data, but then we were also collecting qualitative data from various stakeholders, from the promoters, the imams. So there was a team of interns at LUMS working with me who would very regularly communicate and call the promoters who were on the ground, who would call and talk to different people in the Commissioner’s Office. And then who would also communicate directly to the imams to figure out how can they help to better make this intervention more effective. Rob Wiblin: Yeah, it seems like all of this is just going to require a huge amount of personpower. Like the number of people you’d have to have going to all of these different locations in order to enter data on their phone to tell how many people were wearing masks. The amount of delivery that we’re talking about here… I’m guessing in a city the size of Lahore, there must be a thousand mosques and a thousand marketplaces that all have to be contacted in order to try to persuade them to do this thing. How do you get the people you trust to do this kind of work properly really quickly? Maha Rehman: I think that’s a key question in ensuring the success of any implementation. So I think the people on ground… in this case, we experimented with different groups, but the one that we ultimately used in the intervention was the government’s own people stationed in markets and mosques at different times of the day. And in order to do that well and quickly, the government also established a monitoring cell within the Commissioner’s Office. So apart from us monitoring and giving them regular feedback, they were also collecting information on their own, which was then being collected and conveyed to the Commissioner’s Office over WhatsApp. We were doing it through a mobile app; they were doing it in a slightly different manner. Maha Rehman: And then of course, whenever you are doing it at scale, then you also need to monitor the monitors. That’s very critical. And so the app that we used to collect data has built-in quality measures. It randomly records audios, their time checks as to when was the data collected, how long did it take? And so there are data quality checks that are built into the information collection process. And then my team at LUMS was also visiting these mosques unannounced, and they were also visiting the markets unannounced. And we were also able to quickly tackle the intervention implementation failures very quickly through a multi-arm monitoring system. Rob Wiblin: So it sounds like you got a lot of mileage in making this happen relatively quickly by taking advantage of the resources that the government already had, I guess the city council in Lahore already had. I guess governments in general aren’t known for being super innovative, like suddenly wanting to take on a massive, new, big project that they haven’t done before. Why do you think that was possible in this case? Was it kind of the emergency situation that made something possible that otherwise might have taken a long time? Maha Rehman: Yeah. It was an emergency situation. Then a model was pitched to them by a coalition that itself has a lot of credibility in Lahore, so a coalition comprising Yale, Stanford Medicine, LUMS, IPA, and a few other partners. When something like that is pitched to them, something that’s already tested, that has a lot of credibility. And the fact that we did not just go to them with an ask for… I mean, so we were not asking anything of the government. In fact, we were allowing them to save resources by helping them do this effectively. Maha Rehman: And so I think the fact that you need somebody who’s a little more inclined to listen to a data-driven intervention. So you need to identify champions within the bureaucracy. Not everyone would’ve pulled this off. And so identify the right champions who are able to pull this off, who have both the political capital as well as the required resources to pull this off. And then I think there was a lot of credibility of the coalition that then took the message to them. Maha Rehman: We worked very closely, we kept the incentives aligned and I think there was a lot of listening and planning and proactively following up. One thing that’s key when working with the government is that you’re proactively following up in order to ensure that it doesn’t get dropped at any stage before it is actually launched. Rob Wiblin: Yeah. So was it the commissioner in Lahore who was really excited, and then went and spoke to the academics involved in this in order to get advice on how to implement it? Or was it academics trying to persuade the government in Lahore to take this on? Maha Rehman: No. So, Mushfiq, [Heidi, Neela] and myself, we approached him and we pitched it to him and he got excited by the fact that there was evidence that was supporting this. And we were not just going to him with an intervention that we wanted to do and which we did not know would work well or not work well. So the evidence from Bangladesh really helped in kicking this off very, very quickly. And the fact that we were able to give them planning and strategy support was also very crucial for them. And they really value that. Maha Rehman: And so while the intervention was imported from Dhaka, some of the urban innovations are now being exported back to Dhaka. And so it’s really all about South Asian cities learning off each other. And so Lahore is just one of the scale up. Ahmedabad in India, SEWA also quickly adapted this. There was similar intervention in Nepal. And so I think the model itself is very, very actionable. And then the team takes ownership. So the NORM mask team takes ownership of the fact that if there is a scale up, they don’t just communicate the result. They also see it through. And so whenever you see you have research results, the key thing post-results is that there’s somebody to take ownership of the fact that the intervention is actually scaled and implemented well. Rob Wiblin: Okay. So it was really helpful to have the experimental results to convince them that it was actually worth the effort of doing this. And it sounds like basically the government was bringing just the raw number of people and perhaps the operational capacity. And then was it kind of you and other people with a more academic background, more experimental background, who were coordinating or overseeing how the government would implement this intervention? Because you were saying it was also very important that the intervention was clearly defined and that there were very specific actions that could be taken. It wasn’t a vague ask. Maha Rehman: Yeah. So I was working very closely with the assistant commissioner and the commissioner in Lahore. We had very clear deliverables that these are the things we need to be able to deliver by the end of this eight-week intervention. So just the simple fact that we were sharing the data results back with the assistant commissioner and the commissioner kept them really motivated because it was very exciting to see the numbers register an increase every week. And to actually see that all the effort that they were putting in was actually saving lives. Maha Rehman: And when the stage two results were finalized in Bangladesh, again, Mushfiq, Jason, Laura, and others shared the results back with the commissioner of Lahore. I think that was also very important in building trust and in keeping the commissioner and all of the partners updated into what were the stage two results. Because a lot of partners that we had spoken to earlier were really keen to know how many lives would be saved, because then they need to justify the amount of time and the resources to their own higher-ups. So listening to the policymakers, then communicating back, iterating on the model based on the feedback. I think these things may sound simple, but they have a lot of impact when it comes to doing things right with the government. Rob Wiblin: Yeah. Did you have to make a lot of use of the commissioner’s support in order to persuade other people within the city government to go along with it? Because I’m imagining there’s a whole lot of other people whose enthusiastic participation the whole thing really relies on, and you maybe don’t have time to go and convince every single one of them that this is incredibly valuable. Maha Rehman: I think you do need the commissioner’s push every now and then, but then you also need buy-in at those levels itself. Because unless and until everyone along the implementation chain is convinced or wants to do it, interventions don’t take off. So just buy-in at the top is not enough. You need everyone. The assistant commissioner, his manager, people who were actually supervising the reinforcers in the field, the reinforcers themselves. You need to talk to them and see what their incentives are and how can those incentives be best aligned. And you also need multiple champions within the same office and the same department. So I think it’s very critical to getting to know the partner and his entire team well, and working with all of them, and listening, giving them respect, knowing what their incentives are and actually showing the results back to them as well, and giving them credit for all the hard work that has gone in. Rob Wiblin: Yeah, so I guess you’re going to have a whole lot of different people who are part of this broader project. So you probably have some people organizing flag marches, some people organizing the leaders to have the demonstration effect, some people talking to mosques, some people doing the data analytics, someone receiving these like enormous number of masks, someone working with the delivery company in order to get them delivered. You were saying, it sounded like you had a weekly meeting or was it a daily meeting in order to try to keep people coordinated? Maha Rehman: It was a weekly meeting, but then closer to the intervention, it became twice a week. During the intervention we also met almost every second day, because first you had a launch event, everyone was there at the launch event. Then there were also other events that were held during the intervention to generate excitement about it in the city. There was a cycling event. Now cycling is not something that’s directly related to mask wearing, but that was an event that got people excited about what the message of the campaign was, got the word out fairly quickly. There was a lot of media, there was a lot of curiosity as to what is this all about. Maha Rehman: And so these events also periodically brought the team together where you were able to give back credit, recognize them, coordinate on the next steps. And a lot of the meetings were online, but there were also a fair bit of meetings in person. And so it’s important to do that in order to keep the communication and the implementation moving along. Rob Wiblin: Yeah. I’m very interested in this sort of nuts-and-bolts level of like, really specifics of how you organize things. Because I think people often don’t get that much of a picture of this. So you’re having these weekly meetings or biweekly meetings… How many people come along to that and how long are they? How do you avoid them dragging on and absorbing tons of time where people are sharing information that perhaps other people don’t don’t specifically need? Maha Rehman: So the usual check-ins were kept online. They were half an hour long, they were chaired by either the commissioner, the additional commissioner, or the assistant commissioner. And so most of the partners comply very quickly to a call coming in from the Commissioner’s Office. This commissioner is a very senior bureaucrat in the district. And so a lot of people would respond to it very quickly. And the fact that these meetings were kept very to the point, it was like a half an hour check-in that anyone could join in from wherever, but then there were very clear next steps. And then the next meeting you would follow up on those next steps. Maha Rehman: And the second thing was that people who were part of this campaign really wanted to see it through. So it’s the determination of ensuring that something kicks off I think is also very important. And the commissioner at the onset made it very clear that he wants to see this intervention in the field and he wants to see it done well. And so if I felt that at some point things were not going well, I would go back and talk to him. But otherwise there were other people in the ecosystem who helped move things forward. Rob Wiblin: Did you have to, I guess, massively delegate decisions in order to avoid someone like the commissioner or you becoming a bottleneck on people’s ability to actually move forward? Maha Rehman: It was a huge team and there were different levels at which I had to communicate, but the key decisions I kept to myself. And I ensured that I don’t become a bottleneck, but once the key decisions were taken, then everyone could chip into the implementation. Everyone could chip in and give suggestions onto how to make this more effective. If there were other partners and they wanted to add another element to it, we were open to that and we did not micromanage that a lot. Maha Rehman: But the key decisions that these are the four elements — they have to be there. You can add other elements to it. But as long as the key essence of the campaign is there, then the other sub-elements of the campaign can be delegated. So the cycling event was totally taken care of by [the Commissioner’s Office and] another partner organization. And similarly, there were other elements that were, so the media campaign was totally delegated. But the messaging, we worked together on the messaging, ensured we are communicating the right thing across to the public, but the implementation was delegated across the government machinery. Rob Wiblin: Did you do anything specific in order to avoid the risk that you might become a bottleneck or that, I guess, you were maybe so central that if you disappeared because you got COVID then the whole project might really struggle? Maha Rehman: Yeah. So I think my own LUMS team was key. So I worked with a group of very trusted RAs, and so I did not have to hire anew for all aspects of this campaign. There were some RAs who were already working with me and I was quickly able to onboard them onto this project. I think that is key to quickly bringing a project to life and ensuring that it’s implemented well, is that you have a trusted team who’s already trained that you work with. And then they are able to take on a lot of tasks that you would otherwise be doing. And because I was working on various COVID data sets and interventions even before the mask-wearing campaign, ensured that I had a team of sorts in the running and I was able to quickly onboard them, bring them on board and delegate some key tasks to them. But then also on the other hand, there were some tasks that were delegated completely to the Commissioner’s Office as well. Rob Wiblin: What kinds of things did you just delegate to the Commissioner’s Office? Maha Rehman: So planning the implementation of the launch event, that is something they have… I mean, they do these events every now and then and that’s something they can plan. I mean, they can get the right media involved, they can get the right media collateral, reaching out to the health minister. They have that system in place. And that also aligns very well with their incentives. And so something like that, I would delegate completely to the Commissioner’s Office. Other events that served as reinforcers, I would delegate completely to the Commissioner’s Office. Also the security, the flag marches, that’s something they know how to do well. The one key element of the campaign I was very closely involved in was getting the messaging right. Getting the reinforcement, getting the micro-level interventions right is something I was very keenly involved with. And other things which I feel like the Commissioner’s Office has greater specialization in doing well, that was delegated to them. And they were able to get the required bureaucracy, ministers, everyone involved that gave the entire campaign a lot of limelight in the media. Rob Wiblin: So how many people were you directly managing? Maha Rehman: In LUMS, initially I started off with a team of two RAs and then quickly we got a survey firm on board and that was 40 to 50 direct data enumerators who were working with me. Then there was a team of about 20 to 25 people in the Commissioner’s Office who I was working with. And then I slowly also increased the number of people in my own LUMS team because when the intervention was in the field, and I also needed alternate data sources, and I also needed to set up alternate systems of monitoring. Maha Rehman: So around 10 in LUMS, 20 to 25 people at the Commissioner’s Office, 40 to 50 people who were collecting data in the field. But the data was being collected through the app so that was not a lot of daily communication. I would ask them to systemically record their observations and then that was just collated at the backend. So at one point it does get fairly overwhelming, but I think you just need to keep pushing through. Rob Wiblin: It sounds that way. Maha Rehman: Yeah. Once it is in the field and after the first two weeks, when you’ve troubleshooted the key problems that you come across, then you come across problems of a very different nature. So with the campaign going big in the media, then there’s the media management to take care of. Who’s reporting what? What are they talking about? The right information and the facts need to be disseminated. We also had a lot of ambassadors on board, getting the messaging right with ambassadors, ensuring that everyone in the field is treated with respect, and all partners are also kept happy and they feel that they’re being credited for the work that they’ve done in the campaign. Rob Wiblin: What kinds of things did you change about the messaging or the deployment based on data that you were getting back? Maha Rehman: So in the rural areas, because the communities are more cohesive, people tend to listen to imams a lot more. When the same information is delivered through imams in an urban setting, the receptivity is relatively lower. And so that is a key part of the campaign that you quickly had to troubleshoot around. Then what is the information that the imam should be disseminating? Maha Rehman: And the second thing is, once people come into a mosque, even if they’re wearing a mask outside, they tend to take it off, either for performing ablution — washing their face, washing their hands — or for the simple reason that they think that they cannot put on a mask during the time that they’re praying. And so I think mosques, because they’re an enclosed space, they’re also an area for high transmission. And that is when it is critical to keep wearing masks when you’re inside the mosque. And so how do you then ensure that people are also wearing a mask inside the mosque? I think that also became a critical element. Maha Rehman: Then in the markets, another thing that’s different about markets and mosques in an urban setting is that you don’t see the same people coming to these markets every day. And so the intensity of the reinforcement needs to be greater. And then you also need constant reinforcers there. Instead of a person standing there who’s reinforcing the message, there needs to be a very clear billboard. You need to also then work with the shopkeepers and the imams so that they’re also reinforcing the message to help whoever’s coming to them. And so then you need to ensure the permanence of your message in these areas, just because of the fact mobility is higher and the number of people coming to these areas every day is just not the same, unlike in a rural setting. Rob Wiblin: What did you do to get people to wear the masks in the mosques? Was it as simple as telling the imams to tell everyone to keep their damn masks on? Maha Rehman: So there were a couple of things. We iterated on the messaging in a way that the imam then made an announcement, “Please put on your masks, so that we can begin the prayer.” So before they began the prayer, he made a very explicit call of action. The second thing is instead of distributing masks just outside or instead of just putting a box of masks there, the promoters actively distributed masks inside the mosque. So after people are inside, they’ve done their wudu, they’ve washed their faces. That is the point when they’re given a mask and they’re asked to keep it on. And then the promoters in these areas were asked to play a more active role in talking to people, ensuring that they’re actually complying with the messaging. And just adding the fact that the Commissioner’s Office wants everyone to wear masks also had an impact in some areas. In some areas, where the imam was anti-government, we kept the messaging clear of any government branding. Rob Wiblin: Is it kind of a specific aspect of working in Bangladesh or in Pakistan that mosque attendance is very high, so it’s an extremely useful kind of institution to target because you can reach a pretty large fraction of the whole population if you can get all of the mosques on board? Maha Rehman: I think it’s definitely a very important institution, but the one drawback is that it’s very gendered. You only have men coming to the mosques for prayers, especially in Pakistan. There are very few mosques that also have women coming in. So while it is a very important part of the community announcements, part of the community setting, where people in the same neighborhood get to see each other… And so it’s a very strong institution for disseminating a message like this, but in order to then have a more balanced approach in terms of gender, then you also need to complement it with other interventions. Rob Wiblin: That makes sense. Were there any disasters or emergencies that you had to respond to in the middle of the project? Maha Rehman: There were several. Some I can speak about, some I can’t speak about. Rob Wiblin: That’s understandable. Maha Rehman: But there was a lot of media management that had to be done in terms of ensuring that the partners get the credit for what they’re doing. I mean, so it’s okay to over-credit people, but nobody should feel under-credited. Maha Rehman: Also, when a campaign gets big, then there are a lot of people looking for some sort of criticism. I mean, so because this was designed in a very positive light, that was a lot harder to do in this case, but then there were other ways in which the messaging had to be managed in the media. And so I think that also became very critical. Maha Rehman: A key thing that gave this campaign a lot of credibility was at NCOC — that’s the federal COVID body in the country. They noticed the campaign, the fact that it was done in a very cost-effective manner, and issued a directive that other districts should take up design and deliver similar campaigns. So then this campaign made it to the formal directive that was sent out from NCOC asking all commissioners to adapt and follow a model similar to Lahore Wears Masks. And I think that was very encouraging and heartwarming for the commissioner’s team in Lahore as well, as they had put in a lot of work. Maha Rehman: And because this setup was already there, it was set up and so when there were enough vaccines and they wanted to increase the vaccine uptake, it became very easy for the Commissioner’s Office to do that. They could easily, you know, they already knew how to offer information, how to reinforce that message. If they’re going house to house, how should they deliver information? What are the required resources? What are the sponsors that they should get on board? Maha Rehman: I mean, a key element that was adopted was messaging from the mosques as well as mobile announcements. So there’s a loudspeaker and a cassette that’s attached to a truck, and that truck goes from one place to another making these announcements. And the similar data collection app was then also designed for the vaccination campaign. So I think an externality of this was that it allowed the government to adapt some of these interventions as something that it would be doing as a routine feature. Something that it found easy and it found effective. They already now have a monitoring cell in the Commissioner’s Office. Maha Rehman: And the fact that they’re now building their own volunteer force. Because there were a lot of volunteers that were required in the mask-wearing campaign, that are not volunteers that are required now. So they’re formalizing their relationship with various universities where these volunteers in some universities would also be given credit for being part of these campaigns. Formal credit, a one-credit course or a two-credit course. And so the fact that these interventions are now being formalized as part of the government mechanisms is also I think another big success of this intervention. Rob Wiblin: Is it common in Pakistan to stick loudspeakers on trucks and drive them around? I worry that if someone tried doing that in London they might get assaulted. Maha Rehman: No, that was very interesting for me as well, but that does have a lot of receptivity because it’s something unique and it’s a unique way of going around a community and making those announcements. Rob Wiblin: Yeah. Interesting. Okay, so people didn’t mind it so much because it was an unusual situation. Maha Rehman: Yeah. Yeah. Rob Wiblin: So you were trying to send out surgical masks, and I think they were reusable ones, to every single person in Lahore. I guess you’re saying obviously it would be incredibly difficult to go and collect the addresses yourself, so you basically took that from the electricity network and then — Maha Rehman: Lahore Electric Supply Company. Rob Wiblin: — there was some data cleaning to make it all work. But what does the supply and logistics of receiving 10 million to 15 million face masks look like? And then, like, where do they get delivered? This seems like it might have been a little bit outside of your expertise. Maha Rehman: So in the masks that we use in the current campaign, a lot of those masks were collected from philanthropists and corporates. There’s another mask shipment that’s on its way to Lahore that will deliver 24 million masks. That’s the largest mask shipment that Pakistan will have received to date that has been made possible because of this work with the Lahore commissioner. Maha Rehman: The second-largest shipment that Pakistan has ever received was that of seven million masks. And the fact that we are now getting Lahore 24 million masks that will then be used. So this campaign hasn’t ended, because just like COVID is going to be an ongoing phenomenon — with one wave, the second wave, third wave, fourth wave — we are now getting them 24 million masks that will then cover all of Lahore. So in the first situation we will not cover all of Lahore because there was a specific number of masks that could be delivered. But with the current shipment, Lahore will be well covered with the masks that will be made to every household in the city. Rob Wiblin: Yeah, I guess what does the back end of that look like? Because I imagine 24 million masks, or even seven million masks, is going to just require a huge warehouse to put them in and someone’s got to stick them in an envelope. There’s just so much actual work to be done to get those deliveries to happen. Was it mostly just delegated to the postal service? Maha Rehman: So the masks are packaged with a letter from the commissioner at the Commissioner’s Office. Some of them are packaged at the Commissioner’s Office, some of them packed at the post office. They have been handed out lists of the addresses and a printed list that they can easily use. Maha Rehman: And they are then sorted out area-wise: that these are the letters that go in this zone, these are letters that go in the next zone. And then they’re delivered, in which the postman who’s going house to house is also trained to deliver a pitch, that “This mask is coming from the Commissioner Lahore, you should wear it because it protects you and it protects everyone around you. And so please keep it, it’s washable. You can use it multiple times, you can wash it and then reuse it. It still retains its filtration capacity.” So initially the filtration capacity is around 90% plus, but even after 10 washes it’s still 76% effective — the surgical mask, that is. And so that message is very carefully delivered via the postman as well when that mask is being delivered. Rob Wiblin: Did you say that they’re actually put in the envelopes at the Commissioner’s Office? So the masks are delivered to that office and then I guess a whole lot of government staff who are already there are actually packing them in. Maha Rehman: Yes. Rob Wiblin: I just think the number of hours that would go into packing 10 million envelopes just seems very large. Maha Rehman: Yeah, that’s correct. Rob Wiblin: Wow. Maha Rehman: So they had a limited number of masks for the first iteration, but with the next shipment that’s coming in, that will be a whole lot more work. And they’re also now planning to involve other government departments, for example, the health workers as well as the staff in the education department who may be able to spare time for this activity. So they will also be getting staff on from other departments that can help them seamlessly execute the packing and then the delivery. Rob Wiblin: Yeah. So I guess you needed to get cooperation from a whole lot of other people and the imams, the leader of the electric company, the people at the post office, the people at the Commissioner’s Office… Is there anything that you learned about how to get people to be excited about the project? I guess one thing you were saying is having the data on what impact it’s going to have was really helpful, but is there anything else? Maha Rehman: I think the first thing is you need to have a clear owner of the campaign. And the campaign in Lahore was clearly owned by the Commissioner Lahore. So that’s key. There needs to be one owner who’s driving the campaign forward. Maha Rehman: The second thing that’s key is everyone’s incentive needs to be aligned. So we are all at a point where there’s a lot of COVID fatigue. We want to go back to our normal lives, we want this to end. And so since the government’s focus is on fighting COVID and finding ways that are more effective, so everyone then sort of aligns with what the federal government wants. And so the government department’s incentive was to then play its part in a way that they could come up with effective ways that contributed to the federal government’s overall work plan and the strategy. Maha Rehman: And of course the recognition that comes along the way is something that everyone craves. The media attention, the recognition, and the commissioner also got an award for being the best commissioner in the province. And so all these little awards that come along the way, media attention, and other countries, I think they keep everyone excited and engaged and passionate about the cause at hand. Rob Wiblin: That’s really interesting. You’re saying perhaps it might have been a bit more challenging to make this work if it was invisible work that people weren’t getting credit for, and that they weren’t getting media coverage in the US or the UK about how this campaign in Lahore is going well. The fact that it’s visible, that people appreciate what’s happening, I guess makes it easier to get a wider range of people to contribute. Maha Rehman: Yeah. Rob Wiblin: Yeah, did you learn anything else about stakeholder management or was there anyone who was resistant and perhaps required some extra encouragement? Maha Rehman: So I think what I’ve learned working with the government over the last couple of years is that you really need people who are willing. If there’s somebody who’s unwilling right from the get-go, there’s very little that you can do to bring them on board. And there are always multiple people in the ecosystem who can become your allies. So in this case I had also reached out to some decision makers in the health department, but they weren’t as excited as the Commissioner Lahore. Maha Rehman: So then instead of spending time with stakeholders who don’t seem so excited and who… If they’re not excited from the get-go, they’re not going to put in the energy and the effort required to see this campaign through. And so then it’s no use following up. In fact, then you need to turn your attention to people who are more receptive of your message and who are more likely to take this forward. I think that is one thing that is key. Maha Rehman: The second thing is, because there was a lot of media attention, there was a lot of public commitment to achieving the impact. And so once a bureaucrat or a policymaker makes a public commitment, then that sort of becomes a self-accountability device. Rob Wiblin: Yeah. Maha Rehman: And that, “Oh, I’ve gone on TV and I’ve said so-and-so, and we’ll do so-and-so by this date,” then I think there’s nothing, I mean — Rob Wiblin: It’s hard to stop it. Maha Rehman: — that acts as a very strong driver in which then that bureaucrat is going to drive that team forward. So public commitment really works well, finding the right champion, finding people who have the right political clout. Maha Rehman: And then another thing that really works well is if you can get this entire coalition formalized in some sort. Either get a notification issued that formalizes this as an official coalition or something on similar lines. So adding a bit of a structure to a coalition like this also works really well. Rob Wiblin: As a data analytics person, did you have any issues with kind of unreliable data, or data quality issues in general? Because it’s often so hard to collect stuff that you can really believe. Maha Rehman: So initially when I was shortlisting firms as to who would I want to work with, there were several firms who did not deliver up to my expectations. So I very quickly weeded out people who were not delivering quality results and firms who were not delivering quality output. Maha Rehman: And so I mean there’s a bit of a quick action involved. So quick action over perfection — so you learn something and then you quickly act to take the next best step. So of course there were issues, but then you build in the right data checks: ensure that the data is completely collected, ensure the numerators understand the intervention as you have understood it. And then take quick action where there is needed. At times action may require you to be a bit ruthless. I mean, yes, you do need to take hard decisions at times. Rob Wiblin: Yeah. Okay, so the data collection, you were kind of outsourcing that to other companies. And then it was a matter of telling who was doing a good job and then increasing their role and… and firing a bunch of people who weren’t working out. Your role has been in kind of more academic work in the past, where I imagine you were actually getting to contribute to writing papers, to doing data analytics. It sounds like your role here was like, as a manager, that meant that most of the day I suppose you were in meetings or coordinating with people or getting people to do what you needed, firing people who aren’t performing potentially. And I guess lots of email. How did you find that transition? Because it’s quite different work, I guess. Maha Rehman: I mean, it’s great that you mentioned that. When Mushfiq reached out to me in March, I was at a point where over the past year I had put forward a lot of policy recommendations based on data. But I was at that point where I really wanted to see some of that actually put into practice. So the fact that you’re putting papers out, and you’re working with data, and you’re giving policy recommendations only gets you so far. And there was also a bit of frustration involved, that why can’t we get these basic things right? I mean, how do you make a policy more effective? Maha Rehman: So a lot of my work has been around what data can you use to make policy more effective? What data can you use to make a program more effective? And because I wanted to actually see the implementation of those data-driven results… Throughout, this was a bit of a different domain that I was stepping into, but because I was so passionate about actually seeing it through… So it was a difficult couple of, few weeks. I was also learning something new every day. It’s not easy. Rob Wiblin: Yeah. Maha Rehman: Bringing such a big team together, everyone’s expectations have to be managed. You have to live up to everyone’s expectations. And so I mean it does take a toll on you. The fact that we are already in COVID, there’s also a lot of mental stress involved with that, but the fact that you are pulling such a big coalition together also does take a toll on you. Maha Rehman: And so I think the fact is that I keep going, and the reason I keep going is because I really enjoy working with data. And the fact that I was actually getting to adapt a very credible intervention that had seen very credible results in Bangladesh, and adapting that for a city like Lahore. And I had a lot of feedback also from Mushfiq and Jason. I think that setting kept me going, because I really wanted to learn what would work in Lahore that could actually put an end or slow down the spread of COVID. Rob Wiblin: Yeah. I mean COVID and lockdowns in particular seem to have taken a massive toll on people’s mental health. I can stick up some links to some data that the Office for National Statistics in the UK have collected suggesting that rates of anxiety or people reporting serious anxiety have doubled or tripled since February last year. Rob Wiblin: And I mean, you would’ve been under enormous strain or a lot of pressure to avoid being a bottleneck to keep these things moving very quickly. Did you do anything to kind of take care of your own health, I guess mental health, physical health, to ensure that you could actually keep on going? Maha Rehman: So I run very regularly. So either that’s early mornings or in the evenings, that’s something I definitely do. I always also take time out for myself, whether that’s half an hour just for some mindful activity, whether that’s reading, or something that I’m just doing that will help me zoom out of what I have done the entire day, and just mindfully think about something that will put everything at ease for me. But one thing that really, really keeps me going is my social network, my family and friends. And then of course I always make it a point of incorporating some sort of physical activity, and usually that’s running. I think that’s very important. Rob Wiblin: Yeah. Did you ever have to say, like, “No, I just can’t do this. This is, this is more hours than I can reasonably work and it’s too risky for me to be taking on additional people, additional projects.”? Maha Rehman: That happened at a point where the mask-wearing work hadn’t really ended, and we were beginning to plan the vaccine intervention. But then I was able to time it out in a way. But going into this kind of work, I knew that this would get very overwhelming, and so I had to keep my expectations realistic. Rob Wiblin: Yeah. Is it possible to tell what impact roughly the intervention in Lahore had? Did you manage to kind of get a similar shift in mask wearing from 10 to 40% that was achieved in the Bangladesh villages? Maha Rehman: So we’re just completing the endline results this week. And so then we’ll get down to analyzing the data and we’ll be able to see, but there has been a consistent increase in mask-wearing in mosques and markets. The exact magnitude of that, I will be able to tell once I’ve looked at the numbers and analyzed them. But since the endline data is just coming in, maybe I can report back on that in two to three weeks. I can put out informal results, but the results look promising in spite of the challenges that are there in adapting an intervention to an urban setting. Rob Wiblin: Yeah. In my mind, delivery of services in the real world is kind of an art and a sort of expertise that I feel like I don’t quite have. I mean, I imagine, “How do I get 1,000 different mosques to be promoting masks?” And I just don’t even know where I’d begin, because I suppose I just read articles online and make a podcast. But am I right to think that this kind of logistics operations expertise is a specific aptitude that people could develop? Or is it something that maybe you found that you’re actually just quite good at and could throw yourself into pretty quickly? Maha Rehman: I think my key passion is how can I effectively work with data? And that is something that allows me to drive all other aspects, interventions, tasks associated with it. So working with data makes me really happy. Putting out actionable results makes me really happy. Being able to contribute productively to policy and programs and making the programs makes me very happy. Maha Rehman: So a lot of my family members have been in the government, and so I’ve seen how very simple things don’t work well. In LESCO, for example — Lahore Electric Supply Company — a lot of issues just stem from poor management. And so when the fixes are so glaringly obvious based on the data that you have, why is it that you cannot fix these coordination failures, or you cannot incentivize these government offices well? I think these are very important research questions that I want to definitely investigate in greater depth. Maha Rehman: And I think working with data is magical in a way. You combine data and theory and you have magical, simple results that will allow any policy and program to be more effective. And so that’s just something that motivates me immensely at a personal level. And so anything that involves this at the core of its design will definitely always keep me going. Rob Wiblin: Yeah. Well, yeah, congrats on managing to have so much impact or actually… yes, deliver something to hundreds of thousands or millions of people, even though it’s kind of not your core competency or not the main thing that you were trained to do. It seems like economists have cross-cutting skill sets potentially. Maha Rehman: I think that also brings together two of my core interests. So besides economics, I’m also an Acumen Fellow. And so I’ve had a bit of interest in entrepreneurship. And so when I was beginning my Acumen internship, I was asked what my objective in life was. And I said, “I want to build data-driven institutions.” At that point I did not know what that meant. I mean, what does it mean to build data-driven institutions? And so any opportunity that comes along to have impact using data is always something that I hop onto. Rob Wiblin: Were there any, perhaps, disappointments? Of things that ran late or didn’t quite work out where you had to make sure that you didn’t get demoralized? Maha Rehman: Oh there were lots of disappointments. But the way I deal with it now, after having gone through a lot of rough patches, is to not get worked up about it. There’s no point in getting worked up about it, but think about what’s the next-best solution as to how can you deal with it? That’s the first thing. Maha Rehman: The second thing is there are only so many battles you can pick. So while I would want all elements of a campaign to be perfect, 10 will be, 10 will not be. And so I really need to pick and choose my battles. So what is it that I can be more fussy about and what is it that I can’t be more fussy about? And so pick and choose what you want to fight about, what your battles are going to be, not react to it, not take it personally. People will have different things to say about the intervention, about how everything is being done, about why is mask wearing important, it’s not important. Maha Rehman: There was some critique of the flag march. Some people came up and said that, “Oh, this is just reeking of the privilege that bureaucrats have.” Yes, but then it’s also doing a lot of good to the society. And that, “If you’re going around the city in your Land Cruisers, you can’t really relate to a common man.” So this sort of feedback when it comes, I mean, initially it is a bit of a setback, but then you don’t take it personally and respond to it once you’re not so angry about it. Rob Wiblin: Yeah. Maha Rehman: Because there are only so many battles that you can pick. Rob Wiblin: Yeah. Maha Rehman: And so the harder the challenge, the longer you run that day. That’s a key. Rob Wiblin: Yeah, I’m curious to hear… There’s a real bias in reporting towards things that did work out that people are proud of. Maha Rehman: Yeah. Rob Wiblin: What are some things that didn’t work out, where you just gave up, and what were the reasons? Maha Rehman: So in the mask work, I think I really pushed things through, but there were other things in the past year where I haven’t really. I mean, so there were things that didn’t really work well. And so then you really need to see what’s the end benefit of all the effort that you’re putting in. Are you getting what you thought you would out of it? If not, then sometimes it’s better to let go than to keep pushing through. Maha Rehman: So I think in the mask campaign, I mean we’ve also been fortunate in a way that we were able to work through the challenges. There wasn’t something that we weren’t able to work our way through. And the fact is that there were a lot of willing partners in this, so I had a lot of support. Mushfiq and Jason were also very involved in the campaign. The commissioner was willing to see it done well. His team was well-meaning; there were some people that had to be pushed more than others. So with regards to the mask intervention, I think this is one intervention in which we were able to work through most of the challenges. Rob Wiblin: So pushing on, I’d love to talk about this other research project you’ve been working on for years now which just got published in one of the top journals in the field, The Quarterly Journal of Economics. The paper is titled “The Allocation of Authority in Organizations: A Field Experiment with Bureaucrats.” Tell us a bit about what you were trying to learn with that work and some of the hurdles you faced. Maha Rehman: So another key project that I’ve done that is very close to my heart is we worked with bureaucrats from five different government departments. And in that we were trying to measure what are they buying, what price are they buying it at, and how can we motivate them to make financially efficient decisions? And so there were a couple of interventions we implemented in that: the first one was giving them incentives, the second one was relaxing rules, the third group received both the interventions, and then there was a control group. Maha Rehman: So when we were trying to relax rules, we had to work with the government to get some laws changed. One thing that we were trying to do was to give the bureaucrats petty cash, so that if they have enough petty cash on hand, they can pay the vendor upfront. Because if they delay the payment, the vendor charges a much higher price and that has a large implication for the overall budget. Maha Rehman: And so it took us one and a half years to actually just get one permission or that one approval that allowed us to implement this. And in that we had to work through numerous bureaucrats, had to listen to a lot of things they had to say. And so that was I think my first real experience working with the bureaucracy that now allows me to work with the bureaucracy so effectively. Because in that case I was working across five different departments: education, health, agriculture, communication, and works. And of course also working with the finance department. Maha Rehman: And to collect that data was a massive exercise on its own, right. We worked with the Punjab IT Board to develop a management information system that then collected data on selected goods from all these offices. And one of the baseline results, which was very surprising for us, was that for some goods there was up to 200% price variation in the same good being bought at the same district at the same time. Maha Rehman: And so there was a lot of very interesting insights that went into it that then allowed us to design the interventions that we then implemented. One of the interventions was getting the budget released early, so they would tell you the budget. I mean, that’s a very hard intervention to pull off working with a bureaucracy. Rob Wiblin: Yeah. Maha Rehman: They would tell us a budget has been released, but it would not be released. Even if it was released, the bureaucrats were not able to use that funding. Maha Rehman: When you were working with incentives, one officer would tell stories on another that you’re just offering an incentive of 100,000. There’s at least 10 times more corruption in the system. So when there’s a possibility for 10 times more corruption in the system, why would someone be incentivized by an amount as small as 100,000? Maha Rehman: Then we went to the higher education department, in which the principals… We asked them to make a budget. Then the principals did not have much financial literacy, they were not able to make their own budgets. And we found out that there is a photocopier sitting in a vegetable market who just photocopies last year’s budgets for them and that they can then come in and submit to the finance department as their budget for the current year. They don’t know how to make budgets, so they’re just photocopying an existing — Rob Wiblin: Oh wow. Maha Rehman: — budget that they can then come in and submit. So there were very interesting stories that we entirely discovered in the process. And that I was working with Oriana [Bandiera], Michael Best, Dr. Adnan Khan, and Andrea Prat. And I think that was a very, very rich learning into how to navigate these corridors, how to get people to agree to what you are saying. But that project also, like the mask project, had a lot of policy impact. Rob Wiblin: Yeah. Maha Rehman: The management information system has now been adopted by the government, and they will now be training people across departments. I hope they actually implement that as well, but that had some very stark and promising results. Rob Wiblin: Nice. So this was a study of procurement in government departments across Punjab, not just Lahore. And obviously there’s a huge amount of money that’s getting spent by governments everywhere, including in Punjab, and famously I guess government procurement is kind of a mixed bag. It’s possible potentially for governments to get quite poor value if the incentives for bureaucrats are set up somewhat poorly. Yeah, so what was this experiment trying to figure out? Maha Rehman: Yeah. So in this experiment we worked with different departments across the government machinery, but we were trying to see how can we reduce the waste in government procurement? So why is there 200% variation in the same goods being bought in the same district at the same time? How can we get the bureaucrats to make financially efficient decisions? Why are they not making that decision? Is that active waste or is that passive waste? Active waste is when they’re actively corrupt. Passive waste is when they just don’t care enough about the problem, delegate it, and then waste… I mean, so there’s wastage of resources because the people that they’re delegating to also don’t really care much about being efficient. And so how much of it is active waste, and how much of it is passive waste, and how can we actually reduce that waste in the system? Maha Rehman: And so the first intervention was that anybody who achieved better value for money would be given incentives. That was the first intervention. The second intervention was that we would relax rules: we would make it easier for you to pay your vendors, we would make it easier for you to buy the material that you need for your day-to-day working. And then we evaluated the impact of these interventions, and to our surprise, of course relaxing rules had more of an impact, but when the monitors’ incentives were also aligned with the officers’ — so of course these officers also have somebody they’re reporting to, their supervisors — then you also see an impact in terms of the incentives working. Rob Wiblin: Yeah. So maybe let’s just explain the procurement setup because it’s a little bit complicated. It took me a little while to understand it. So of course there’s bureaucrats who are trying to procure goods potentially for their offices or their projects or whatever. But there’s an interesting and important process that one has to go through in order to do procurement in this context, which is part of the reason why I think you were interested to study procurement in Punjab in particular. Maha Rehman: I mean, so to explain it as simply as possible, it’s a long process where first a request is made, then different price quotations are sought on that request. But in this experiment we were working particularly with petty procurement, so procurement of less than 100,000 rupees. And so that involves various kinds of goods, but then we selected and focused on goods that were similar across all these offices. So in that there is a particular system that is followed: a request goes to an officer, who then signs it off, then price quotations are sought, and then the product is ultimately purchased. The payment can either be made upfront if the government officer has cash, or if the officer does not have cash, the bill will then go either to the AG [accountant general] office or the district accounts office, and they will approve the bill and then they will release the cash. And so we often talk about middlemen in agriculture being a mafia. But in this experiment, we discovered that the district accounts office were also a mafia. Rob Wiblin: Yeah. Maha Rehman: They were also making several excuses before they actually approved the paperwork, released the payment, and they just make life a lot harder for any procurement officer who’s going to get the bills cleared. At times the procurement officers also have to travel a lot out of their offices and get their bills approved. So this in turn, it sets its own rules. It works when it wants to. The officers there, they don’t work when they don’t want to. Maha Rehman: And so what we did was, we mimicked this entire system and developed an online system. So the system that was collecting data for us was also following the same processes, that first a request will have to be made, then that request is approved, et cetera, et cetera. There was a separate account for the procurement officer. There was a separate account for this officer who then signs off on this procurement, who’s known as a “drawing and disbursement officer.” Maha Rehman: And so the entire system was automated, but in spite of that, they were not allowed to use that. Even the printouts from the system did not really have legal value. And so they really had to still… I mean, these two systems literally then ran in parallel. They were still working with the old cumbersome manual system. And because for some reason, some laws, financial laws in Punjab need to change before this entire system can be automated. Maha Rehman: I feel a lot of inefficiency in the entire supply chain can just be taken care of if you automate a lot of these processes, and there is no face-to-face interaction. Because face-to-face interaction allows another person to either ask for a bribe, or to delay the process, or to not approve the paperwork. And so there are several inefficiencies at several levels. Rob Wiblin: So part of the service, you’ve got these procurement officers who are trying to buy stuff. And then even after they’ve bought the thing and the thing has been delivered, they have to send off the receipt basically to the accountant general’s office, where they could potentially hold it up. And they kind of are accountable to no one in particular, so they could potentially delay the payment if the procurement officer hasn’t been doing what they wanted or perhaps chosen the right person or pays them a bribe in order to simply process the paperwork. Rob Wiblin: The interesting thing is the accountant general’s office and these monitors were added, I think, in order to try to reduce corruption, in order to create more accountability for the procurement officers. But there’s obviously a risk that they’re just adding another level of bureaucracy and corruption on top of it. And I guess to some extent, this experiment was able to pick that up. Maha Rehman: I think it was able to very clearly show what the inefficiency in the system is. And so there’s several proposed solutions and ways forward. But I think at some level it will require a lot of legislation to be looked into before other alternate processes can be adopted or the recommendations can be adopted, or even the MIS [management information system] that we’ve developed, trained government officers on, iterated on the design… Even for that MIS to be adopted across the government machinery it will require legal changes. It will require cultural change. I mean, because then these monitors and people at the AG office and the district accounts office, they’re also forgoing a large share of their power when they allow such things to happen. Rob Wiblin: Right, yeah. So the different interventions you were testing, one was just giving the procurement officers money that they could use to pay for things directly without having to consult with anyone else. The other was doing performance tests, where if they were able to get really good value on the things that they were buying, then they would get incentives. And there was also just, yeah, cutting the accountant general’s office out of the picture, which I guess, presumably they weren’t too keen on this happening? Maha Rehman: No, that’s something we did not try to do, but that’s something we proposed as a way forward. So in terms of relaxing rules, one was giving the government officer petty cash so that they can pay upfront. The second was just listing out all the rules in simple, plain language so that the procurement officer knows what the rules are, and is not misled by other people in the system, and can efficiently just follow those rules. So we sent out a list of rules in very simple language, taking account of all the scenarios — that was known as the AG Checklist. The third thing, that was getting budgets released early. That is something that the government did not comply to as we had hoped. That was just a difficult intervention to pull off. They would say one thing, do another, et cetera. Rob Wiblin: And which of those things did you find worked and which one didn’t? Maha Rehman: Yeah. So in terms of rules, we see a clear-cut effect on the fact that relaxing rules reduced the per-unit price that officers end up paying for the goods at hand. The incentives only work if the monitor’s incentives are also aligned with that of the officer who’s signing off on these documents. I mean, so there was also a lot of data collected in terms of the DDO’s [drawing and disbursement officer’s] personality — their personality surveys, their behavior was mapped, their preferences were also mapped. In terms of the incentives of the monitors, we also used data as to whether they are actually delaying payments or they’re approving payments on time. And we mapped out their behavior through their tendency to either pay on time or to delay payments. Maha Rehman: And so in that case, the monitor’s personality, preferences, and behavior also had an impact that then determined whether the incentive treatment worked well or it did not work well. It worked well where the monitor’s incentives were also aligned with that of the drawing and disbursement officer, the agent at hand. But to just put it simply, if there are people around you, or if anybody who needs to approve your bills is acting in good spirit, the officer will be motivated to achieve efficiency. If that person is not, and any efficiency that you otherwise achieve in the system ends up going as bribes to this monitor, then of course there won’t be as much motivation or an incentive to do well. Maha Rehman: So that’s this intervention put in very simple words, but this intervention had a lot of externalities. We were able to train the principals and the government school teachers on how to create a budget. We were able to give the government a well-developed MIS system that they can just now adopt and use. We also left them with a lot of learning in terms of what are the procurement processes. There were a lot of conversations with different bureaucrats involved. The policy results were disseminated. The then-chief minister had a keen interest in this experiment and the results, and he wanted to implement that and integrate that within the machinery. And so all the groundwork is there. It has been set. I think it’s now for the government to change the rules required before it can also finally be embedded in the government machinery. Rob Wiblin: Yeah, I guess, so it seemed like you were picking up that there were kind of two different classes of monitors, some of whom were mostly harmless and another group where if you managed to get them out of the picture, or at least not require them to approve things or force them to approve things, as long as particular criteria were met, then you were able to buy all of these different things for less money. And similarly with the incentives, if someone was working with a bad monitor, then it didn’t matter what incentives you gave them because they didn’t really have very much flexibility to improve their performance anyway, because they were just being controlled by this person higher up on the chain. On the other hand, if the monitor was good and doing their job well and wasn’t corrupt, then the incentives could improve the outcomes because the procurement officer actually had the flexibility to perform better if they were motivated to. Maha Rehman: Absolutely. Yeah. Rob Wiblin: Yeah, is there any kind of maybe generalizable thing from the story? I guess it seems like the government has probably added this extra level of oversight in order to try to reduce the amount that people were paying, in order to reduce corruption among procurement officers. But adding oversight has merely potentially increased costs, or it seems like it has increased costs on net. Does this teach us anything about how to design organizations? Maha Rehman: Some of these behaviors are no secret to anyone who works in the bureaucracy, and so I think we just need interventions that will be able to change the system legally in a way that there is no room for these monitors to engage in behavior like that. One way is to reduce the face-to-face interaction and automate these processes. So anything that’s going through the system, you don’t change the amount or decrease it or increase it based on what the other person at the window is asking you to do. And so automating these systems, I think, will have a big impact. That’s the first thing. Maha Rehman: The second thing is whenever we’re talking about change and we’re thinking about what’s the wastage in the economy, we are building on this corruption narrative. That’s everyone’s favorite in Pakistan these days. We need to stop and ask ourselves if that’s the real problem, because unless and until we get to the underlying reason for that problem, we won’t be able to treat it. So more often we are treating the symptom of the problem and we’re not treating the underlying cause. If you’re not treating the underlying cause that’s actually leading to the problem, you will not really deal with the problem. Then you would only do away with the problem for a month or two, and then it will come roaring back again. Maha Rehman: That also comes very frequently… So I’ve also worked extensively on teacher recruitment and rationalization. Rationalization means that teachers are posted in areas where they’re most needed. At times that’s away from where they’re living, and in cities that may not be very attractive in terms of the living standard and the living style, and the places where teachers would not want to be posted. And so every time you come up with a policy that tackles this problem of placing teachers in these far-flung areas, I mean, it only works well for a month or two. And so we went into the field and we asked the clerk. I mean, so this time there was a lot of money from World Bank and other donors going into it, and yet we still land back on the same problem. And the clerk was like, “I don’t know anything about the new policy. I just photocopied last year’s list and I sent it out.” Maha Rehman: And so a lot of planning and a lot of thinking went into designing the right policy, but one, it was actually… got down to the implementing officer and nothing changed. And so there was nothing wrong with the design of the new policy, but the fact that it wasn’t actually even implemented is what needs to be dealt with in the next intervention. Maha Rehman: So hence my point that you need to identify what’s holding everyone back from achieving the desired impact. And that’s also a big learning you get in terms of… because whenever you talk about inefficiency in the system, the first narrative that anyone wants to deal with is corruption. Though this study is giving us very different results in terms of people’s behavior, what interventions work, what don’t work. And so you really need to understand the ecosystem, the stakeholders, their incentives, what’s working, what’s not working before you’re designing a policy that will work. Rob Wiblin: Yeah. Yeah. What do you think was the underlying cause in this procurement case, if not corruption? Maha Rehman: So I think there are just way too many rules, and some of the rules are not well understood by the drawing and disbursing officer. And that lack of information, and that lack of clarity, and the fact that there are no laws that allow for this interaction between the monitor and the officer to be transparent, that leads to a lot of manipulation at the end of the day. Maha Rehman: And so the incentives, the effort needs to be rightly rewarded. The mechanism needs to be transparent. These officers need to be better trained. The rules are just so complex, and they’re so detailed, and they’re so long. There’s so many workshops that the procurement authority holds on these rules. Nobody goes to those workshops, nobody knows what the real rules are. And then at the end of the day, you can also put them in jail for not following those rules. So there’s also a real fear of not actually doing anything because you may just end up doing it wrong. Rob Wiblin: Yeah. Yeah. I think this is a story that will be familiar to a lot of people. I think at the end of the paper you speculate that these rules — these procurement rules, these anti-corruption rules — haven’t really been designed to minimize the government’s procurement bill for a given number of things that it wants to buy. They’re there more to reduce the risk of a big scandal that can produce negative publicity. Rob Wiblin: Because you’re getting like paper cut to death here because all of these different processes increase the cost. They will increase the staffing cost of course. They increase the amount of time that goes into procurement. And interestingly, people who sell things to the government demand to be paid a whole lot more because it takes so long for these bills to be paid sometimes, because they’ll just get stuck indefinitely without any recourse to complain that your government bills haven’t yet been paid. So it’s possible that in efforts to reduce corruption, in fact, the government has ended up paying more basically for everything. Maha Rehman: Absolutely. Rob Wiblin: It sounds like it was a huge amount of effort in order to get cooperation across all of these huge number of officers. I think the data set seemed like it had millions of procurement decisions potentially, and thousands of monitors, and thousands of the procurement officers that you were tracking that you had to actually get access to all of this information. And of course there’s people who weren’t motivated to provide it, to actually assist the experiment. Do you think, given the delays, is it worth trying to coordinate with government as an academic, or is it better perhaps to find other places that you can do experiments more quickly? Maha Rehman: I think you need to build a smart coalition around that. You need to have people on board who will help you get that required buy-in from the government. And so it’s critical that you have a smart coalition where you have the skills on board that are required to actually get that buy-in and keep the experiment going. It does take time, but then you learn, and the next time you do it, you will do it faster and you will be able to do it more effectively. A lot of academics and a lot of policymakers and a lot of data scientists working in this field shy away from working with the government because working with the government is hard. However, if they actually want to impact change, working with the government, making it an ally, I think is very, very critical. Maha Rehman: And just getting down and doing that work it takes to actually see an impact, or your results actually being implemented, or your results shared with the policymakers, or helping them actually implement it, I think that’s the real gain at the end of the day. And so a lot of policymakers, a lot of academics, don’t really want to do that. And I think that behavior towards “who should we partner with and who should we not partner with” really needs to change, because when there’s a lot of hard work going into it, you also want to see those results actually being implemented. So I think that’s something that really irks me about who we choose to partner with and how we actually scale the results that we see out of an experiment. Rob Wiblin: Cool. Okay. Well yeah, obviously you’ve got to get back to continuing to scale up this stuff and I guess push on with the vaccination drive. I guess… yeah, what’s next for you? Are you going to keep using the skills that you’ve built here to actually deliver services in Lahore? Or maybe once enough people are vaccinated you’re going to go back and do more academic work? Maha Rehman: I think any work and experiment that I undertake will always be in partnership with either a private sector partner or a partner in the government. There are a lot of research questions that I’m interested in, in public health, in energy. And so while seeing the end of this pandemic is a very exciting thought, I can’t wait to get back to the other research questions in areas that I was interested in. How can we effectively reduce losses in the energy sector in Pakistan? And that’s an area that very few people have access to. That’s an area that’s not been researched. Maha Rehman: And I think being a female academic and somebody who works with data, I bring a very different perspective to the table. And so I can’t wait to get back to the questions that I was already working on. But before that we will need to increase the vaccine uptake in Lahore, ensure more people wear masks. And while we are working on these two questions, also ensure how the health systems be better prepared for such pandemics in the future. Rob Wiblin: Well yeah, thanks so much for being willing to jump into the field and fill a gap, and I guess try doing something difficult and new that you hadn’t potentially done before and actually take a risk in order to try to save lives. It’s super inspiring. I’m not sure that I personally want to go on and take something quite as ambitious as what you have, but maybe some people in the listening audience might be inspired to take on something, a really big challenge. My guest today has been Maha Rehman. Thanks so much for coming on the 80,000 Hours Podcast, Maha. Maha Rehman: Thank you so much. Thank you. Rob Wiblin: We’ve got over 250 hours of content across all of our episodes now. People still regularly contact me to say they’ve found the show and managed to listen through every hour of it, which is mad because it represents 11 days of constant listening without sleep. And while I love the sound of my voice but I think even I would be sick of it after that. Of course, most folks aren’t quite as enthusiastic as that, or if they are, they’re going to struggle to find 250 hours to spare any time soon, which I couldn’t be more sympathetic to. So we’ve created two compilations of core episodes of the show that between them will introduce you to most of the key concepts of 80,000 Hours and the effective altruist research project, as well as the most important empirical facts that guide what people actually decide to do. We chose the episodes we chose such that that they’d mostly include interviews that are favourites of ours and listeners based on your feedback, so that they’d cover a wide range of topics, and so that they’d deal primarily with the things we think are most valuable to know. You can get either set by searching for effective altruism wherever you get podcasts. The first, which already existed, is called ‘Effective Altruism: An Introduction’. It’s focused on how to think like a global priorities researcher, so you can understand the world more accurately and make better decisions about how to improve it. We’ve tweaked that one to include more coverage of global health and wellbeing work which was previously a bit lacking. The second, which we just released, is called ‘Effective Altruism: Ten Global Problems’. It aims to brief people on the most important things they need to know about some popular problems the effective altruist community is researching and working to solve. These ten episodes cover: The cheapest ways to improve education in the developing world How dangerous is climate change and what are the most effective ways to reduce it? Using new technologies to prevent another disastrous pandemic Ways to simultaneously reduce both police misconduct and crime All the major approaches being taken to end factory farming How advances in artificial intelligence could go very right or very wrong Other big threats to the future of humanity — such a nuclear war — and how can we make our species wiser and more resilient and one problem few even recognise as a problem at all The selection is hopefully ideal for people who are completely new to the effective altruist way of thinking, as well as those who are familiar with effective altruism but new to this show. You can find it by searching for effective altruism in whatever podcasting app you use, or you can go to the webpage for the series by going to 80000hours.org/ten. And there you can also find information about all of the interviews that we’ve included. It could make sense to start with either of these compilations, it’s just a matter of which theme interests you more. If you know someone who might be interested in learning more about the ideas we discuss on here, maybe let them know these series are available and a great place to get started. Again you can find them by searching for effective altruism wherever you’re listening to this show. Alright, the 80,000 Hours Podcast is produced by Keiran Harris. Audio mastering by Ben Cordell. Full transcripts and an extensive collection of links to learn more are available on our site and put together by Katy Moore. Thanks for joining, talk to you again soon.