Latest / Computer Says Maybe / The Collective Fight for Our Humanity - REAL Resistance
Transcript
- Alix: [00:00:00] A mere handful of tech companies have enormous power over the systems we rely on, but the conversation about how these companies impact people are really vague and superficial. We need to hear from experts and advocates focused on understanding what's actually happening, knitting together a global understanding that empowers resistance. Welcome to Real Resistance, a series produced in collaboration with Real ML, the network doing just that. We're talking with members of that community who have the receipts, the stories, the research, the insights to build a future that isn't governed by tech billionaires. And in this conversation, we'll be reflecting on all the work we've learned about throughout the series and what it takes to build movements across a global community of practice with two people who work hard behind the scenes to make Real ML what it is. Anna: I'm Anna Bacciarelli. I'm the executive director of RealML, and I also research AI and human rights at Human Rights Watch. Shazeda: I'm [00:01:00] Shazeda Ahmed. I'm a postdoctoral researcher at UCLA, and I've been a twice co-chair of RealML. Anna: So RealML is a community of practice that really tries to connect researchers, practitioners, anyone really working on issues of AI and tech accountability around the world so that they have the skills, the knowledge, and the contacts to get their work out there in the world For about six years, we've been building Real ML as a community. Our aim is to create a global movement that can push back against the sort of injustice, the lack of accountability, the lack of transparency, the power inequity that comes from AI and algorithmic systems in particular. We mostly do this at the moment through kind of intensive workshops and skill shares. So all of the people that you've heard from in this series, they've been to our workshops, they've developed their work there. Some of them have met through the workshops as well and gone on to form [00:02:00] partnerships. I like to think of us as a behind-the-scenes sort of connector. We're your fixer when you are looking for other people who work on AI accountability issues that can help solve the problems in your own work. Alix: How has Real ML been different than other spaces you have been convened in in your work? Shazeda: Part of why I got involved with Real ML first as a participant, so the format is that you apply with a project that's in progress. It shouldn't be completed. It shouldn't have been at the point where people have read about it and critiqued it, but it can be an early-stage idea or writing or a campaign or a film that's partially baked. I really enjoyed being part of a community that was reading our work and giving us a ton of great feedback, and that was many years ago when Vidushi Marda and I were working on emotion recognition technology in China. I loved that, like, six months later, there was a session where we got to all talk about projects that had been completed and how they had been received in the world so that it wasn't just that you dropped in for that initial feedback [00:03:00] round, and of course, made amazing connections, but that we stayed in touch and we got to celebrate each other's wins. And then I was asked to be, like, a co-chair designing the next few conferences, so we had one workshop virtually, and then last year we had an amazing workshop in Mexico City in person. And part of why I was so delighted to get to keep being involved in Real ML is I'm always talking about how do we make people's perspectives on AI and critical AI more global. And honestly, many of the spaces that I've been in that do that do it in a really perfunctory or sometimes performative way that doesn't mean actually sustaining communications and community with people who are living all over the world and working on the specific issues in their local environment. So that's been one of the reasons I really, really enjoy it. We're not asking people to shoehorn their projects into some kind of frame that is only recognized in the West, but we're also learning from each other. When we put out the call, right, we put out themes as well, and [00:04:00] we read the applications thinking about not just are you a fit for the theme, but are we making sure we're not having people whose projects are super similar because we want there to be a kind of diversity and for everybody to be maximally exposed to things they might not be familiar with. We think about is this somebody who's already very well-resourced and fairly well-known? That's great. We're not looking for that here. Um, so yeah. I mean, that is a big part of why this exists, right? Good for you. Um, yeah, if that makes sense, it's that we're trying to maximize what we get that's going to kind of upscale everyone and, like, level everyone up, and then of course organic connections form between them. Part of how we designed this event is that even when it was virtual, there's a fair amount of time for social space, getting to know people, and finding out what your work has in common. It's always really sweet to see people end up collaborating because they met at Real ML and even just, like, getting the inkling of that collaboration [00:05:00] from being at the workshop. We even tried when it was virtual to pair people up where we saw connections between their work that they might not have known otherwise and just sort of make time for them to meet each other and talk. Anna: I think just building on that, I had kind of expected things to change a lot more than they had in the projects at Real ML over the years, and I think there's something in it for me that there is a continuing thread. And from my perspective, it's Shazeda: a sort of representation of how we're seeing AI Anna: be deployed in the world, and it's just sort of, like, worse all the time and more prolific and more harmful pretty much constantly. But the themes that we're seeing around, I don't know, like the most egregious use and, like- Military and, and borders and policing or, you know, stories of discrimination and bias, AI used to kind of like colonize and reinforce power structures and all of this is like there is a recurring narrative that I think floats through most of [00:06:00] the projects that come to Real ML, and it is around that entrenchment of power structure. A lot of the work that we seek to support is about really who is hardest hit, who has most to lose from AI, and how can we help tell their story, what are they experiencing and, and how can we position research and really like support advocacy that comes from the perspective of the disenfranchised, whatever that looks like. Alix: How did you settle on the themes of the year? Like, is there... Like, what's that process look like? And then how do you see humanity benefit and militarization kind of interconnecting? 'Cause it is really interesting that you all picked those themes, and I mean, it's unsurprising to me that you all were, like, on the nose about, like, exactly what, what, um, this group should be talking about. But yeah, like, how did you, how did you pick the themes? Shazeda: The first one, humanity, when Anna and I were very early starting out to set up, you know, the programming, I was like, "Anna, this is gonna sound kind of ridiculous because we [00:07:00] do such specific work with this community, but hear me out." humanity. This is showing up everywhere in these tech companies. Nobody's actually talking about what it means. And then Anna, of course, amazing, is like, "Yes," and like, "Look at... You need to talk about, like, the UN and the way that they're using that framing while also giving AI companies a complete out." And so we were like, there is something happening there. We know it might look odd to people in our community who expect something more like accountability, which all of our projects in some way deal with, but we're gonna like, okay, this one's gonna be a bit odd, but we can explain it to people and situate it in history and, you know, decolonial projects, right? The work of people like Frantz Fanon, who talk about how humanity is denied to Black people. Like, all of this is intuitively helpful, and then we see how people are trying to reclaim that word, right? Like, a v- variety of philanthropies that fund tech justice work have created this humanity AI, like, megafund too. So we already kind of saw that some of these words that are taken for granted because of their subverted and perverted [00:08:00] uses are now trying to be wrested back by people who care about human rights and, you know, civil liberties. For benefit, it was the same thing of the humanity and benefit thing always go hand in hand of I promise this is benefiting all of humanity. Why would you wanna get in the way of benefiting all of humanity? It makes me think of Marc Andreessen's techno-optimist manifesto where he says standing in the way of creating these technologies is tantamount to murder because they could save so many lives. Really, that is the kind of logic we're in right now, where resisting technologies that we know are destroying people's lives is, quote, unquote, "murder." You know? Like... And then with the military piece, it rounds it out because that prior narrative is what's used as cover to just militarize these technologies and use them in all these contexts. We wanted to bring it back to, again, kind of the example I gave with Abeba and Riya's work and other people's work of how did those prior narratives enable these horrific uses of technology that people feel like they were completely blindsided by when they weren't actually at [00:09:00] all, that these two projects were continuing in parallel all along. Anna: We have these, you know, kind of recurring themes, d- discrimination, accountability, whatnot, and Shazeera was like, "Oh, hear me out, but I have this, like, edge case." And I was like, "No, this is the one. This is the one." Because at the time I was working on, lucky me, the UN Global Digital Compact. Thrilling. Um, but it's just, like, seeing the co-option of The UN and, you know, other places that set the norms and standards now, but also like this, this language has got to be resilient like 50 years in the future. So it's got to make sense, right, in your, you know, international treaties. This language of humanity, benefit, neutrality, efficiency, cleanliness, that bears absolutely no relation to the kind of brutal, inefficient, and often violent systems. Yes, it's policy language, but I think the language really, it shapes your conception of [00:10:00] how you imagine these technologies. It stops us resisting. Policymakers who often don't understand these systems are like, "I can't disagree. I want something to benefit humanity." And then, you know, in the meanwhile, I think this, this policy language has really kind of taken hold, captured imaginations of people within the tech industry that frankly mean well, but they're just not exactly going down the right way to truly benefit humanity. And I should say that we do have some people from tech industry who are part of Free the ML. You know, it's not just the kind of human rights see people like me and academics like Shazeda. We do have our corporate tech friends involved, but like those are people that are r- are really asking serious questions about accountability and power and what that means, both within their organizations and within the structures that they, in industry that they operate in, and is, you know, it like I think not the kind of people that would come out with the cleansed language of humanity and benefit. Alix: Yeah. And I think [00:11:00] it also connects back to that question of abstraction and like that question of what is global. Like to some people, global is access to a large market, and to some people, global is a pluralistic set of interconnected contexts that all require a different set of politics and research to understand what's happening. I think the latter is where change and meaningful uptake and like wrestling with like the true implications of some of these technologies sits. And in the former, it feels like it's a, like it feels like global as a large market is the place where people who want to steamroll those political questions sit. Um, and I think humanity I don't know. I, I guess until sh- I feel like it was you, Shazeda, who, like, w- eventually, like, started saying humanity all the time to me over, like, maybe a six-week period when we were preparing for this. And then now anytime I see it, I'm just like, it's such a political project in a way that is so subversive 'cause you don't immediately think of [00:12:00] it as such unless you've spent some time thinking about why people use that word. It's like, well, who are we talking about when we say humanity? Well, let's... We'll deal with that later, you know. Like, or benefit. Like, who's it actually gonna benefit and how? Oh, don't, don't be so precious and specific. Like, you should just let it all rain down upon us all, the, the benefits of these, you know, world builders. Anna: Shazeda's work on safety as well is another. You know, there are these- Yeah ... keywords, and they come, and they're buzzwords, and everyone's like, "Do you want a society that is unsafe?" And you know, it, you sound ridiculous trying to argue with this on an abstract basis, but that's a win, right? That's a win for them. Yeah, Shaz, I wonder how you see it as, like, the word safety and AI safety sits in this sort of trio of benefit and humanity and safety. Shazeda: AI safety is a field and, like, an epistemic community. This is their lingua franca, right? Like, they use this [00:13:00] phrase ad nauseam. You know, my coauthor, Andrew Smart, has also noticed another one that we're, like, tracking, which is make AI go well. Okay, bar's really low. Like, great. Um, but with all of these words, I was just talking to Adam Becker about this the other day, where I was like, "You know, in our work, things like safety and humanity and benefit and recently progress studies, these are designed for people to slip right past them, but I want them to be spikes. Like, I want them to catch your clothes and make you stop for a minute and ask, 'Wait, what just happened?'" And that was what we were preparing people to do at Real ML. People who already have these really incredible instincts, like, born from years of doing research, is saying, now we're a community who are all watching out for these things that we have noticed together, and how do we get other people to care about them? How do we get other people to notice them, too? But yeah, with my work on AI safety, and I've talked about this on this podcast, too, right? It's just the complete opposite of what we do at Real ML. And You know, the way that this has been portrayed in [00:14:00] media is that there's like a culture war between, quote-unquote, "AI ethics" and, quote-unquote, "AI safety," I've always said is not a one-to-one match. Almost no one who came to Real ML is like, "I'm an AI ethicist." That is so vague, and part of what I love about their work is it's super specific, right? It's, "I'm a legal scholar working on this particular thing. I'm a historian who can, like, archivally prove to you that there's a history of surveillance in Mexico. I'm a journalist who, you know, has investigated these algorithms that are denying people in Kenya healthcare." Like, I want us to normalize that specificity rather than having people feel like they're choosing a camp that doesn't really exist. Alix: Yeah. Yeah. Anna: Shazeda for president. Alix: No, 100%. I mean, I feel like that noun thing of like I'm a historian or I'm an anthropologist or I'm a cognitive neuroscientist or I'm a... is something that I think really strikes me about Real ML as a space is it's multidisciplinary, so, like, everyone has a different noun, but all their nouns [00:15:00] come with this, like, incredibly deep specificity. And, like, I think it's actually one of the most fun things about the event is any one-on-one conversation you have with someone, you can just keep asking them questions forever and, like, keep learning so much 'cause they're all such deep experts. And I feel like the cost of attempting to scale some of the spaces and make them global on the negative or, like, harms of these technologies, the cost has been abstracting out what that would look like in a way that kind of neuters everyone's field specificity in a way that I think then makes it really both boring but also unmoored from any specific political question that could actually lead to change. And it really feels like Real ML is the place where You can get a foothold in a context on an issue with an expert lens in a way that might convert into actual traction and clarity and understanding about, like, what needs [00:16:00] to happen. And I feel like that's just such a different... Like, I don't know any other space that has managed to do that in a way that's also interdisciplinary. Anna: I'm glad. I'm really glad to hear that. Good. Um- Great. Alix: Mission accomplished. Anna: Yeah. We can shut down now. Yeah. What you're saying around apolitical language seizing control really chimes with me because I think that having run Real ML for the past six years, it's like our world has got more polarized, our field has got more politicized, and I think that's just because there is more technology everywhere, but it's also being used to, you know, control, repress, yeah, like discriminate, whatever. This is inherently a political act. I always say that, like, you know, the choice to put an algorithm in something to automate something, that's a policy decision, and that, that's an active choice. Even if it's made in the name of, like, cost saving, efficiency, whatever, there is a policy choice there. I think that, that we've become a much more politicized [00:17:00] movement as a result of that. And so it does mean that we have to be kinda careful about who we're sharing work in front of. And I think, you know, last year in, in Mexico was wonderful in that we all came together. One of the participants said, "I like being in spaces where I feel like I'm not the most radical person in the room." And everyone was like, "Yes, I'm not the most radical person," which did lead me to think, like, well, who is the most radical person in this room if n- if no one thinks that they are? But also, like, it shouldn't be radical to ask questions around why are you doing this? Does it work? This isn't for me. How do we push back? You know, I think some of these, like, concepts often framed as radicalism, whereas actually like... I mean, sure, I'll lean into that description, but at the same time, it is much more politicized because it is specific as well. I mean, it's also resource-intensive work, and I think that, like, yes, we're supporting people to partner, like, to make progress with their work and to kind of put things out into the world. [00:18:00] But at the same time, we work in little silos. It can be quite dogged. Like, these investigations can take ages, years in many cases. And so just trying to give people a sense of community, that they're part of something, that you have a peer network that is globally dispersed, but they've got your back, and you can learn from another, one another. I think that is like, yeah, it's a bit woo-woo, but also I think it's really important. Alix: I don't think it's woo-woo. I think other people wanna frame it as that. I don't think you should accept that frame because I feel like that's part of all this. So we went through four different clusters of themes, and I'm thinking maybe we just go through one by one and talk a little bit about them, 'cause I know that you all were involved in helping figure out like what the themes would be and like how those-- these particular projects would kind of interconnection. I feel like this is where the interesting bits are, is when we get into specific [00:19:00] thematic sort of political questions rather than the kind of abstraction of, of infrastructure around them. And I'm wondering... So if we start with Claudia, Alia, and Yannick, each of them is working in some way on the intersection between large language models and platform power as it connects to marginalization and language and kind of the globalization of some of these concepts. And I feel like-- So just to give you guys, I mean, I know you know these projects. But so Claudia was talking about essentially like if chatbots are mediating public interest news and information, like there's like tremendous number of political questions about what, what that means. Alia was talking about essentially the difference between translation and localization and how, uh, you can like technically translate something into a language, but actually the ability of a, a model to be able to output something that resonates and sort of feels actually properly localized is basically impossible, and actually the [00:20:00] effort itself is kind of a colonial exercise in saying these companies should be able to do that. Yannick has this like builder attitude and is trying to engage young people in how to have an educational journey that's custom to them in a way that also connects to kind of localized dialects, but also kind of has a political project behind that about figuring out like, what does it mean to engage with these models? How do you get something out of them without essentially training them to own your culture in a way that is disturbing? And so obviously all of those connect in interesting ways, but I'm wondering, like when we were selecting these themes, was that an obvious cluster to you guys already? Or like how do you see the kind of role of language and information ecosystems connecting to, to, to ReAL ML's work or any kind of reflections about how those projects, um, sit together? Anna: I mean, I think all of them really engage with generative AI and LLMs through that lens of positionality and challenging power [00:21:00] structures. You know, when I think about Alia and, and Gabriel's work on kind of decolonizing LLMs, that was like, I think 2023 they published, which was really the start of the Gen AI boom. And that was when we're starting to ask questions around English language dominance and all of the repercussions from that. And then Yannick and Claudia, they were on our workshop last year in Mexico, and I think as you described the kind of-- Yannick is sort of, he has an inside perspective, or he's based within the sort of tech world trying to think about how he can use his influence to sort of decolonize, for want of a better word. And then Claudia has this more bird's eye perspective on You know, how our information ecosystems are shifting, how people's perception of what is and isn't real is shifting. Shazeda: Yeah, I think something that we saw as a through line in these three projects as well is the let's get under the hood of the systems we're talking [00:22:00] about, but also talk about the cultural context, right? So with Claudia, it's let's test all these different LLMs for how they produce "news" or like how they essentially plagiarize news, and then it becomes misinformation, and there's no real attribution to the sourcing. But then let's think about what all the downstream harms of that are for news consumers, and especially for people who are not privileged enough to realize that this is what is happening in their news ecosystem. And at the same time, with Gabe and Ollie's work, it's also just like instead of taking these models at face value, let's think about all of the different languages that are being underrepresented. And I was thinking back to a lot of these projects. When I think about what am I looking to to understand the future of technology, even within the next year, the two things I think of are artists. Artists often can kind of presage where we're going, and RealML, right? That work has since influenced a lot of other work on LLMs and [00:23:00] "low resource languages" and helps people understand like what, uh, the scholar Seyi Olojo, who writes about this issue in Nigeria, calls epistemic violence, right? That this isn't just a it doesn't work in your language, what an inconvenience. It's also about how-- I mean, and then they, they talk about it a little in the episode about what if in the UAE where being queer is, uh, considered a crime, like LLMs reflect that, and people aren't getting information that they want that would make them feel supported. And then with Yannick's work, it's really cool. Like he was fine-tuning models for Bambara-speaking communities, right? Like again, this hands-on approach while still being very rooted in a community and a place is one of my favorite things about how these projects at RealML work. Alix: Yeah. And I think also that chain from kind of high level concepts like epistemic violence into very specific use cases, and I feel like that's also, to me, one of the values of RealML, is that like getting contextually specific is helpful. But like I feel like the research projects themselves [00:24:00] oftentimes bring to the fore stories that make these topics and issues so much more salient, I think, than they are in more academic spaces where there's like a love and a joy of saying a series of linked concepts that like if you're not like in it, you don't know it or understand it. But then when you get concrete examples, it immediately becomes clear what those like deeper academic discussions are about. Let's move to automation and governments. I feel like this is another one that tends to run abstract, but then when you look at the examples of the real ML projects that connect to it, or at least the ones that we talked to, um, for this, like, cluster, it gets so specific and consequential and real so fast. Just as a reminder, Purity and Gabriel talking about proxy means testing in Kenya. Um, Maria Pilar talking about how judicial environments and courts in Latin America are using generative AI in the context of proceedings, arguably to make them more efficient with [00:25:00] backlogs, but obviously having, like, loads of implications for what it means to run courts. Um, and then Divij Joshi talking about digital public infrastructure and the kind of histories and futures of how governments are using digital systems and digitization to nominally, like, provide better services. But also there's obviously all these other political, economic, and, like, authoritarian implications when we're talking specifically about India, um, although he talks a little bit more globally. So reflections on this, like, cluster. Anna: I mean, I think similar to other episodes, this shows how people build their research with a nod to one another. So, you know, Divij, understandably with Aadhaar, has been documenting this for a long time. And then, you know, the likes of, of Gabriel and Purity and Maria Pilar, they've referenced kind of Divij work through their work You know, when Aadhaar was like a sort of experiment a few years ago, now this is digital [00:26:00] public infrastructure is a thing pretty much everywhere. And so I think the rollout of that globally is notable, but then also like the ways that new technology is being applied. So what struck me about Maria Pilar's work is LLMs just absolutely polluting the information ecosystem in the most critical circumstances, like the criminal justice system. You want, you want judges to have the right information for the law to be applied kind of accurately and fairly. And if that, the kind of like cornerstone of like justice in society is being eaten up by LLMs along with everything else, like extremely, extremely worrying. It's about making this real. So how does this impact people's lives day to day, whether they're trying to access like healthcare services in Kenya or they're, they're kind of going through the courts in Argentina, for example. Shazeda: When Maria Pilar talks about how the court's use of chatbots is a kind of efficiency theater, that applied to all of these projects, [00:27:00] right? The creation of a digital welfare state is often advertised in media as being this big revolutionary thing, and yet there are so many people who get left behind, fall through the cracks, don't get to have access to benefits that maybe weren't necessarily that much easier to get before, but there was a bit more accountability than there is now. And then in thinking about Gabriel and Purdy's work, it's just incredible, right? Like, it's a mix of Purdy having this really good instinct of understanding, oh my goodness, right, there's this algorithmic system that has the potential to widely discriminate against people because of her understanding of how that worked in Rotterdam, right? This is how I love when Real ML projects work is something's happening in another locality, but what are the actual consequences when it's ported to someplace with a different culture, a different government, different priorities? And I loved that their project wasn't only an audit essentially in trying to figure out how does this system work, but that they were incorporating like social media that people who were experiencing [00:28:00] this proxy means testing system and saying, "Oh my goodness, now I can't pay, I can't afford healthcare and I'm being turned away." A mix of that let's get under the hood, but also the bottom up, what are people saying about this and how do we incorporate their perspectives? We wanna just see more stories like that and less of the headlines about isn't this such an amazing technology, right? Because that's the kind of thing that as they mention, right, these are pressures from like the IMF and the World Bank in terms of like giving loans just to have this other narrative that cuts through the feel-good ICT for development narrative. Alix: I feel like Divij digs into that really effectively, um, especially on like the story of like, let's not talk about when it doesn't work, let's talk about what might happen when it works well, which is like such an irritating frame 'cause it's like, but how do you make it work well when we can't have honest conversations about when it doesn't work? Um, I don't know. It feels like also a very common pattern [00:29:00] in technology broadly is there's like, you know what Abeba says, the potential benefits next to real harms is a very like kind of bizarre juxtaposition that becomes this like story that tech evangelists tell us that like, "Don't worry, it's gonna get be- this is the worst it will ever be." So like it's... And if I hear that one more time, yeah, I might scream. Shazeda: It reminds me, something I think back a lot, um, when I read stories like this is the journalist Hal Hodson had once mentioned, you know, there was this big Economist article many years ago about China and technology, and he was like, "They used the word could more than 40 times." And really I'm like now my, my ears are pricked up or I'm looking for like how many of these kind of future potential words are you using versus just explaining the reality that is happening before our very eyes. And that I think Real Mel again cuts through that to say you don't need to read, quote-unquote, "news" that's really just a projection of a future that very likely won't come. Here are the accounts that you absolutely need to hear. Anna: And I think [00:30:00] a lot of the time the governments and the tech companies rolling this stuff out, they aren't doing the documentation. That's a problem. And it's... I, I sort of like sometimes feel a little bit chippy the fact that like it's left to these people to really go through, you know, try and discover where harms are happening, how they're impacting people's lives. But I think that the reality is that, you know, evaluations or whatever they look like, they don't capture this. They don't capture people's day-to-day experiences, and so we're gonna have to rebalance that. And yeah, RealEMAL is, you know, trying to address that in some way by saying, "Well, like, look, here's what you say you're gonna do." And then, you know, most of these researchers are going, "Well, yeah, but here's the reality." And can we bridge that? Sometimes yes, but a lot of times no. So, you know, I think using the evidence and making decisions based on that. Alix: Yeah, I feel like it goes back to abstraction Like there's a desire from people that wanna continue to advance these technologies and like benefit from an uncritical adoption of them at [00:31:00] scale, that they would like for us to have conversations, um, about evidence and research at this level of abstraction that like both takes out the kind of people and context that would probably lead to people being enraged when they actually see some of the real stories and real effects of these technologies. But also to keep the conversation at what feels like a non-political layer of evidence and research where it's like, "Oh, we're just doing an evaluation of a model using this like large scale set of tests that aren't really about how a person engages with these technologies, but instead is kind of this abstracted out quantitative metric that we have come up with as a way to grade our own homework." And I feel like there's a political desire to return to abstraction because it's a safer space for people that want to talk about could rather than is. And I think if we're talking about abstraction or like a desire to make this feel, quote, unquote [00:32:00] global, you know, the other frame for global is colonial, um, 'cause globalization is called globalization for the people that get to go wherever they want and extract whatever resources they want for projects that have the air of engaging, you know, quote, unquote humanity. Getting into the specifics, there's a couple of projects that we talked to that were focused on resisting AI colonization. So, um, looking specifically at Angela and Kanna's work, Paula's work, and Fernanda's work. Paula talking specifically about the histories of resistance in Mexico as it relates to extraction of resources, um, by things like data center infrastructure and like what it looks like. She uses this phrase in her work, mechanisms of dispossession, which is now like living rent-free in my head, and I think about it a lot. That like there's these common echoes historically of mechanisms of dispossession that connect to colonization and now connect to how big tech's engaging with Mexico. And then Fernanda talking specifically about facial [00:33:00] recognition technologies and kind of colorism in Brazil and, and what happens when a technology like facial recognition is deployed in a society that has these histories and sort of social relations of racism. And then Angela and Kanna talking a little bit about the labor and workforce who is actively constructing value for AI supply chains, and then having all kinds of mental health effects being economically and politically disenfranchised in terms of like how low can we drive down these wages? How little labor protections can we provide? How much can we prevent organizing? And obviously, Kanna and Angela have been working on using organizing as a means of resistance. So all these are nationally specific, politically, historically rooted assessments of like how, not just saying AI is empire, AI is colonialism, but instead saying people's reaction to these systems and ways that they're resisting is similar and like usefully similar when thinking about like decolonizing efforts [00:34:00] historically. Shazeda: One thing I really loved about this is that everyone was able to situate their work in a historical context. And this is again, part of what is so great about this workshop is people come in knowing, you know, this is an international group of people who don't have the history or like the nitty-gritty understanding of what informed this technological intervention that exists today. And I think so much work in our field, whether it's from academia, civil society, wherever, is so focused on the present when in fact there are these longer histories and it makes so much more sense that you see where, you know, these technologies originated when, for example, in Paula's work, right? She talks about mining in Querétaro and how that was always a place that was extracted by colonizers. And so that logic predates this moment where data centers are being built and people don't have access to water. It was just immediately clicked, right? In thinking about this is a fresh spin on this issue I've been reading about for quite a while now. And the same goes for Fernanda's work in talking about the history of [00:35:00] slavery and lack of reparations or real protections after it was abolished in Brazil. In thinking about talking about that in São Paulo, Black people who understand that facial recognition will be inaccurate and discriminate against them and can lead to false arrests. Really liked that she talked about, you know, I'm going to name the person who, who was falsely arrested multiple times because the technology didn't work. I don't want them to just be a statistic. I want this to be a real person that you understand, you know, has-- is facing all these consequences from this happening. And just talking about how people who understand how that technology works are afraid to go to these cities, right? That like these aren't technologies that just passively operate without having a chilling effect They shape how people move through space and time and what choices they make and what choices they're not allowed to have. But yeah, I thought there were so many great things about this too that also reminded me, because we did Mexico City as the location for the past year, we tried to take more projects that were [00:36:00] from Latin America. And I think that's something that's evolving as we develop and, and can go from our COVID-19 era of doing this workshop virtually to now being able to do it in person, is making those connections of between people who all live in one region. But of course, these are, like, major countries, huge populations, big cities. How can we sustain connections and, and have people see, like, something that's happening in Brazil and Mexico are related? This episode really brought out that kind of Latin America component Anna: of, of Real ML this past time. What struck me is that there's, like, some core methods of resistance that I think come through around documentation, especially with Paula's work. Fernanda kind of using legal mechanisms and instruments to achieve accountability ultimately and to shape the law in a way that it is, that it is truly equitable. But then Kana and Angela's work in, in labor movements and kind of resistance through unionization. I think that the [00:37:00] core means of resistance to new technologies are actually pretty old, much as the, the systems of repression are pretty old. And so kind of borrowing from historical movements, you know, is something that I think we can all really learn from. And it is also notable to me that all of the guests in that episode and in a lot of the series, and in fact a lot of the people involved in Real ML, are women, and they're women of color. And they're coming from a place where this is really meaningful. This has history. It has personal history for them through their families and their communities. And so on the one hand, you know, frustrating that it's left to them to tell these stories and push back against like the almighty power that is Silicon Valley, you know, AI hype. But on the other hand, it's empowering, you know, telling your own story and crucially kind of leading the resistance to that kind of, you know, male-centric whitewashing of AI. That for me is really [00:38:00] powerful. Alix: Yeah, and I think that, um, connection with who participates in Real ML with the capability that that means that Real ML has that other academic spaces don't have, I think is really important. There's a lot of people that talk about global work, and then there's organizations that, like, actually do the work to make those spaces possible. I think that, like, a lot of people don't realize how hard it is to do this in a way that doesn't end up just being a bunch of, like, wealthy elites from a lot of countries, which is also a very different type of global, and I think that the results is that you actually get analysis and insight that is contextual and not in like a condescending way, because I think sometimes people frame, "Oh, that's nice. You've done research in a place. That means that you're now an expert in that place and not able to make insightful analysis on [00:39:00] global trends in these topics." Um, leave that to, like, the European or the American who gets to make sweeping generalizations based on their limited vantage point. And so I feel like that point about the backgrounds of the people that come and like the ethos, and it's not just about projects, and it's not constructed as like a contest. It's like it is, it is properly a community, I think is a really, yeah, a really important part of what makes it, what makes it work. So thinking about the intensification of surveillance, so looking at Mariel and Crofton's projects, where Mariel's looking at the history of the surveillance state in Mexico and, like, the infrastructure that's been built up, um, in Mexico to allow for surveillance, and then Crofton looking specifically at the kind of marketplace for surveillance technology and, and how that has completely changed in the last few years or just intensified, and the worst [00:40:00] parts of it seem like they've just gotten worse. But reflections on, I know surveillance is something that cuts through basically all of the projects at some level, although it looks a little bit different. But in particular projects, these two projects, sort of thinking specifically about nation state surveillance and digitization of the ability of the state to see, reflections on this one? Shazeda: Yeah, one thing I'll say about this one in pairing the two projects together is their method is kind of similar. Even though Mariel is a historian and she's going through archival kind of records, and Crofton is sitting with a lot of documents from today to try to figure out how do these surveillance networks work, they're both kind of doing the same, like, we need to read reams of documents to figure out systems that have been occluded or obfuscated, how do they really work? So it's different from some of the other projects that are more, like, interview based and, and still, like, a really important methodology that has become more common even since Crofton did his Real ML project years ago, which is kind of a reminder of, like, how Real ML [00:41:00] has just a finger on the pulse that shapes where this research ends up going. Anna: I mean, I th- I think, you know, it just shows that there's, there's always a customer for surveillance technology. And, like, working in this world of AI, and there's AI hype in our field as well, right? So if you're, you know, if you're working on, like, the next thing, then you might get more funding, you might get more speaking invitations, whatever. Stuff like mass surveillance seems like they're like, oh, you know, like how very 2010, but actually that is still a huge problem and it's underpinning, you know, it underpins, as you said, so much of our kind of new AI algorithmic in- infrastructure. A lot of it is just purely foundationally mass surveillance with bells and whistles on top. So I think we have a duty to remind that tech persists and that, you know, there is a market for it. Like Mexico is one particular example, but this is a global industry and at this moment [00:42:00] in a kind of political history as well, I think we have a lot of lessons to learn sort of from history as Mariel dives into, and then, you know, Crofton kind of looking at the current day market and sort of exploit, you know, how companies are exploiting kind of holes in that. There's always gonna be a buyer for this kind of shit. Alix: Yeah. I think it's a really good point that like the market exists because there's demand, and that demand is coming from governments, and those governments are changing in terms of their willingness and appetite to surveil, and that the norm setting from certain nation states, like I think about Israel as like an edge lord nation now. Like basically like there's a couple of countries that are like pushing the boundaries of norm setting, and then other countries are very quick to follow and be like, "Oh, well, if that's now normal, then I guess, um, we'll start doing that," which means that marketplace is only gonna get probably- Bigger, but also maybe give us foothold into advocacy that might not have otherwise been possible. Like, I think the, [00:43:00] the moment of like Anthropic as an example, and I know this is a military context, but like that gives a foothold to say, "Oh, actually this is not good," or, uh, a- a- where- whereas before if you had said, "I don't think algorithmic decision-making and probabilistic decision-making should be included in decisions about lethal force," it's really hard to talk about that in the abstract without AI, which is kind of reinvigorating and, and it makes it feel frontier, both in terms of the technology, but in terms of the political and policy questions that we get to ask. The fact that people hate AI is I think maybe a useful device, 'cause they don't hate their phones, and I feel like the surveillance conversation has oftentimes been a consumer convenience conversation of like, "Oh, don't do that because it's exposing your, quote-unquote, 'private data'" or something, and it's made it a consumer kind of decision. And I feel like the predominant answer to that is, "I don't care enough about the political harm of private data being [00:44:00] exposed to corporate or government actors, and so I'm just gonna continue along the path of being a consumer using these products." Whereas with AI, I think people are grossed out by it in a way that creates opportunities. Yeah. Shazeda: Last year was in June 2025, and the themes we chose- Yeah ... were humanity benefit and military/genocide, right? I know. Yeah. We didn't know at that time that Anthropic, a quote-unquote, "public benefit corporation"- Yeah ... whose mission is to make AI benefit humanity, and also OpenAI's having just become a public benefit corporation has a similar kind of BS mission, was working with the military. But we had conversations about these words, especially words like humanity and benefit, that everybody's eyes glaze over and you don't really think about what work they're doing, really in-depth conversations about what work is that doing in the fields that we work in? What could we possibly do to kind of counteract this? So in a way, this p- workshop and these convenings, they really prepare people for what's coming. Like, I think everybody in that room had really amazing insights that then just [00:45:00] made them ready for when this horrible news dropped. Anna: And it also reminds me, I mean, going back to the mass surveillance thing of, you know, one of the sort of contractual obligations that Anthropic were absolutely A-okay with is mass surveillance or sort of mass buying of non-Americans. And they're like, "No, no, that's fine. We just don't want the lethal weapons bit." And so kind of normalizing these quite insane proposals, I think, you know, it's just like that, yeah, it goes back to mass surveillance. And like you said, Shaz, like, there's sort of- idea that these technologies conceived and sold to us as, as, you know, they're gonna benefit humanity, whoever humanity might be, and then they're being used in, in killing people. I think that says it all. Shazeda: Also, going back to the could, right? Like Amodei is saying it's not even that we're opposed to, um, autonomous weapons. Right now, we should only use like partially autonomous weapons, but one day in the future, we will have the perfect autonomous weapon that can use AI, which is again, kicking a ridiculous [00:46:00] can down an absurd road. You know, the humanity piece also, like Real ML is exactly the kind of place where in the room everyone immediately understands why that is not a helpful framing and that all of the groups that we are focused on are in some way dehumanized or denied, you know, access to being considered part of humanity. I think about when we got to the military piece of Real ML, so the third day, the final plenary, we had Abeba Birhane and Rhea Kalluri there, and they've done work on how loads of computer vision research will refer to human beings as objects. They're already using this dehumanizing language and that quite a lot of that work goes on to then be used for like military and other kind of violent uses. Alix: I'm wondering, how do you find people in terms of participants at Real ML? Anna: Ha. Yeah. I think 'cause we started off basically through sort of invite only. I kind of joke that in 2019 we brought together [00:47:00] the, like, 30 people that were working on algorithmic accountability and, and put them around a table in Berlin in the world. That is a joke. But, you know, there were a couple missing. But at the same time, there weren't that many people doing this work, and then we expanded, you know, the field expanded. We went through a recommendation system by encouraging our steering committee for every workshop to recommend folks that they think, you know, should be part of this community. And then now we're, we're, we're pretty slowly public facing. You know, we had a public call for applications this year, and honestly, they were all incredible. If I could have taken 100-plus people to our workshop in Johannesburg, I would have, because this is all work that needs support. It needs, it needs to be... Well, to, to help get out into the world and to be read and, you know, listened to and so on. So yeah, I would say that we're, we're sort of becoming more open, but there is a tension, I think, between wanting an environment that is trusting, where people can share works in progress. There [00:48:00] might be confidentiality issues. There might be, you know, concerns they have around getting something wrong. If a project is half-baked, like are they okay with sharing that with the world? This isn't, you know, some kind of polished paper or PowerPoint Olympics. And so it has to be a really trusting environment, and to that end, we have to kind of vet people and, and talk with them and really understand what they want out of the workshop and then, and, and from Real ML in general, and if we're the right people to, to support them as well. There's this point in time where We, we have worked with so many incredible individuals, and really I feel like we've, we've quite carefully and slowly but thoughtfully built up our, our community over the last six years, where we're at a point where these people, their, their work needs to be profiled. They need to be rock stars. Many of them are not. Many of them are not famous, but they should be. And so part of our job at Real ML is to kind of [00:49:00] nurture them and help them get to a place where they are, you know, resourced and connected and, and so on. But then also to, you know, give them a platform in whatever way that we can. And so sometimes people meet each other through the events, and they may, you know, access funding that way. But then also, you know, I think through this partnership, really exploring some, you know, stories that may have been kind of underrepresented in, in some spaces is really our intention behind this series. Alix: This series was produced in collaboration with Real ML. For the past six years, Real ML has brought together people around the world working to challenge the power and inequities built into AI systems, not just through critique, but also through practice. And many of the people you hear from in this series met or developed their work directly through Real ML workshops where ideas are tested and collaborations are formed. And actually, last year, someone said soulmates were found. I've also served on Real ML's board, and it's one of my favorite communities, and I mean that sincerely. I really love any time this group of people gets together because [00:50:00] magic always ensues. And to learn more about Real ML and future workshops, you can check out the link in our show notes. A special thanks to Anna Bacciarelli, Isha Keegan-Nushavati, and Shazeda Ahmed from Real ML. And thank you to our production team, Sarah Myles, Georgia Iacovou, Kushal Dev, Marion Wellington, Van Newman, and Zoe Trout