Latest / The MonkCast / Anne Bertucio de Hollembeak on Why Trust & Safety Needs Open Source

Transcript

Anne Bertucio de Hollembeak on Why Trust & Safety Needs Open Source · The MonkCast

• speaker-0: Hello there, my name is Kate Holterhoff. I'm a senior analyst at Red Monk, and my guest today is Ambertuccio de Hollembrock. ⁓ she is the COO of Roost, ⁓ which is the acronym for the Robust Open Online Safety Tools. Anne, great to have you here. speaker-1: Thank you so much for having me. I am super excited. speaker-0: Okay, so before we dig into some of the cool stuff that you're up to at Roost, ⁓ I'd love for you to lay out a little bit of a background here. I'm sure folks ⁓ might not be ⁓ as familiar with some of the other cool stuff that you've done before arriving at Roost, which you've been at now since what, for a year? Is that right? A little over a year. Okay. All right. Well, before we dig into all that cool stuff, ⁓ tell us some of that background. speaker-1: Been a little over a year. Yeah, just ⁓ gosh, it's been a very winding journey in the best way possible. I feel like, you know, it's all roads have led to something interesting in the next one. ⁓ so yeah, Roost is only about itself 18 months old, and I joined shortly after launch. Before that, I was actually running Google's open source office, which was really great. I had come there from the OpenStack Foundation, which is now known as the Open Infrastructure Foundation. I was working on a project very much in the container security space. Google was using that project at the time and says, gosh, you wanna you wanna come over here and keep doing that work? And you know, eight years one thing led to another towards the end, was kind of having the honor and privilege of running this team of one of the companies that probably produces and ⁓ uses some of the largest quantities of open source in the world. So it was really neat. And there, I'd be remiss say if there wasn't, there was a big detour in there. May maybe not there's no detours, Kate. All roads, leap next. There there was a big journey in there around ⁓ open source security, which Okay was very exciting. I thought we were all gonna have our moment with the ⁓ log for shell and s solar winds and then AI came and took our press moment away. That's okay. The little speaker-0: Yeah. I believe it. ⁓ yeah, no, AI does have a tendency to suck all the air out of the room, I'm afraid. So yeah. Okay. Amazing. All right. So you joined Roost about 18 months ago. I want to hear about the mission of Roost. What is it that you are trying to achieve? speaker-1: Okay, they give good problems to talk about. Yes. I mean, we believe that online safety is for everyone. Everyone deserves it. But from a technical perspective, the only way we're going to get there is if everybody has the tools to do that. I think right now, if you are not familiar with the state of trust and safety tooling, a lot of it is very much inexpensive proprietary ecosystems. ⁓ it's kind of unusual to me coming into it from the security and privacy side, being newer to safety, looking at this and going, ⁓ they don't do things over here the way The rest of computing does. And I think there's an interesting history and a good reason for that. but you know, we look across networking, security, databases, you have these open source primitives that kind of hold up how we build things, how we this how we design our systems. They didn't have that in trust and safety. And so ⁓ this group of folks said, We really need that. We desperately need it to be public a public good. We desperately need it to be public technology. So let's go build it. And that's what we're up to. speaker-0: Okay. So much to dig into there. I think I first want to talk about how you're defining trust and safety because those ideas are a little squishy in my head. Like I can I trust it. I you know, I feel safe. I don't know. I I just I I w I wanna understand what that looks like. Is this like compliance y type of trust and safety? Or like how how vague or specific is it? speaker-1: It is very amorphous in the funnest way possible. ⁓ like so if you're a liberal arts student, come on over. It's it's it's there's something really enjoyable and intellectually challenging about the fact that it is it is vague and it overlaps, I think, from a technical perspective too, with privacy and security. ⁓ there's a great organization, the Trust and Safety Professionals Association, TSPA. And they've tried to build up this curriculum around it, but again, the defining the actual what is safety. speaker-0: That okay. Right. speaker-1: It's squishy, but I would say at the end of the day, it's very human centered. And I think the people who work in the field are trying to think about what are the out the best outcomes for my users? What are my legal obligations by the jurisdictions I'm in that sort of fall under this user outcomes umbrella? And then on top of that, what kind of community do I want to have on my platform or, you know, in my application and whatever I'm building? So it really runs this huge gamut of harms from, you know, your most egregious of Kate, let's just start diving right into it. ⁓ things like child sexual abuse material, terrorism, violent extremism content, exploitation, sexploitation, all the way up to things about what kind of speech might I allow on my platform, including things that are innocuous, like what topics. This is a knitting forum. I need a way to kick off all the car mechanics. You know, it it it is very broad, an incredibly broad field. speaker-0: So in terms of personas who would be interested in this, if I'm hearing that you're going to be kicking people off, I'm thinking of like developers and folks who are actually building. Is that right? Or is that are there like legal teams involved? Like who who's leveraging Roost? speaker-1: Yes. So s the safety workflow is really interesting. ⁓ I would say you think about it, there's two parts. There's the rules, which are what is allowed on my platform. And again, that is everything from what are my legal obligations to what kind of company are we, what do we want to be, to what is appropriate by user design. So, you know, if this is a platform for ⁓ small children, looks very different from this is a platform for adults. You know, there's design considerations. ⁓ I think there's also then your things about editorial and topical and how are going to control that? So we have all these things around policies, but then we need the tools to implement them. And that's maybe more where the engineering comes in. ⁓ policies tend to be for the most part, sometimes they're written in legal, sometimes they're written by teams that are specialists in the in the field, you know, trust and safety specialists. Sometimes it's a policy team. ⁓ but sometimes I think particularly now that we're in this sort of new era of who's a developer and talking about who's the builder type thing. you might find yourself being, hey, I vibe coded this chat app. Now I suddenly have to be the safety policy specialist also writing these things. Roost is really for that whole gamut, which from a design perspective is very fun because our users are everybody from I am the one-person shop who's You know, I have a real amount of users on the thing I've designed and I'm trying to keep them safe to I'm Discord, I'm blue sky, I have robust policy teams, I have robust safety engineering teams, and these are the tools I'm running. speaker-0: Okay. Okay. so you've alluded to some of the use cases around this, ⁓ very dark and very necessary. ⁓ but I I wanna finish the arc of like, you know, the who would be implementing this? Because I I feel like the the why and the who we'll we'll we'll get to it all. So ⁓ just to to round out here, ⁓ open source is a big part of your mission as well. So in terms of the relationship between safety and open source, you know, these developer concerns. How does that relate to this this sort of dis disconnect of ⁓ for understanding open source and like where where does that come in? speaker-1: Yeah. So open source as a concept is really new to this field, to the field of trust and safety. ⁓ you know, I think because historically a lot of it came out of legal and policy teams and sort of folks focused on the human aspects of safety rather than folks approaching it from the engineering side. The technologies that are possible and the things that we have in other parts of computing, the norms around things like open source, they're they're not there yet. And these concepts are brand new. So I think as Roost, it's also been fun that we're doing a lot of education as well of what does it mean to have shared software? ⁓ what does it mean to have software that is doing defense on a very sensitive area out in the open? What does it mean to have a community where people actually need to be quite private or prefer to be quite private and offline? That's been very challenging from a like contributor community building. That's been an interesting one to navigate. ⁓ so it's it's been fun. I think even just In the last year though, we've seen a big shift of how much people are embracing it. Where, you know, we're finding other people working in this space are putting their tools free online. They're, you know, embracing open source licenses for them. So I think it's caught on because when people hear, no, this is not a product differentiator, this is something everybody should have, they go, Yeah, I I agree. And I felt that for a long time. And we're like, Great, get in. It's like it's this is open source car. speaker-0: Yeah. I mean, it almost seems to intersect with ⁓ jokes that I've heard that like developers just don't care about security. ⁓ that we're in this brave new world where suddenly NPM is getting ⁓ hacked all the time. We got all these mini shy haluds. Yes, supply chain security is is newly ⁓ problematic in ways that we haven't seen historically, ⁓ because of AI. And ⁓ so it sounds like the AI is also part of this move that you ⁓ are seeing that's saying that suddenly we need to make teams aware that trust and safety is a concern ⁓ across the board, but particularly in the open source community. speaker-1: Yeah. I mean, across the board, it it I think because of that human element, you know, when you explain the use cases or what might happen on a platform you've suddenly built, ⁓ people go, ⁓ yeah, I don't want that. Or I don't, you know, I don't want to see that. And so from that sense, you know, once people understand and they have it, they're go, yeah, what can I do about that? But I think that's still really new, you know. I am grateful for all the people who work in trust and safety to date because of a lot of the reasons people don't think about this stuff. It's cause they haven't seen it, they haven't experienced it because somebody in safety was doing their job. speaker-0: Right. So now that we've laid that groundwork, I'd like to hear some of these specific ⁓ uses that that you are working on right now. ⁓ I know Mila's among them, Coupe One. So take your pick here. Tell us what you specifically are doing around ⁓ trust and safety in in this age of AI where we're s increasingly having to deal with these ⁓ problems in ways that maybe are accelerated and and just more critical than ever. speaker-1: Yes. So I think the biggest one for us is child safety. When we think about just the tools we're working on, that's one of the use use cases that we kind of have at the core because we feel like it's the most complicated and it's the most complex. And so if we can get it right from a workflow perspective in our tooling, that'll let us get a lot of the other harms right. So things we are building there. ⁓ we have a tool called Coop, and it really is your foundational content review console. Where you want to be able to detect your content coming into your platform, label it. And if it's child safety related, if it's child sexual abuse material, you actually have some reporting obligations. Again, varies by your jurisdiction, but the biggest one is an organization called NICMIC, National Center for Missing and Exploited Children. There we go. ⁓ NICMIC. And so we have built-in NICMIC reporting in Coop because actually it's a fascinating, challenging problem of what makes a very actionable and helpful report to NICMIC, who then passes it on to law enforcement. But you're, you know, your average developer is not going to be able to have time to dive into all that research, all that, you know, years of discussion between industry and NICMIC and law enforcement to conclude like here are the best practices. So Coop is just like, here are the default fields. Here's what you should put in there. Go, okay doke. Can do. ⁓ So it really is from a infrastructure side, you can think these are things that look a lot like I'm just shuffling data around. But then we put the safety layer on top where there are things, features in Coop like moderator wellness is really important. So how do I say, you know, this image is coming in, I'm putting it in front of a person, I'm going to put it into grayscale by default, because I think that's going to have less harm on the person who has to look at this image. Or I'm going to blur it to a certain degree by default. And then I'm going to maybe turn off audio by default. These are just some of the features that ⁓ our team is and the community members too have built into Coop, coming from their experiences working in safety and thinking about how do we want to pass these on to all the people who might be entering into this field. So one of the exciting things I think we found that really like validated our mission and why we exist was a small messaging app called Kyoto. ⁓ There are about two developers who made an app that's got hundreds of thousands of users. And ⁓ in the first Prior to f ⁓ coming across our tools, they'd never been able to do child safety reporting. And then they were able in the first week, I believe, they caught nine potential instances of child predation on their platform and reported to Nick. And by the second week, it was up to 19. so it's that for us was exciting. Like that's the way that child sexual abuse material should be handled in these platforms. And we were very grateful to be able to give them this free solution. speaker-0: That's huge. And how are you getting the word out? My goodness. I you know, I I could see that this should be used by way more platforms. And nineteen as horrible as a number ⁓ as that sounds, I I feel like we we need to ensure that all of these are getting caught, right? speaker-1: Yeah. I mean, honestly, we're this is the fun part of the open source. We're an open source community. We're just doing the doing the circuits, doing the rounds. We have an amazing community manager who's, you know, knocking on every door. ⁓ and these doors kind of vary depending on is it somebody who's maybe familiar with trust and safety, they just need a better tool, or is it somebody's like, hey, let me walk you through the absolute basics of what might happen here. I think sometimes we also are in the unfun position of being the bearers of bad news. Where a lot of people think that'll never happen on my platform. I don't need that. And then it does. And then they call us back. speaker-0: All right, so you're getting the word out, you're at that phase. ⁓ in terms of education, ⁓ you alluded to that. ⁓ so going to conferences, attend it you know, trying to be in places where developers are. ⁓ are you creating like a certification program, like I don't know, YouTube videos? Like what what does this look like? speaker-1: Yeah, no, we're not I think there's some really great partners in the safety field that are probably better placed to take on things like certification, like the TSPA. they do curriculum and sort of training like that. We're we're the tool nerds. We're just here to give you free tools, you know, we're just here to ⁓ ship code. That's really our our specialty. ⁓ yeah. We're we're the tool nerds. speaker-0: And in terms of like community feedback, are there folks that are in your GitHub repo creating issues, like talking back or like how are you knowing what the best strategy is moving forward? speaker-1: Yes, that has been really fun because like I said, the open source is new to safety. And so I think we've had to do a lot of education and sort of coaching folks along when they, you know, email us with a s private suggestion of, hey, the tool should do this. Like, mm, you should do a GitHub issue. And I love the open source one on one stuff. So that's been really fun of encouraging people to do that. ⁓ if you go to our repos, you'll see there's a good variety. I think one of the things that ⁓ Roost has decided to do, which will be a little bit unusual is that we're actually going figure out how to put some ⁓ what are essentially core contributors on staff. So, you know, we'll you'll see we've been hiring for engineering positions. Our thinking being if we're trying to help the people who don't have the ⁓ resources and the financial means to have these, you know, expensive tools, then they probably don't have ⁓ all the time to be the super active contributor and keep up with that core. So we're gonna, we're gonna take that on as best we can. speaker-0: Right. Okay. ⁓ and I I'd like to hear more about the Mila suicide prevention guardrail because ⁓ that seems like I don't know, it it it intersects in significant ways with with ⁓ the coupe one point zero that you've ⁓ described. speaker-1: Yes. So Mila is a Quebec AI lab. They're just very thoughtful, incredible people. ⁓ I was gonna say up there, because I'm north of or south of Quebec, but yeah, incredible, thoughtful people. ⁓ you know, really specialists in AI. And so there's something interesting here. We should actually maybe back up a couple steps and talk about safety's relationship with language models and AI and all this. So yeah, a framework called Dire that my colleagues worked on, and they will tell you, because the situation is dire, which is true, but it's a horrible pun. But I it is a good memory, it's a good memory mechanism. Dire. ⁓ recognizing a safety workflow as detection, investigation, review, and enforcement. So when I was talking about coupe, that addresses review and enforcement. We can talk about our other tool, Osprey, hits investigation. But D detetection is your first step. Like you have to be able to figure out what what is this, what's coming in, what are the patterns, what's the content. And for Forever, the state of the art and safety has been to use large language models to do that. Like you're you're looking for pattern matching. So as AI has advanced, there has been a lot of interesting defenses and increases and advances and all the all the good stuff has trickled down to some of those detection things. And that's where I think we're starting to get some really sophisticated, interesting things, like the work that came out of Mila. So what we did with them, and this is incredibly specific, but I'm I think it's a good It's good to start small, validate what you're doing, and then expand. So what's important is how are we thinking about youth mental health with AI chatbots? How are we thinking about, you know, how we protect these outputs, how we ⁓ really prevent some of the egregious things people have seen, like youth ⁓ ending their lives through suicide because of directions that they received from a chatbot. And so Mila created this safeguard that is very focused on how do you stop a chatbot from giving suicide directions. It applies to anybody, but in particular, their training was looking at youth in particular. I think one of the really interesting things they found was that in training, to train on a general corpus of human-generated ⁓ directions wasn't giving them the results they were looking at. They really actually had to train on chatbot language. And I think there's something very interesting that, you know, they're I'm sure their researchers are continuing to unpack in why was that so such a significant change in output? But that's what they ended up training on was more the chatbot language. And now we have this openly licensed ⁓ safeguard that if you're saying I want to integrate in open or actually any chatbot into my application. This is really important to me. I'm gonna run this alongside. I'm gonna run this on top to make sure That my youth users don't receive these directions if they're asking for them. speaker-0: That's huge. Okay. ⁓ so ⁓ to be clear, this a dire project, so that was something that Roost, that was an initiative that Roost was involved in, or is it just a project that some of the folks at Roost ⁓ kind of knew about and and had some partnerships with? speaker-1: This was some of the pre-work that inspired the idea for Roos. Yeah. So, like Camille Francois, who's our president, Juliette Shen, who's our director of product, were part of the group that built out this dire framework. And as part of that, they were doing research, ⁓ looking, talking to people about their tools and saying, you know, here's this framework. What do you have? How do you work through these different quadrants? And I think they found everybody's tools were very similar. People felt that their tools might be different because their policies are different. the rules they have on their platforms and how they think about what's allowed on their platforms are different, but the actual tools and the infrastructure that does label action, move to investigator, you know, take action, follow enforced rule, it it was all very similar. And that's what kind of sparked this conversation about, okay, not only do we need to solve access in trust and safety, but open source is going to be the way to do it. speaker-0: this idea of it being a guardrail too, because that has a very DevOpsy feel. so yeah, I d I don't know, was it what was the thought behind calling it a guardrail? speaker-1: I think that's kind of become the the nomenclature of of this ⁓ you know, we could think people call things safeguards, people call things guardrails. And I think everybody's always saying something slightly different. But ⁓ kind of this idea of I wanna stop a thing. I I push my bowling ball down the lane and I want a bumper to bounce it back in. but I don't think we wanna call them bumpers, Kate. That's not exciting. That's not exciting. speaker-0: Okay. Bumpers are right out. Guardrails are in. Okay. Fair enough. I like it. So you're also thinking about maintainer burnout as it relates to AI slop. Talk to me about how Roost is also helping with that. ⁓ it's something I've written about a fair bit, so I'm I'm voice got my ear to the ground with how we're trying to help maintainers ⁓ at this time. speaker-1: And buckle her out. Yes, yes. This is something as we were, you know, putting our community together. We were trying to figure out like what is our policy about AI contributions going to be. one of the things too that was important to us was we're Apache 2 licensed. That was a very intentional choice because of the commercialization opportunities it provides. We think there's a route in which Roost Tools being out there kind of becoming hopefully this de facto standard that there's a huge commercial ecosystem around things. All that to say we need to have the right to distribute what what we're building. And so from a bunch of different angles and all the stories of people being like, I can't take 20,000 line code changes. That's not reviewable. We're like, we have to do something here. ⁓ I think our approach was to really try and keep things very simple until we have the next problem. I ⁓ there's a pal in open source, Sarah Navodny, who has this phrase minimum viable governance, and I absolutely love it for You know, any problem in building an open source community thinking about what is the minimum thing I need, and let's not put the cart way before the horse and build something really complicated before I get to the get to the basics. ⁓ for us, we were like, okay, we are seeing some of these challenges. We are seeing things where people drop in massive code changes that you can't really figure out. You drop in a thing that's not related to an issue. But we tried to kind of pull it back and say, well, to other developers, these are actually very helpful tools. At the end of the day, AI is just another developer tool in this context. And so our engineering, you know, expectations of what makes a good commit message, what makes a reviewable code review, that hasn't changed at all. And so our policy as a community kind of became, you know what, use the tools you want to use. and that includes AI. Use what you want to use. But at the end of the day, it just has to be work that you have to be able to explain, that you're ready to stand behind, and that you can say, yes, I wrote this and I'm submitting it to this project. speaker-0: It's amazing. And how did you codify that? Did you create like ⁓ what, like a policy? speaker-1: Yes, yes. We've got we've got a policy in our repos and says this is, you know, part of contributing. Read our AI contribution policy. Here we are. Yeah. Yeah. And you know, it's not a hundred percent foolproof. You know, we still we still get the occasional. I think we had one the other day. You know, 7,000 lines, here it comes in, and you're like, this, I I can't read this. I can't digest this. but I think when we've talked to people, you know, we've gone back and said, hey. I understand what you're trying to do here. Here's why we can't take this as is. Here's our recommendation of how you break it out. Please follow these policies. I think people have been receptive. I think to date we haven't seen anybody say, No, you must take all seven thousand of my lines at once because Claude told me to. We we haven't seen that. speaker-0: Small mercies. I I I appreciate that. Okay. So you've got your AI policy in place. ⁓ you are permissive, it sounds like, in terms of allowing it. And have you been inundated with more issues than you anticipated? ⁓ because of course you were birthed in the time of AI. So you were within the slop the slop was already happening. ⁓ but did did you prepare for that like preemptively? Is this like how new open source projects are going to have to enter the fray and say like, well, get ready for it, here they come? speaker-1: Down. You know, I mean, I think it was on our mind of that this is gonna be something. And that's where we tried to see what would happen and get ahead of it with our thoughts about what do we wanna do here when this becomes a problem. I I think great, thankfully, we're niche enough that we're not inundated with a bunch of weird issues that are irrelevant and, you know, gamified. I don't think people really find us unless something else relevant has led us there. Although I have been told on multiple occasions we have good ⁓ GEO. Is is that generative one. Instead of SEO, this is like generative AI engine optim I don't know. speaker-0: Vertical AX, if it's what I'm thinking that you're referring to, like agent experience, where the agents are able to use your code. It's it's findable. speaker-1: ⁓ that sounds Ooh, that sounds much better than GEO because I'm from the Pacific Northwest and so that sounds like gooey duck, which is a slippery, slimy delic delicacy. AX sound we have good AX. All right. So I've Okay. So people are only finding us to date, you know, when they're ⁓ it's relevant and they were already looking for that. Yeah. speaker-0: Okay. Okay. Yeah. I mean, it comes down to incentivization in a lot of ways in in terms of like being inundated with slop. So like the curl ⁓ project, you know, they had a very famous bug bounty ⁓ historically. And so I don't know if if at Roost you you are paying people to find these bugs, but it sounds like yeah. Okay. Not yeah. So that probably saved you a little bit at least. Yeah. speaker-1: There's no incentive here. I don't have T shirts. I mean I d I have good stickers actually. But aside from that, ⁓ no. Actually I do have T shirts. We just made T shirts. Okay. Th but there's not gonna be any T shirts. But no T shirts, no stickers. ⁓ we're not speaker-0: T shirts. All right. You're finding bugs. It's through the goodness of your heart. Yes. Not because you're being incentivized. Okay. speaker-1: Yes. And ⁓ today we are not ⁓ you know, we're we're only 18 months old and part of my work in open source has been in incident response and vulnerability management. And ⁓ you have to have a really good, robust response process before you're ready for the bug bounty where you're dangling money carrots at people. We're not we're not there yet. We have a good process. We've had our first security issues and all that, but we're not ready to dangle money carrots. speaker-0: Understood. Okay. All right. So to continue on with these bird metaphors, we're roosting. I want to talk about Osprey now. ⁓ talk to what what is Osprey? ⁓ and and what what does it do? How are you leveraging it, etcetera? speaker-1: Yes. So if you remember that dire framework, the I investigation. So you can also think about coop being if you were to say I'm brand new to this, where do I start? You would start with coop. That's the thing that goes content goes in front of a person, human moderator, you know, actioning, making decisions. Osprey is ⁓ a little bit more mature. You're thinking about I am seeing some sort of pattern or there's something I'm hunting for on my platform. And I'm trying to figure it out and I'm trying to automate what is Automate away what is obvious, things like spam, fraud, just say, I am very high confidence that I don't want this here. Here's a rule to just kick it. ⁓ or you're like, what is what's going on? Why are all these accounts suddenly being created that share these keywords or they share these set of IP addresses or these timestamps or something funky is here? Osprey is the tool that lets you query, query, query in these user-defined ways to figure that out and then decide what kind of action you want to take. ⁓ Osprey actually has a very interesting backstory and it kind of speaks to why Roost exists. ⁓ yeah, are we here for stories? We're here for stories. So there was a safety tool called Smite. ⁓ this is probably I have to get my timestamps right here. ⁓ but maybe the 2016 time or so. 2016 time. ⁓ and it had a bunch of different users. People, dirt platforms really liked it. Safety analysts really liked this tool for doing their investigations. ⁓ a company. speaker-0: Ready for a story. speaker-1: with birds bought it because they also really liked it. ⁓ but they didn't have any interest in running the business. They bought it because for them it was, you know, penciled out that we want to have this as a tool internally. And so they shut it off for everyone else. And ⁓ I think it just goes to show how, you know, overnight people were missing this critical investigation tool because one market just said, nope, we're we're done with this tool. And it hurt everyone online. So There was a group of folks who ⁓ were at Discord and said, This cannot happen again to anyone, to us, to anyone else. And so they rewrote their own version. and they ended up donating that source code as the core to Roost that became the core of Osprey. So when we encounter people who have been in safety for a long time, say, Do you remember Smite? And they ⁓ yeah, I'm like, We we've we've got Smite back. Smite's over here now. ⁓ so it's it's kind of a fun story, but I think it does speak to how we just we can't have such critical technology tied up in proprietary ecosystems. speaker-0: Very good story. You're right. Was Smite ⁓ like, did it go under the like business source license? Like what how did how were they able to do that? Was it MIT at any point? Like yeah, how'd that work? speaker-1: The original one, it was never open source. It was just a vendor tool. It thought a lot of people had. And then they acquired it. And then they're like, ⁓ we don't want to run this for other people. We just wanted to run it for ourselves. And people got the API shut off. speaker-0: I see, I see. So they were paying for it. Get it. Okay. And so I mean that's kind of like an expression of your philosophy a little bit, right? Like security, safety should be available to everyone, open source. speaker-1: This should not happen. ⁓ you're you know, if for some reason things change, you should still be in control of the technology you need to keep your user safe. This this shouldn't happen. speaker-0: Now many folks in your position with this open source version of Smite with Roost, go with the foundation route. Why did you opt to not do that this time? speaker-1: To not put it in a different foundation? speaker-0: Like an i well, is Roost a foundation? speaker-1: You could say we are. I I think we didn't we didn't use I think we didn't use the phrase foundation, but we're a nonprofit. And we're we're an open source nonprofit. ⁓ yeah, we just decide not to use the name Roost Foundation or Safety Foundation. But yeah, no, I'm I'm I'm in the the the what do I say, the table where the open source foundation people come together and go, What are your problems? That's where we're that table. speaker-0: Yeah. That's what we're At that table, ⁓ is there any backstories in not calling it a foundation? Because I feel like that is what many people call it, when they're like, we run an open source project, we are a foundation. speaker-1: That's great question. ⁓ You know, I don't think it was. I think maybe the bird puns were strong and like that. Roost foundation is a mouthful. Yeah. speaker-0: Yeah, sure. Just didn't happen that way. Just didn't happen. Referring to Okay. That's great. I mean, you you're doing good work here. Nonprofit. That's great. ⁓ okay. And talk to me about some of these partners because you have an amazing logo page. ⁓ how are you working with some of the the big ones here? And I won I won't reveal Eddie. I'll let I'll let you do that. speaker-1: Four. Yes, I can reveal them. We have ⁓ both philanthropic and corporate partners and ⁓ you know other ones who are fellow travelers in the nonprofit open source space like Mozilla. It's it's a wonderful group of people. so the more kind of direct connections on the corporate side, these are people who understand and realize that if the bar for safety isn't lifted the bar, the boat, the ride, the tide, if safety is not elevated for everybody, ⁓ it's Bad business for them. It's not right for humanity. ⁓ I think they see that. So Google, for example, is one of our corporate partners. And that's because they have something called the Google Content Safety Ape API, which is a free tool for CSAM scanning. That's the child sexual abuse material we were talking about earlier. However, ⁓ you can only get access to this tool if you are if you have something like a content review console to properly do NECMEC reporting. And so Pun intended, there's a chicken in the egg problem there, in which, you know, Google wants these smaller companies to become bigger companies and these startups to exist. And CSAM scanning is part of that. But if they don't have the content review tool, they can't give them access to it. So here comes Roost. We're here's the open source free tool. Now this ⁓ amazing thing that Google offers with the free CSAM scanning is available to way more people. So that's kind of the flywheel that happens there with our corporate partners. speaker-0: I also saw Blue Sky on there and you've got a Blue Sky story, right? ⁓ I I I'd love to hear more about that. I I actually was hanging out with some some cool Blue Sky celebrities ⁓ at at Render ATL, which was ⁓ just in Atlanta. So ⁓ I definitely have ⁓ blue sky I'm blue sky pilled. speaker-1: Yes, yes. ⁓ they've been so great. Such a great partner. ⁓ so they're an Osprey user. And they came in pretty early on it as kind of that first pilot user of like, it's leaving Discord. ⁓ we're trying to build this more generic project together that's, you know, works great for everyone outside I was gonna say outside Discord, Discord included, they're also an Osprey user, but y you know, it's generalized, it's good for everyone. And ⁓ they were just so generous and willing to be that kind of first trial of saying, here's what we would need, here's what's working well, here's what's not. ⁓ so they've been a fantastic partner in that way. And obviously running at huge scale, which is really helpful for us to be able to figure out, you know, what's sort of the edge of this, what are improvements we need to make, they've been wonderful. speaker-0: That's fantastic. Okay. Well, shout out to Paul Frazi, ⁓ CTO over there. I've mostly heard him talk about like at pro, but I love that he's also talking about Osprey then. This is, you know, how else are we thinking about scaling some of the problems that are unique to these social ⁓ media sites? speaker-1: Right. And I think the the gener in general, like I guess we could have a whole other hour of decent the decentralized and roost story. Because I think too, for these organizations, this is something they desperately need. The Mastodons, the folks building on AT proto. when we talk about who might not have access to expensive proprietary safety tools, those are great examples. So it's been it's been really neat to have conversations with those different communities. interesting challenges about how do you scale safety in decentralized communications, but everybody's just been so wonderful and it's ⁓ we have fun challenges to tackle ahead of us. I'll put I'll put it that way, but with great people. speaker-0: I mean, I know that they're thinking through a lot of ⁓ challenges that maybe had been left in black boxes historically with some of the older, more legacy ⁓ social media sites. So it's interesting the the idea of trying to distribute them and make them owned by the user. That's that open source mentality, right? I I I want this to be mine so that I I have the ability to leave afterwards if I decide to go on a different Mastodon server or whatever. Can you give us a teaser of like what would you say those challenges are that are so unique? speaker-1: Yeah, no, no, no. So I think one of the ⁓ most challenging ones when you think about whether it's ⁓ activity pub or AT proto, you want to enable people to build platforms on top of that protocol. And that does put them in the driver's seat in many instances of moderation. However, if you think about if I ⁓ think of a world where there's, you know, eleven thousand instances of chat things and chat communities outdoor, do I really want eleven thousand groups of people to have to moderate? ⁓ terrorism and extremism and violent content and child sexual abuse content. Like from a burnout and a mental safety of the people doing this work. I don't think we want that. So there's some something to be figured out and to be sorted out that respects the decentralized philosophy without saying we decentralized the worst part of running social platforms. You know, I I think there's there's something to be found there. And I think these are the conversations that are being had and people trying to figure out because They do want to have that decentralization at different levels of safety, but maybe not some of our most egregious harms. And so that's kind of the the challenge of the decentralized movement and the scaling when it comes to safety. speaker-0: Yeah. ⁓ so Discord and Roblox are also partners of yours. Are they also users of Osprey or are they thinking about it in a different way? speaker-1: So Discord is. ⁓ Roadblocks actually just the other morning announced that they joined something we have called our Roost model community with some interesting open safety models that they've released. what's interesting about when we think about how do we make generalized safety tech is that in some of these verticals, sometimes the content is so specific that the platform, you know, maybe the the tools look very similar, but those policies start being Like a policy written for gaming is probably not applicable at all to LinkedIn. ⁓ you just think about gamer language and other things like that. It's ⁓ it's not transferable. And then you you add on top of it things that like this is a platform built for youth, their language, their customs, not transferable to platforms built for adults. so Roblox has been kind enough to open source some very interesting tech though that they have built around things, ⁓ detections in particular for grooming, for child grooming. Because what's challenging from a technical perspective about that is that grooming is a thing that happens over a long period of time. So if I'm thinking about how do I look at data and detect this is it, there will never be one instance. And in fact, if I look at that instance in isolation, it probably looks innocuous. Like the exchange doesn't clue me into anything. It's that it's happening over a long period of time. And within the context of that window, then I say, ⁓ wait a second, something dangerous is happening here. So They have a library called Sentinel that does look over that long window, but then actually flags things at the user level. Because again, if I just pull out any one message and say, this is dangerous, that's not super helpful. I want to flag the user to an investigator and say something is going on here that somebody needs to take a look at. So that's kind of the the Roblox story. They've ⁓ also been great partners in thinking about like how do we change this industry overall and are bringing, you know, some of the insights of what's happening on their platform to. To everyone else. speaker-0: Hm. Well now I'm depressed again. So thank you for that. ⁓ speaker-1: I don't know hopeful stories. I mean I I ⁓ the people of this are the hopeful story. The you know. speaker-0: Okay. I don't know. speaker-1: No ⁓ speaker-0: We got it's important work. I know. It's just I thinking about grouping. I don't know. It's really it's it's bad stuff. speaker-1: It's challenging. It's challenging. But these kind of ⁓ multi turn conversations like this and ⁓ things that happen where people in particular youth are bouncing between platforms from a technology perspective are some of the hardest problems. ⁓ so I'm you know, the hope is that everyone is coming together and maybe we can solve these together a little bit better. speaker-0: I I love that and I think that's great. So I that is you're right, that is the silver lining and me being a mother of small children with her head in the sand is not a good look. So I should look up and realize that we need to confront these problems. So I'm I'm doing that with you here and ⁓ this is this is all for the good. ⁓ but ⁓ I yeah, but I you know, I I I enjoy the story about Roblox, you I I think of like Minecraft and a lot of these ⁓ places where young people ⁓ are chatting with strangers on the internet. Yeah. speaker-1: you know, if I'll give you some optimism here, like the reason we're doing all this is that the internet is fun. Like, you know, it is valuable, it is fun, it's where you go to learn, it's where you go to find new interests, find new people, reach corners and ideas that you couldn't reach sans internet. digital spaces are great and they have huge power and potential. So, you know, even though some of the things happening on them right now are horrific, ⁓ I think we can get o we we can't get over that. We can win. We can we can win. We can win. We can all right. We can do the work to make sure that these s spaces are safe. You know, it's it's work that's never gonna be done. ⁓ but I think we can do it smarter, we can get it out to everyone and we can have the fun internet again. speaker-0: Okay. All right. I'm all for fun internet. ⁓ I do want to talk, so I'm gonna go dark here. I want to talk about the dead internet. How are you dealing with the bot issue? I know Cloudflare's been thinking about that a lot. A lot of folks have been doing thought leadership about, you know, X percentage of the internet is whatever. I know I lost many of my ⁓ my my Twitter slash X followers are gone now because they were my very loyal bots and now they're gone. And so, you know, it's very depressing. I'm under 10,000 followers. I feel My self worth has just plummeted. ⁓ anyway, so the internet's full of bots. I is there a way that you're you're trying to grapple with the the ⁓ what, the the automization of ⁓ of these, you know, fake people that we're interacting with? speaker-1: Yeah. I mean, this is the ⁓ the challenges AI has has brought this field is that things are at massive, massive scale. And the consequences from a bad actor's perspective of having what I essentially would like a low effort campaign just went to zero. And what I mean by that is that, you know, before, if you were like, hey, I want to scam this person, ⁓ you maybe would put some effort into a couple different targets. And now it's like I've got nothing to lose. I can just roll this all automate it out a hundred times, a thousand times. If one bites, great. And it the before you could never operate that way. Now you can. So I think when it comes to things about how do we think about what tooling we need, it we need tools that are gonna act at that scale and find, you know, be able to handle that volume, be able to find those patterns ⁓ at a world in which a bad actor can just roll out hundreds of of fraudulent things if that's what they want to do. speaker-0: Okay. I wanna ask one more dark question. So a thing that occurs to me is if you're open sourcing some of these policies, then bad actors could ostensibly point their LLM at it to bypass them. Is that a concern or like how are you remaining nimble enough? Like I can see openness being kind of a counter to ⁓ to these abil like the fact that we've all got ⁓ a hundred X developers in our back pockets with Claude. ⁓ So, you know, how how is that a concern? Like how what do you how does that work? speaker-1: This is something we talk about a lot at our organization and with, you know, people in this space. ⁓ I think it's part of the history of why trust and safety has not really had a robust open source movement until now. ⁓ because I think obscurity has been a main defense and a strong defense around a lot of these practices. I think for us though, it's sort of like anything else in ⁓ open source software where we're thinking about what is the net benefit that is gained by people having much more access to this tooling, by people being able to customize it, by people being able to redistribute it, compared to could some bad actor abuse it? Yes. Will it happen? Can't say, don't know. ⁓ But we have we can't pretend that that's not a consideration. But I think when we look at the net benefit that could happen, we go, that's bigger. That's better. That's the thing that we, you know, we need to do. I think from what Roost is building too, there's something to keep in mind that without all the other things that make up a platform, like the user data, you know, whatever those special fields are in the content, the tool that just says label route, you know, cue, it's not much. It there's nothing special there. ⁓ but I think just the idea that the critical tooling for trust and safety is open source, it it does make people a little twitchy. They're a little uncomfortable. And try and go in and say, Hold on a second, let's really think through, ⁓ you know, do our adversarial testing, what really could happen by these tools, the source code being available, ⁓ probably not the the ⁓ dire outcome that you you're have in mind. Yeah. speaker-0: Okay. Okay. All right. Well, I feel like that is a good place for us to wrap up ⁓ thinking about this. So before we go, Anne, what links would you direct folks to if they're interested in learning more? ⁓ and even your personal musings. Like do you have a particular blog or social media account that you ⁓ prefer folks to to look for you at? speaker-1: ⁓ sure. ⁓ so we are roost.tools is our website. Our GitHub is Roost org, and they'll you'll see Coop, you'll see Osprey, and you can also find the links to our community channels, like on Discord where people hang out. So we definitely encourage people to check out the tools. ⁓ even if you're not in safety, if you're just a developer who's really passionate about it and you're familiar with open source, we'll take your contributions. I think that can be a a nice way for some folks to feel like they give back in this space. ⁓ and then me personally, I'm really enjoying Blue Sky. ⁓ Nice. Yeah, I find it fun. I find it fun. I've see now I'm like, ⁓ dear, what's my handle? I think it's Anne B D H. I think speaker-0: All right. I'll confirm and put it in the show notes. How about that? That way we'll be sure. Yeah. speaker-1: But no, I I think it's a fun platform. you know, I I again I remember their early days of social media stuff and ⁓ I s I see that spark back and so I'm having a good time there. speaker-0: That's fantastic. Yes. That is it is my preferred place to go. I, you know, shout out to to Jeremy Mice, who has recently created a pop punk emo two thousands bot to share, yes, to share lyrics with me whenever I'm feeling like I need that in my life, which is, you know, most days. So ⁓ yeah. So anyways, good good fun stuff going on at Blue Sky. and ⁓ yeah, so so that we we'll we'll keep that energy going. ⁓ speaker-1: So next time we talk, do we have to have swoopy bangs? speaker-0: I I am not a bangs person. My hair does not do that. That is not a thing for me. ⁓ but I I love I love the commitment for people who do. And I I applaud them and I all I want to be like Zoe Deschanel as well. ⁓ but ⁓ it just good bangs I know know thyself and it was it is not a thing. But you know, and I I you know i i if this is the the journey you're on, I I support you entirely. speaker-1: No, no, no, no thy hair. It's it's it's not happening for us. speaker-0: Not happening. Okay, beautiful. all right. Well, I have really enjoyed speaking with you today, and this has been a lot of fun. ⁓ again, my name is Kate Holterhoff. I'm a senior analyst at Red Monk. If you've enjoyed this conversation, please like, subscribe, and review the mugcast on your podcast platform of choice. If you're watching us on Red Monk's YouTube channel, please like, subscribe, and engage with us in the comments.