Latest / Oxide and Friends / Scaling Bluesky with Paul Frazee

Transcript

Scaling Bluesky with Paul Frazee · Oxide and Friends

• 0:00
  Bryan Cantrill: Is that true? I can join the I think I've always joined by putting up my hand. Have I always done this? I thought that's how you joined with, but I thought I put up my hand and then I joined.
• 0:11
  Adam Leventhal: It's like, are
• 0:11
  Bryan Cantrill: you just like core Brian always raises his hand before he
• 0:15
  Adam Leventhal: It's like that guy in section. It's like, dude, we are in college. We're not in middle school. You don't have to raise your hand. Just like you've got
• 0:21
  Bryan Cantrill: Oh, I'm the I'm the guy that likes to stand while he's presenting a section. Is that wrong? I guess so. I mean, I thought everyone did. I I I get Okay.
• 0:29
  Adam Leventhal: Holy smokes. Alright. There is Paul. Not even raising his hand. He's just waiting patiently to be called up.
• 0:35
  Bryan Cantrill: He just
• 0:37
  Paul, thank you very much for joining us. I am super stoked for this today.
• 0:42
  Paul Frazee: Linda, thanks for having me.
• 0:44
  Bryan Cantrill: Oh, I
• 0:47
  look. I'm not convinced that there's only one of you. There you just especially in those early days of Blue Sky, you were like everywhere at once. I just felt, I mean, it it just it's been remarkable. Anyway, so, I I'm not sure how many of you there are, but I appreciate your your collective allowing one of you to just come speak with us today.
• 1:11
  Paul Frazee: Yeah. The one with social skills.
• 1:14
  Bryan Cantrill: There we go. Exactly.
• 1:17
  Paul, there's so much to talk about here, but, I mean, you all have been on just an incredible ride. I mean, I I it's hard I mean, having watched this thing from when it was in invite only for kind of a protracted period of time, and then Mhmm. Just going, I mean, non linear, not in the metaphorical sense, but the actual, like, literal quantitative sense, has just been amazing. And I I wanna get to a bunch of that, but before do you mind can we can we talk about Scuttlebutt? Is that is that is that inbounds?
• 1:53
  Totally. Yeah. That's totally inbounds. Let's do it. Because I wanna talk about Scuttlebutt and Bigger.
• 1:59
  That's people think they're in here to talk about Blue Sky and Blue Sky, but we're actually talking about Scuttlebutt and Bigger. The, so Scuttlebutt so could you describe what Scuttlebutt is and how you got involved in Scuttlebutt? Because this is obviously a problem that you have been thinking about, and I think this is true for all, it sounds like a lot of folks on the Blue Sky team. People have been coming from this from different angles thinking about this problem for a long period of time. How did you kinda get in get into Scuttlebutt?
• 2:25
  Paul Frazee: Yeah. I mean, so, like, everybody on the team has been at this for years. So, like, up collectively, I think we have 20 to 25 years of, like, decentralization work, like, leading into the team. Sculbot was my first one. That was back in 2012 ish, and that was a technology invented by Dominic Tarr.
• 2:46
  And it fit into actually, all of us came. Everybody on the blue sky team came from, like, the p two p world. Right? Which was kind of like, hey. Could you take some of the techniques from BitTorrent and then do some modifications that actually try to build sort of, real time or or a large scale or social applications depending on what you're up to using those kinds of p to p techniques.
• 3:09
  And Secure Scaldbot was directly geared towards social networking. So it was a peer to peer social network. And it had this very aggressive kinda local first mentality. It used, it was called, Secure Scullebook because, actually, it was based on a Scullebook, gossip protocol, where you're just, like, having each node kinda, like, rebroadcast logs to each other in a kind of best effort way. And that meant that it had a very fluid topology of connecting together.
• 3:42
  Like, anytime you were able to catch up from, one node that you were able to connect to, you'd be able to, which makes it actually quite ideal for even extreme cases like a sneaker net if you were so inclined. And then what was sort of interesting about it was that we merged together the gossip protocol and the social layer where it would use your follow graph to decide which account logs to synchronize. So if I was following 5 people, the default, kind of intuition there is whenever I would connect to a note on the system, I would ask those notes like, okay. Here are the 5 feeds that I'm following. Can you, like, catch me up on anything that you have for them?
• 4:17
  And since you're connecting sort of, like, to a forest of different nodes, you would presumably, you know, at some point, catch up to everything in a kind of an eventually consistent way. And, we actually would take it further where you would do friend of a friend expansion. So you would actually ask for, like, the 5 people you're following plus, like, 2 hops out if I remember right. Really sort of wild way to do it, but the it was aggressively decentralized. In fact, I would call it, an anarchy and not in, like, a pejorative sense, but, like, quite literally, no authorities were were encoded in the system.
• 4:50
  Interesting. We were yeah. So it's very, very kind of
• 4:55
  Bryan Cantrill: a response. Because, I mean, this is, like, pretty early. I mean, this is before the animals were walking on their hind feet in terms of social networks, or maybe they were already. Maybe maybe they were born on their hind feet.
• 5:07
  Adam Leventhal: Yeah. What's the era? What like, when when was Scamba?
• 5:10
  Paul Frazee: Yeah. It was 2012. So, like, I can say that that vibe at the time is correlated probably to 2 different things. For 1, it was pretty clear by then that, like, the major social players were taking the you know, sucking all of the oxygen out of the room. And so there was a collective of, like, open source hackers who were feeling really excluded from what was exciting, about, like, social computing and wanting to be able to get in there.
• 5:38
  So you had a a general the through line throughout all these projects has always been, like, frustration with monopoly and feeling like you're not able to get in there and make meaningful change because because we're talking about Internet technology. We're all programmers. Let's get in there. Like, let's have that Linux philosophy or something like that being applied, and so that bugged all of us, and that was a big part of it. And then, you know, you also had, still, I think even then, there was starting to be a little bit of initial disillusionment with the major social companies.
• 6:09
  Who knows if we were kind of, like, on the right target with that or not. But we, in general, were starting to you know, this was, like, not that far after occupy Wall Street, which was a big animating Oh, interesting. Yeah. Yeah.
• 6:23
  Bryan Cantrill: And a lot of us were thinking
• 6:24
  Paul Frazee: a lot about, like, where we expected. Like, people still use the term new media in 2012, if anybody remembers saying that nonsense.
• 6:31
  Bryan Cantrill: Right.
• 6:31
  Paul Frazee: So, like, there was there was still there was a quite a bit of thought being put in at that point about, like, what what are the power structures that are gonna be a part of the future of Internet and social media? And, and how are we, like, designing systems to be smart about that? So we were we were thinking pretty heavily about that stuff right from the get go.
• 6:49
  Bryan Cantrill: A pretty impression I have to say. Because I feel like in 2012, honestly, maybe I'm just like a duds, but it felt like it was the kind of the good old days for Twitter. I mean, it felt like it was fun and light. And I mean, and maybe gamer gate changed that. I'm trying to get you kinda play that era in my head about, like, the but definitely, like, that this was I I mean, this is, like, why wouldn't anyone like Facebook?
• 7:10
  Facebook just feels like a net good for society in 2012, says I. I I mean, I just feel like I I was anyway, I was very naive. I mean, I felt like at that era, the I I mean, it was more like Twitter was finally kind of functional. I mean, there was such a long period of time where it was just not very functional, and it kinda keep up with everything and everything else. So I really admire the impressions of people saying, like, no, there's like a there's a there's a centralization of authority here that's actually problematic.
• 7:43
  And so then we're okay. Then I gotta ask you, were you were folks like, you know what, I'm burning my boats on SecureScuttlebutt and I'm now like, you can't find me on any other social network or was
• 7:54
  Paul Frazee: we never got that far. It was, janky at best. And I say that with all love. But, there were we we had a lot of challenges that we just did not, get through. And the the kind of Yeah.
• 8:08
  If we're gonna go through this history at one of the meta arcs I would follow throughout all of it was, like, learning how to, get serious about delivering, you know, the level of quality that's necessary for something like this. Because you you're on the one hand, you're pressing really hard on, like, okay. Novel technology, novel way to do things and throwing out a lot of assumptions. And along the way, everybody, you know, by the time we get to blue sky, almost everybody at the team had been spending, like I said, years working on this stuff, and nothing was quite working. And it wasn't until we got to this project, we all had a bunch of kinda collective collective realizations as we came together about, like, you know what?
• 8:47
  Okay. If we keep these pieces, but then, like, throw in a lot of the kinda complexity and the novelty that what we're doing, We can keep what we think actually is important about the systems, but have this work actually be usable to end users.
• 8:58
  Adam Leventhal: It initially, not working and making it work. Are you are you talking more from, like, a product perspective rather than, like, a technical perspective?
• 9:05
  Paul Frazee: And real really both, to be honest. Both.
• 9:07
  Bryan Cantrill: Yeah. He,
• 9:07
  Paul Frazee: just that. Yeah. Yeah. Yeah. They, but, and I don't know even if I would give more emphasis to one or the other because it was a really, it would affect on both sides, reliability, scale, and performance on the technology side, and usability on the product side.
• 9:25
  Almost every project that would that the group of us worked on all use, like, client side signing keys up until Blue Sky. You know what I mean? So you have, like, key management problems. And you would also have, it was all local first, all these projects. So, like, you would have device synchronicity and, like, essentially, like, you'd had to get them, like, CRDT territory to just make a post.
• 9:44
  It felt like rocket science to try to implement a content section. Yeah. It was it was very complex software that we were messing with. Yeah. So Wow.
• 9:53
  Bryan Cantrill: With with blue sky. Yeah. We're we're just like, hey. What are
• 9:56
  Paul Frazee: we doing to ourselves?
• 9:57
  Bryan Cantrill: I mean, honestly, it explains so much too because I think and I just wanna, like, not lose a say kinda off the jump. The Blue Sky is so remarkably available and remarkably usable in a remarkably short period of time. And, you know, it was it it's not surprising, of course, that you all come with this scar tissue from these other endeavors where you had tried other things. Because, I mean and I think that I mean, speaking as a as as a user, I mean, for social networking is really important to me personally. And what you all have done is so extraordinary.
• 10:36
  And, like, there's been no fail whale. You know, I mean, it was like, the closest we get is like Health Red, which obviously I love, but the the there have been I mean, I'm I'm sure that there have been I I I I sure you're like, listen, like, let's not paper over it too much. There have been plenty of, of Yeah. Yeah. Yeah.
• 10:52
  But the just like the the the response I mean, just when you're adding millions of users a day and you are retaining I mean, it's so responsive and it feels, you know, quick and lively and you're not getting these anyway, it's it's not at all a surprise that, of course, you're learning a lot from these previous systems. So you said, like, there were a couple of things that you you knew you wanted to that you all kind of collectively, from your experience, knew that you wanted to replicate and some things you wanted to improve upon. What what were some of the things you wanted to replicate?
• 11:24
  Paul Frazee: Well, like, the big one that we knew for sure was what we call account portability. And then that one comes to mind first because it has to do with, like, why you bother with peer to peer in the first place. You're trying to turn the services that you use into something that is actually like oh god. I'm gonna say it, fungible, that you could actually replace a hosting service or, like, an application service, but keep all your data. And we knew that was one of the really interesting properties of peer to peer because, like, you know, their client side.
• 11:50
  Right? So, like, your storage and your signing keys will be kept on your device. And so the first thing we sort of figured out was, okay. How can you take advantage of using a server, but, like, maintain that ability to, like, adversarially move away from a server? And that's what the entire account portability system is for.
• 12:06
  And that was a pretty big move forward, because using a lot of the same techniques but putting it on server, and not losing those properties was, like, the the kind of the biggest, like, scale and ease of use unlock that we had throughout the whole system compared to our previous work. We knew we wanted to keep that. Yeah. The other thing that we wanted to keep for sure was the what is now starting to get called a, like, a shared heap model. This kinda notion of, like, an an open sort of broadcast of all of the public data so they can be repurposed because that was a pretty big design element that we had landed on in the peer to peer stuff that we've been doing about how to build these large scale applications is this notion of, let the it it works actually quite a bit like the web where you it's almost like everybody's publishing JSON of, like, posts and and their profile and their likes and stuff like that.
• 12:58
  And then you just, like, aggregate them in the applications, and that's how you build out an application. We knew we wanted to keep that because that's what makes it possible to build other applications without some kind of a hard binding to the hosting. So you're really separating the hosting and the the application layer, and that maintains that hackability and, like, capacity to repurpose these applications in interesting ways. So those are probably the 2 that come to mind.
• 13:21
  Bryan Cantrill: Well, and and I can say, I I mean, I I love both of these things, I think, are extremely important. I think that the, obviously I mean, the portability, it should go without saying, but, one of the the ways that I've realized how just viscerally important this is to me, this is a super stupid thing, but, just in terms of my own age and and social networking tracking my own life, my first two kids, I announced with a blog entry, but my 3rd kid born in 2012, I announced with a tweet. So that tweet, for whatever reason, like the replies on that tweet have been deleted by Twitter and, or I can't find them anyway. And it's like, damn it. You know, that's like a in terms of, like, that that that kind of that portability of I kinda feel like and, of course, like, what idiot would announce the birth of their children on social media and expect it to be retained anywhere?
• 14:10
  Like, look, she doesn't have a baby book. Okay? She's a 3rd kid. This is this was her baby book. This is gonna be the replies on this tweet.
• 14:16
  Okay? If you if you are a 3rd kid or you have 3 kids, you know what I'm talking about. She's I she's lucky she's got a tweet. Okay? She knows it.
• 14:23
  She knows she's lucky she has a tweet now. She's great. But the so I mean, just like having that portability is is is really, really, really important to people. And it is a in order for us to really engage in social networking, we absolutely have to have that portability. So I I mean, obviously, I I love that.
• 14:42
  I Yeah. What's her name? You're saying. I do know her name. Hold on just a second.
• 14:46
  Let me actually, you know, Adam, I know I've said this before, but she the day she was born, we the CEO of the startup I was at was fired. Yes. And, unfortunately, she's got a very good sense of humor. So she's like, hey, hey, dad. It was 12 years ago today your CEO was fired.
• 14:58
  I'm like, you know what? It's is this gonna go on for your entire life? Probably is. But it's true. That is how I remember it.
• 15:05
  I it's just easier to remember the firing date, and then I get to her birthday in one hop. But I Paul, I also love what you've done with the fire hose because when did Twitter turn off fire hose access? I wanna say that was like in when was that? 2013, 2014, maybe earlier? I don't know.
• 15:24
  Paul Frazee: I don't know the exact timeline. I know it got a lot more turned off recently. But Yeah. Ports were
• 15:30
  Bryan Cantrill: turned off.
• 15:32
  Paul Frazee: Yeah. And, like, beauty fire hose.
• 15:34
  Bryan Cantrill: Sit on the fire hose is just, I think, amazing.
• 15:36
  Paul Frazee: It's super cool. Yeah. And that's also, like, what a lot of people have started to play with first because it's just so obviously cool. Like, okay. What can I do with this?
• 15:44
  It's cool in, like, 2 fronts, actually. Because, like, actually, it's just it's really like you're just getting into the innards of our data center, because this is the backbone. Like, it's you normally like, an event processing architecture in a way. And, like, normally, you would use, like, a Kafka or something like that to, like, send all these things through. That's what this thing is.
• 16:01
  That's what this fire hose is. All of our application is downstream of that just running imputed views off of that fire hose. So whenever you're tapping into it, you're just jumping straight into our data center to tap into the data set. So that's really fun. It's it's it's neat to amazing.
• 16:15
  Yeah. Yeah. Like, a meaningful thing, not just like a replication of all the activity, but, like, that's it. That's the network.
• 16:21
  Adam Leventhal: So, Paul, that that seems, that seems tough to, like, to to handle. Like, as a consumer and as a sender,
• 16:28
  Bryan Cantrill: do you
• 16:28
  Adam Leventhal: have a sense of, like, is there a scale at which that becomes no longer becomes feasible? Or in might that have, like, contribute to, to Twitter kinda limiting that and then turning it off? Or do do you Oh, yeah. What are your what are your
• 16:40
  Paul Frazee: thoughts on that? Yeah. There's a or there are going to be some interesting learnings about the economics of this over time to be sure. So I don't wanna be, you know, walking around as if, you know, there's that's not a big reality about all this. I said, it's it is a lot.
• 16:59
  There is a version of the fire hose that we offer that is a kind of a reduced form that, drops all of the cryptographic proofs to make life a little bit easier. It allows you to do filters on it and things like that. But one thing to also one thing to also remember is that the fire hose is kind of a convenience service, to be honest. So if you like like I said before, it works kinda like the web. You post all of you you you broadcast or or publish these records, and then those flow into the applications to, to to be processed into the views.
• 17:29
  The relay is kind of like a crawling bot that just helps out. It just looks around at all the different repos that are in the in the in the world and kinda goes ahead and grabs their replication streams and puts them into one stream for you. If we decide at some point, like, you know, what we can't afford, like, all the consumers on this thing, you can run your own relay and directly do those calls. So, again, you know, that we're trying to make sure that there is openness to the system so that if there are it gets gets to a point where we're, like, you know what, we can't afford passing this long to everybody. Somebody else can do it.
• 18:00
  You know, you can directly do it yourself. Yeah.
• 18:04
  Bryan Cantrill: Well and I I mean, I gotta believe. I don't wanna editorialize too much, but based on what I know of Twitter at the time, if I think that there are, the the fire hose were kind of the crown jewels of Twitter. They did not wanna let people on that. Had to pay a lot of money to go into the fire hose, and I don't think that that was a I mean, I I'm sure folks correct me if I'm wrong, but, I question whether that was a technical limitation. I think that was a a business decision.
• 18:27
  And I think as folks are saying that
• 18:28
  Adam Leventhal: An an injection of ads decision? Is that what
• 18:31
  Paul Frazee: you're saying?
• 18:32
  Bryan Cantrill: I think it was an injection of ads decision. Yeah. And and I also think they they had also made I mean, and this is another thing that I think that, you know, as kind of reflecting, Paul, on on the the terrific work that you all have done. There is an advantage to you coming after having the the world having figured out, 1, how to architect a Twitter that actually performs, which Twitter itself spent half a decade doing. And then 2, they're just like the systems are just better that you can go use.
• 19:00
  Right? Like, it's not you you don't have to deal with a Ruby VM, for example. I mean, you've got other you've got other things that you can go I mean, even Ruby has improved. I don't wanna I know I'm gonna hear it from the Ruby folks, but the, all of that that that software has really improved a lot. So you've just got Yeah.
• 19:15
  Yeah. And then that Firehose is is so extraordinarily valuable to everyone, I feel. I mean, I think that I I love what what you had said in the and I wanna drop a link into the the paper that you all recently published, which is terrific, by the way. I I love this ACM paper that you all wrote, on Blue Sky and the app protocol. And I the just in terms of the the value to researchers to be able to get this whole thing, is is extraordinary.
• 19:48
  Paul Frazee: Yeah. Yeah. I, I am really excited to see what people end up doing with that. I'm really excited for its actual computing and industry potential as well. Again, like, it does form the basis for building applications, beyond just the kind of Twitter style of application.
• 20:03
  So I'm pretty excited about that. You you just kinda nailed, like, what was on my mind as you were talking about scaling and managing to keep things online. Software has gotten better. The resources available have gotten better. The patterns have been really ironed out.
• 20:17
  And in fact, Martin Clubman, who actually, helped us with that paper quite a bit, really really helped drive that paper. He's been consulting with us from the beginning, and that was a huge asset, an incredible win. I was when we first got him, I was over the moon. That's the kind of you know, he he wrote a book that's sort of famous for helping engineers understand how to build these kinds of applications, these kind of high scale data heavy applications. And, so we were able to really apply just pretty much everything the industry had learned about how to do this sort of stuff, both to, like, our internal systems and also just to the protocol, since scaling was always a big part of what we're trying to do.
• 20:59
  We wanted this to be able to be at the scale that you expect out of these kinds of applications. So that, I think, you know, that plus just a really great team that, you know, did some great work to keep those servers running throughout all that growth. That's that's how this happened. You know? So
• 21:15
  Bryan Cantrill: Okay. So but I've gotta say the the level of growth I mean, and and maybe, like, maybe Elon called you up and, like, look, I wanna help Blue Sky out all I can. Please tell me the things that I should do to my own social network to drive people to you as quickly as possible. I mean, that actually a lot more makes sense, honestly, if I knew that. But I I'm not saying for you to confirm or deny that.
• 21:35
  I'm just saying it's, like, kinda consistent with what's happened. I mean, the the level of scaling that you've seen, were you expecting that in your kinda wildest dreams? Is this Oh, man. Yes. This is going exactly according to plan.
• 21:48
  This is what I expected.
• 21:49
  Paul Frazee: Totally. 100%. Yeah. Just everything, every moment. No.
• 21:53
  I mean, I feel really pretty fortunate, across the board. And, honestly, the when we were starting out the project, we were originally just a protocol, like, consultancy for Twitter, which is a complicated and really interesting history. But, yeah, when we started on this thing, we didn't expect there to be either the relevance or the market opportunity to actually have this thing go into production quite like it did. So every step of the way has been really surprising on that front and, you know, exciting. But, you know, when we when we first started to do the beta and, you know, open it up, I remember people were starting to come on.
• 22:36
  And this was, again, like, I don't know, the 3rd probably more like the 4th time I had launched a product in the decentralization space. And up until then, none of them had worked. So we had to see users coming on. I was like, okay. Cool.
• 22:49
  Yeah. They'll be here for a couple days, then they'll leave. You know? Like, I really, really learned that, like, yeah. You'd everything you do fails, of course.
• 22:56
  You know?
• 22:58
  Bryan Cantrill: Right. Right. The crops have failed again, of course. Yeah.
• 23:00
  Paul Frazee: Exactly. Like, that's when they die. Yeah. Exactly. But, no, they stuck around, and it was, like, that was the moment that really shook me the most.
• 23:07
  It was, like, oh my god. They they came back today, and and it's been kind of that level of surprise ever since.
• 23:14
  Bryan Cantrill: That's amazing. And so alright. So you all because, I mean, I originally, when you launched your private beta in, March, April of 20 23? When when was that? Somewhere somewhere in there.
• 23:29
  Paul Frazee: Around there. Yeah. Yeah. Yeah. Around there.
• 23:33
  Bryan Cantrill: The, and which was great. We got I got lucky enough. Fortunately, we've, I I always felt Steve Klobnick is here. I I always felt I may have put Steve in an uncomfortable position when I was asking him for a blue sky invite. He's like I mean, I'm like, I guess I'm technically your boss, but, like and I know I know this sounds like but, you know, I mean, Steve is here either to defend himself or to to bring up his attorney on stage.
• 24:00
  Paul Frazee: Hi, Paul. Good to you. Hi.
• 24:02
  Steve Klabnik: So what I remember it being like is, like, somebody would posted, like, thank god this website is so small that my boss isn't on here and can't see what I posted what I post. And it was, like, 3 days after I invited you. And so I was like, oh, shit. And that was that's my recollection of the story.
• 24:20
  Bryan Cantrill: Right. Well, I I really appreciate you, sharing an invite with me. And, and invites were super precious. And then, Paul, somewhere along the line, the decision is made to, like, really okay. We need to, like, we've got enough users on here to know what we need to go do, and we're going to more or less shut it off for a little bit.
• 24:40
  I mean, keep the site up, but we're not actually adding more invites. Is that or was Steve, lying to me this entire time when he said he was out of invites?
• 24:48
  Paul Frazee: Sorry, boss. That's a good lie. I I'm I'm gonna try to give an answer that doesn't blow that up.
• 24:58
  Bryan Cantrill: Steve is furiously DM ing you.
• 25:02
  Paul Frazee: Yeah. No. So, that's actually not, an unfair characterization. We we, everything moves so much faster than we were ready for. Like Yeah.
• 25:13
  We started, working on this protocol in 2022 at the beginning of the year. And then by the October of 2022, we were starting to realize and you may understand the timing if you can flip me play back every day that happened.
• 25:27
  Bryan Cantrill: 22. What was going on?
• 25:28
  Paul Frazee: Yeah. I went out and then.
• 25:30
  Bryan Cantrill: Yeah. What happened then?
• 25:32
  Paul Frazee: So by then, we were going, well, we're gonna need to make it up. And, suddenly, we were productionizing the protocol, which the org wasn't designed to do. We were not prepared really for that. And so Right. And and we were we were kind of initially we did have a a client, I think, already going, but it was really just like a test bed to make sure that the technology worked.
• 25:52
  We suddenly were hitting the gas to get features in there and get the servers to a place where they could start to accept users. And then once we started to have them on, you know, the single post era lasted, you know, for a lot of that private beta. Right? So there was a yeah. So there was a moment of Welcome
• 26:15
  Bryan Cantrill: to Oxide and Friends. You don't have to dig too far to get to the post cross era for anyone. So this is this is good. This is very on brand.
• 26:22
  Paul Frazee: Oh, if you don't start that way, you're doing it wrong. You definitely should. And because what the transition after that is into the solo DB and, you know, the event pipeline era where the engineering gets way harder. So you only wanna do that once you actually are scaling, but that's what we had to transition to. So, like, by the time we're in the fall of 23, we're looking at this and going, we we gotta do a a stop everything and rearchitect this thing, or we're not gonna be able to handle this growth.
• 26:51
  It was just the demand was too high. Yeah. So we were you know, the one thing I I know that Jay would want me to clarify about the invite period is that that was always it was a lot of there
• 27:03
  Bryan Cantrill: were some people who were like, it
• 27:04
  Paul Frazee: was about exclusivity. It was not about exclusivity. We weren't trying to make it like a hip thing. It was very much just that the servers could not handle the load, and we needed to get, like, our t n s team scaled up. And, like, that stuff takes time.
• 27:17
  We just weren't ready. So the demand was, like, way higher than than our supply. And that's what we were able to buy ourselves by doing that in my only period. And but, you know, we moved as fast as we could. It wasn't until February of 23.
• 27:34
  Oh, 23, 24 that that we could Yeah. Right.
• 27:37
  Bryan Cantrill: You're just rewriting it. You you you're in the post process in in the,
• 27:42
  Paul Frazee: All of
• 27:43
  Bryan Cantrill: 20 Yeah. That so the what I can say, I really admire the discipline of the team because when we I mean, at some level, and I know, like, it's blue sky is interesting because you're a public benefit corporation. You're not so there's there's a bit of both sides to this in the terms of, like, you are see you know, you're seeking to be to to make profit and to be a a financially viable company, but you are also seeking to serve the public good. So I know it's a little bit different, but I think it still takes a lot of self control around here when you because everybody wants product market fit. And Yeah.
• 28:18
  When you have product market fit, which I I think we could not not to disparage Secure, Scuttlebutt, or Beaker, but this was this was like, probably things that didn't have as much product market fit as
• 28:33
  Paul Frazee: Correct. Correct. The inventory. I see I see what you're getting at.
• 28:36
  Bryan Cantrill: Right. So you can see we're having this conversation about Beaker, but poor Beaker. The, but the you, you've got this enormous product market fit, And to have the discipline to say, like, if we don't fix this, like, we can't fake it till we make it, move fast and break things, you know, all these other kind of Silicon Valley mantras that, actually, you're gonna end up with something that's broken at the end of it. Like, you actually won't have something that works.
• 29:02
  Paul Frazee: That is completely correct. Yeah. It it is actually well, first of all, this is all pretty integrated. So the PBC specifically, what that does is it gives us space to pursue a mission in addition to the profit motive. And so the mission is to create this decentralized protocol for public conversation and and for applications for into it.
• 29:19
  Right? And that's actually just remains like the main goal. Product market fit, scaling out of having, a profit come in so that we can stay in the organization. That is still all in service of that mission. It's just the vehicle that we've chosen to do it by.
• 29:34
  But that, you know, was something that when we were sitting down to attack this project, we had all realized, like, okay. We are all trying to get a change in how technology works, but what's your theory of change? How do you expect to reach the market? And the way you do that is with a so all the mechanisms that come along with the startup are really we try to maintain that that is the vehicle by which we're doing accomplishing this mission. That is not that's not the mission itself.
• 29:59
  And either the the discipline around, like, the scaling side of it or the moderation side, that that was tough. But the other one that we maintained was making sure that the protocol was sufficiently
• 30:09
  Bryan Cantrill: at
• 30:09
  Paul Frazee: a one point o that, like, at least the self hosting part of it was active before we launched. We did not want to go into the public launch prior to having that set up because we wanted to make sure that we hit those targets for what this is about. So that was another thing that we delayed launch for to make sure that we were actually acting on the protocol. And it's kind of always that balance.
• 30:31
  Bryan Cantrill: Which I gotta say is another act of real discipline because I think that part of what makes this challenging is that you are trying to build Blue Sky as kind of an exemplar of what's possible on atproto. And it's in any of these things, it's tempting to just be like, just like, okay, can we just get like Blue Sky working and then we'll we'll we'll figure atproto out later and allow allow people to do this kind of on their own later. I mean, it must be, I mean, obviously, that's not a temptation that you all had because of the strong call to mission. But I just think it I I it takes an admirable amount of discipline to really say, like, no. We actually need to do this the right way, and we need to allow the we wanna be sure that that this thing is actually that app proto is the actual basis that we're building this on, and that that Blue Sky is button instantiation of it.
• 31:22
  I'm really impressed about that. We was there I mean, there must have been some level of tension of, like, hey, we've got a, like, we're missing a window of opportunity. Did people feel that way? Or did you feel that the window of opportunity did you know that your agent on the inside of x, mister Busch, was gonna continue to operate per your explicit wishes?
• 31:44
  Paul Frazee: No. We had no guarantees. What we're gonna have for sure. And there was certainly we had the 8, you know, a deadline if I said we didn't have that anxiety in the conversations about making sure that we're capitalizing on opportunities. But it I would never say that the, protocol or mission focus was ever in question.
• 32:05
  That was just something that this team never really is willing to throw overboard. You do get into some complicated conversations about capacity assignment towards protocol work versus product work, and they're definitely sometimes where we're kinda looking at the quarter and going like, okay. What are we gonna focus on? In fact, a lot of last summer
• 32:24
  Bryan Cantrill: Yeah.
• 32:24
  Paul Frazee: We were sitting there and going, you know, there's just a lot of features that are missing if we want to be, you know, what users are looking for. We just had a lot of requests and things that were absent. And one area that we did, I think, you know, make a I I I believe defensively pragmatic choice was with DMs. We knew that not having DMs was killing us. And the protocol is not ready for DMs.
• 32:50
  It's really geared towards public posting at the moment. So we're like, alright. We're gonna give ourselves a cheat on the DMs. We'll go back and clean that up later. But for all of the summer, yeah, that was like, we're gonna make incremental progress.
• 33:03
  We're gonna do a little stuff on the protocol, but we really gotta get this product shipped up because if it's not working then again that, like, then the protocol may not get its chance. Right? And I think Totally. We're in a better place. Yeah.
• 33:16
  I think we're in a better place now.
• 33:18
  Bryan Cantrill: I guess I I really Just speaking as a as a consumer, I really appreciate that choice because Yeah. There were very few one of the reasons that I was still sadly on the fucking bird site was because of DMs. And Yeah. Like, I actually do need DMs. And Yeah.
• 33:37
  If this it it DMs are a place and my DMs have always been open on both platforms. It's been very interesting to watch the the the supposedly bought free.
• 33:50
  Paul Frazee: I'm just gonna say wow.
• 33:52
  Bryan Cantrill: Oh, no. My god. It is just doing on blue sky
• 33:55
  Paul Frazee: because I'm not sure our DMs are bought free either right
• 33:57
  Bryan Cantrill: now. New blue sky was arriving. I thought it was great when blue sky. It's like, oh, I got some I got, like, unsolicited crypto and and, oh, without noise and some porn is here too. It's like, this is great.
• 34:06
  Like, this guy's gonna make it. This is so this is so performing.
• 34:08
  Paul Frazee: Yep. My idea is to
• 34:09
  Steve Klabnik: monetize mods, is just like 3 quarters of them is like, report a spam, report a spam, but it's so essential for the times when it's not that though. So, you know, it's good.
• 34:19
  Bryan Cantrill: Yeah. But we'll work on that. And I also feel it's like DMs are I think that you all made the right call there and that, like, DMs are a good place where it's like, yeah, I like yeah. I I I I try this does not need to be a part of AdProto for this. I actually just need, like, the Blue Sky app to have this functionality.
• 34:37
  I guess you all have done such a good job mimicking and Steve did the right so Steve was a Twitter dead ender until he wasn't. And I thought, like, there was a time Steve's like, yeah. I'm like, I'm just I'm going down with the ship, man. Like, I I just don't care how bad this place gets. And I'm like, I don't know, man.
• 34:55
  I'm kind of and then I'm I'm like, you know what? I am I'm gonna go down with the ship too. And then I look over and I'm like, where's Steve? Steve's gone. And I'm like, Steve is Steve has, like, deleted his Twitter.
• 35:05
  Like, Steve is gone gone. And I'm like, oh, okay. This is actually I I maybe I'm being liberated to get to get gone gone. So I mean, I I desperately wanted to get off of that thing, as it was just, like, absolutely descending into the muck. But I the the thing that has happened to me a couple times, and I don't know if this has happened to you, like, I look look, I would love to tell you that I've been off for a long time, but it's a process, man.
• 35:29
  One day at a time. Okay? Like, every day is a struggle. Some days I gotta call my sponsor. And the number of times that I have you've you all have done such a good job on just, like, the look and feel.
• 35:41
  The number of times I have been, like, hey, you know what? Actually, looking at these these posts are actually surprisingly good for the hellscape that Twitter is oh, no. Wait. No. I'm blue sky.
• 35:51
  Or the then then the the the the, the also what it has has often happened is that I will get a a you know who's not on Blue Sky? Only the good VCs are on Blue Sky right now. This is not gonna this is where this will change very shortly, but the but the VCs that really troll the hell out of me are all, like, have shackled themselves to the mass of Twitter, and I I just don't have the self control to not actually go and and, like, go to the rage bait. But then I will the the the the the inverse also happens where, I will accidentally leave a Twitter window up and and I'll be like, oh, man. Blue Sky's probably lose.
• 36:27
  So you've done an excellent, excellent, excellent job of
• 36:32
  Adam Leventhal: And I would say as distinct from Mastodon where you knew you were there because it was boring as fuck.
• 36:39
  Bryan Cantrill: Yeah. Mastodon. I the problem with Mastodon, I love Mastodon.
• 36:44
  Paul Frazee: Do you?
• 36:45
  Bryan Cantrill: I love aspects of no. No. No. I don't love Mastodon. I I was trying to find this is, like, where, like, I'm I'm sorry for not being sorry.
• 36:52
  I love the idea of loving Mastodon. That's what I love.
• 36:55
  Adam Leventhal: Yeah. Oh, yes. Right?
• 36:57
  Bryan Cantrill: The but Mastodon is basically like has got the usability of a BBS, and I love BBS's. The Paul, I'm not sure how much that that that this is like modem era social networking. The but the problem was the and and this is what I I part of why it's really important to and I thought you all did actually a really good job of talking about, like, why not activity pub and why not Mastodon? And there's some really good reasons for that. And the and I think that Mastodon should I mean, I think it's gonna be We're gonna continue to be on Mastodon.
• 37:34
  I think it's great. I think that it it serves a great purpose, but the problem is that the, like, I'm a I'm a I am a baseball and football fan, and I like social networking because I like sports Twitter. What was sports Twitter? And, like, you're just never going to end up with that on Mastodon. It's just not gonna happen.
• 38:00
  And you need and also or or or Paul, the the shake test. Are are you I'm not sure if you, so the the real acid test for social networking is like, if you're in the Bay Area and you feel that you just felt a quake.
• 38:14
  Paul Frazee: Oh, yeah. Yeah.
• 38:16
  Bryan Cantrill: And, and I I'm sure you know this, but, the, you should know that blue sky has passed this test with flying colors because we didn't just have a quake, we had a tsunami warning recently. Yep. And the, my phone, like every other phone in the Bay Area, exploded with what I assumed was an amber alert. You're like, tsunami warning on a quake I didn't feel. Like, my first thought is like, this is like an IT error somewhere.
• 38:44
  This is like a missile alert in Hawaii. And, but it's actually obviously real and it was what was really interesting was blue sky was just just exploded with, you know, people talking about it, talking about the the, tsunami know your zone site, is down. You can't get a database handle to know your tsunami zones, and no one can figure out if they're gonna die to tsunami or not. The and Blue Sky and and what I also really loved is then my geosciences feed on Blue Sky was great. I mean, it was it it and the and meanwhile, on the other side, it was just like they're all it's all garbage.
• 39:25
  So they I mean, it was it really passed this task with flying colors. Nice. Could you talk a a little bit about the feeds? Because this is something that, again, I've really I I think a few I've started to make use of, I haven't done enough with, but I absolutely love, and it's really kind of a first class thought in the way you built the thing.
• 39:43
  Paul Frazee: Yeah. Yeah. It is an interesting area of the product that actually, we're gonna be investing even more in in the future. I'm really excited to to get into it more. Yes.
• 39:54
  So, like, feeds gosh. The the basic idea with feeds was just how could we I'll I'll tell you the real mentality to it. We knew going in that we liked some of the ideas about, like, improving how there's back pressure to the organization beyond just at the really kind of systemic level. Because, like, the entire protocol that we've created, it makes it possible to make alternative applications and have alternative hosting, but those aren't, like, super tangible in the day to day to people's lives. So we sat down early on, and I was like, okay.
• 40:23
  Well, what could we do with this that would allow people to more tangibly experience an open network and and decentralize the what is, like, meaningful to the average person. And so then the two answers that came up were could we decentralize moderation in some way, and could we decentralize algorithms in some life? Yeah. And so the feeds came out of that. And the feeds, I think that they can I I there are ways that I think we can improve on them and make them even kinda more central to the experience and really get the most juice out of them?
• 40:58
  But I will say that from a kinda, like, validating the concept perspective, man, they totally struck gold right from the get go.
• 41:06
  Bryan Cantrill: They absolutely struck gold. Yeah.
• 41:08
  Paul Frazee: Yeah. And I'm really pleased about that. It's actually it's really fun how they work. They are hosted on other people's servers. They do not run on our systems, but they feel integrated into the app.
• 41:21
  And with that, we kind of invented this whole principle that we call 3rd party as first party. But this idea of being able to integrate in systems posted outside of the application, but make them feel native. And so the, way they work is the simple answer is that you contact the feed server, and then it just sends back a list of URLs. And then those list of URLs get hydrated into the feed.
• 41:44
  Bryan Cantrill: But the way they have those URLs is
• 41:46
  Paul Frazee: they tap into that fire hose, and they use whatever kind of algorithm that they wanna use to decide, which posts to select, to make the feed. And that ends up being a really nice compositional, boundary for, that thing. They can use any kind of logic they want from social graph, you know, queries to, computing ranking off of likes and reposts and other kinds of signals. You can get, you know, text queries running in there, which a lot of them are. A lot of them are kind of operating off of, like, a smart hashtag in a way where they're just looking for particular topics that are being discussed.
• 42:22
  Honestly, you can get really far with just social graph limitations plus text query. Right? Like, the basic logic being, these are these are people that are experts in this topic, and here's the topic, and let's get a feet going. It must have
• 42:35
  Adam Leventhal: just felt so right. Like, it just it just feels like so spot on from an architectural perspective, from a usability perspective. And it's it's gotta clear up both the some of the the, the real concerns around Twitter, around Facebook, around threads, around these other social sites where you're like, as as you say in the paper, they're prioritizing engagement over all else, like, both of the mental health of the users. And on the other side, you got Mastodon saying, look, there's algorithm is is time stamp ordering, which which, you know, leads to a less than simpler
• 43:09
  Bryan Cantrill: Time stamp ordering of the people you follow. Yeah.
• 43:12
  Paul Frazee: It's like,
• 43:12
  Bryan Cantrill: oh, you're like, you're interested in in the tsunami that may be coming to your place in an hour? I don't know. You should follow more geologists. Maybe in the next life, you'll follow more geologists.
• 43:22
  Adam Leventhal: You should have done that in time for you
• 43:24
  Bryan Cantrill: to You should have done that. You should have been carefully following geologists. You're like, okay. No, I can't I
• 43:30
  Paul Frazee: got I got a time out and give, like, you know, I'm glad Mastodon is there pursuing the same mission. I'm just gonna say that, and then we could move on.
• 43:37
  Bryan Cantrill: Yeah. Yeah. I look look.
• 43:38
  Adam Leventhal: I I I think I I continue to to like and use Mastodon. I think I actually like it more than you do, Brian. I wouldn't say I love it, but I still like it.
• 43:45
  Bryan Cantrill: I was one of the most famous Mastodon. You can't say But we know that I actually just love the idea of loving Mastodon.
• 43:50
  Adam Leventhal: Yeah, exactly. And actually for a while, I liked that it just helped me put down my phone and not keep on scrolling. So there was that benefit too. But, but it just feels like it plugs such a nice hole. It's a it's a really lovely, like, design that you have.
• 44:08
  Bryan Cantrill: Well, in the feeds also, Adam, what you're saying about, like, it's one of those ideas that you know is it is an important idea whose time has come because you're immediately, like, why are we doing it some other way before? Like Yeah. This is Yeah. It it it you know what I mean? It's not that it it not at all obvious because it obviously wasn't, But the fact that you I mean, it's one of these things where you make Paul, you all made a bunch of good decisions that built on one another, and then out of this comes this, I think, artifact that's gonna be, I think, more and more important as time goes on.
• 44:38
  Because I think that, you know, one of the the posts I've seen is that that one of the great things of Blue Sky is that you get to bring your entire self because all of us consist of like different things and different things at different times. And the feeds allow to allow Give us agency over that, which I think is extremely important.
• 44:59
  Paul Frazee: Yeah. I mean, I I have to agree. I mean, there there are also times where I'm just kinda like, you know, today, I'm in the mood for the more quiet thing, you know. And that's I've got feeds for that, you know. Or today, I'm actually, like, voraciously consuming stuff.
• 45:11
  I've read all my feeds, and now I'm going over to the one that shows me more of the fire hosey kind of experience. Like, just whatever you're vibing on, then you can you can get there.
• 45:20
  Bryan Cantrill: John, I also do
• 45:21
  Steve Klabnik: the by
• 45:22
  Bryan Cantrill: the way.
• 45:23
  Paul Frazee: I was just gonna say Skype. I was just because yeah.
• 45:28
  Bryan Cantrill: Steve, could you explain what that feed is? Because I I I love that, and you had talked about it in your post about how ad product works.
• 45:34
  Steve Klabnik: So quiet posters is basically a feed that takes the people that you follow who don't post very often and only shows the posts from them. So, like, you know, some people who, are mentally healthy don't post as much as someone like, say, me, who is horribly broken as a human. And so if you wanna see those, like, good, healthy posts, so the people that post say maybe, like, once a day instead of once every 10 minutes, you can just load up, quiet posters. And that way you can, like, keep up with friends who, you know, maybe get, like, lost in your stream because you're following too many people or, you know, whatever else.
• 46:06
  Bryan Cantrill: I love that, Steve. Yeah. And I I think it's it's so and I think we're just gonna see a lot more experimentation in that. And, again, giving people more agency. You know, Adam, perversely, what it reminds me of, and sorry, Paul, just to go overly gen x on you here.
• 46:21
  But the the day MCA died, I wanted to just listen to all of his music from beginning to end. And Pandora is like, let me give you something that sounds like the Beastie Boys. I'm like, my man is dead, Pandora. I'm not interested in your algorithm right now. I am mourning a a dead musician.
• 46:40
  You need to actually give me and and that's when I'm like, I'm going to Spotify. I'm not coming back.
• 46:45
  Adam Leventhal: I I'm I'm a little embarrassed that what what Steve's anecdote just triggered in me is the recollection of on Facebook. I had a friend who would paste who post every 20 seconds. And I only just now remembered that I muted him 15 years ago. So I need to go take a look at Henrik and see if I can unmute him.
• 47:03
  Bryan Cantrill: Yeah. You've he's lived this extraordinary life that you've that you've missed because you just, like, couldn't
• 47:08
  Paul Frazee: take it.
• 47:09
  Bryan Cantrill: I just couldn't take it anymore. It's like mister peanut on Twitter. I finally muted mister peanut because I couldn't take the ads. Jesus. So the I mean, Paul, this must have been one of these breakthroughs that you just kind of like this feeds must have feel felt like a lot of things were clicking.
• 47:24
  I mean, it's, such a such a I mean, actually, let me ask this. Did you kinda know immediately, like, oh, of course, this makes a ton of sense? Or was it when you started experimenting with it that you saw it?
• 47:36
  Paul Frazee: Yeah. I mean, I, I wish I could say it was like, You know, like, run it away. I will say from a technical point of view, it felt really good. Actually, this is this is we're kinda talking about, like, the the experience of, like, the engineering side and the product side. I'll I'll just actually tell the whole story.
• 47:51
  Like, we initially did not do custom feeds. Initially, we did something called scenes. And, scenes were like a hybrid between a community and a custom feed because we were trying to we knew what we were aiming for, but we didn't quite have it right. And the way that a scene worked was that the members of a scene when some number of them like to post, it would trend automatically within the scene. So you can kinda see there's some approximate thinking there, and you would actually get, like, a notification whenever your post trended within a scene.
• 48:23
  And we obviously didn't think this through very well because people would create scenes of, like, 3 people. And so then suddenly you're getting, like, your post is trended. Like, the same 2 people would like a post, and then it would trend in 5 different scenes that they're all in with just 3 people in it. So that wasn't brilliant. But the other thing that also jammed that up completely was nobody understood this idea.
• 48:45
  It was, like, weirdly, like, both a community and not a community, and and people were like, I'm gonna post into it. You don't know you don't do that. It just shows things that you're liking. So, like, we did that first, and I probably did very many people seeing the product. We pulled it out.
• 49:02
  We just like, this is this is quite terrifying. We pulled it out. And, we brought in the custom feeds thing, as an alternative. I think, in many ways, it was kinda taking a bigger, swing on a technical level because it is yeah. Because it it it's, you know, it's a more sort of nuance.
• 49:20
  Like, we're we were, at that point doing a at runtime request out to a 3rd party service, which in and of itself, I think, is going to be an interesting thing to kinda see how that plays out because you you know, anytime you're doing that, you gotta talking about, like, out to these services that are not under you know, we have no SLA with them or anything like that. They're what how's the performance gonna hold up? Things like that. So far, it seems to be actually working. But, that definitely was a little bit more of a swing.
• 49:46
  And then once we got in and saw it working on
• 49:49
  Bryan Cantrill: a kind of design of the technology side, we were like, yeah. Okay. This feels
• 49:50
  Paul Frazee: pretty great. And the the compositional side, that that felt good right from the get go. I think we still have some open questions about how we can really make the experience of using them as good as we want them to be. We have this, like, sticky problem around the management of them where there's more feeds that are interesting than you could stick onto your home tabs. And, how do we find the right way to surface them to, like, easily, like, oh, right.
• 50:22
  Right. I've got this great feed over here. I wanna go to it, especially on mobile. So we're still, I think, picking through the right UX to really make these things, shine. Yeah.
• 50:34
  It's, I would say that somewhere around maybe a year ago, it was starting we were starting to kinda realize, like, no. This is this is, like, pretty solid, though. Like, even though we had gripes about it, we were pretty happy with the outcomes that we're that we were getting out of.
• 50:50
  Bryan Cantrill: I yeah. I think it's I agree with you that there's some there's some UX things that you could probably do to to, but I think that the the the bones are really, really sound on this one. And I because I also think that you've got the thing that I also loved about it just, like, being able to follow my geosciences feed and my earthquakes feed during the tsunami warning, is that you are able to I mean, there's so many, but part of what made Twitter great when it was great was you would stumble into one of these, like, delightful communities and of, like, super nerds. And, you know, suddenly you've got, you know, top scientists in the field who are, I mean, just absolutely nerding out and going very deep on a particular subject. And it's very nice to be able to, like, when you find those, you feel like you tripped over something.
• 51:36
  And I think it's like feeds are gonna make it way easier to find those. And Mhmm. I just think that, like, the there's so much we can go do to find amazing feeds. I also want, I want feeds for podcasts too. It's like my own now we're getting into my own, like, weird agendas, but, like, podcast search is so bad.
• 51:57
  And I I I I I, I think I mean, I want feeds for other things in my life, not just not just posts, not just skates, is what I gotta say. Oh, I think it's amazing.
• 52:08
  Paul Frazee: Yeah. I got 3 things.
• 52:09
  Steve Klabnik: 1, scenes sound amazing. 2, I understand why you removed them. 3, naming a feature scenes is not beating the blue skies made for 38
• 52:18
  Paul Frazee: year old's allegations. Like Yeah. Yeah. Every time somebody comes in with there are 2 critiques that feel really personal, and they're not really critiques observations. And that is that it's elder millennials, and then it's, theater kids.
• 52:33
  And seeing as I'm both of them, I every time I'm like, how does this happen? I got a I got a month left
• 52:39
  Steve Klabnik: of being 38, but I've only been 30 theater kid adjacent. So I get, like, hit 1.5 times instead of 2 x on that one. But yeah.
• 52:46
  Bryan Cantrill: Yeah. But, yeah, but, yeah, you know, you're gonna turn 39 when all the other elder millennials turn 39, Steve. You're not beating the allegation here. This is, you know. Listen, Paul, this is why we're trying to talk about, you know, like, you know, the the BBS's and, you know, the Beastie Boys just to, like, just to just to pull out just get Gen X here, pull out it a little bit.
• 53:02
  Paul Frazee: I appreciate that. Yeah. Yeah.
• 53:05
  Bryan Cantrill: But so feeds were obviously, I think, really important. I think and I think feeds are gonna be going to be even more important. I think it's gonna be I think we're gonna see a lot of creativity. I just I don't know. I'd I'd be interested to get your take on it, but I just feel like, boy, there's so much stuff that's possible there, especially when you go to look at things like LLMs and you've got I mean, just like there's a lot you can go do there that hasn't been done yet that I be really exciting.
• 53:30
  The It it just
• 53:33
  Paul Frazee: so to your point, our biggest problem is that we have too many ideas about what we wanna do with them. And it's hard to pick about, like, where to take it next. It's and then then we finally started to make some progress once we, like, broke it down into, like, bite sized chunks. But I yeah. I agree.
• 53:46
  There is a lot of really cool ideas that we're kicking about what to do with seeds.
• 53:50
  Bryan Cantrill: Well and I think that also, like, part of what I what I'm really excited about is that I think that, you know, we we kinda had this era of, like, maximum engagement. And has we are seeing kind of the logical conclusion of that in in Twitter. And, and I think, I actually want a world where we're actually able to find the hidden nuggets a lot more easily, and we're able to find that that, like, the delightful bits that are at the edges, are to me way more interesting and way more delightful. And I just feel that, like, feeds are are an important way of getting that stuff. It's just like the quiet feed, Steve.
• 54:25
  It's like, get let's like, let's get to the people that don't say very much, but how when they say something, it's meaningful.
• 54:33
  Adam Leventhal: The so I I hesitate to mention like the R word, but it seems like there are either third party revenue opportunities. Like, would people pay to like get an improved feed, whether it's from a first party thing from blue sky or others? Like, it'd be I'm sure it'd be great to see I mean, obviously, you folks making money off of it, but other folks making money off of it as well. Meaning, meaning the meaning the platform writ large.
• 54:57
  Bryan Cantrill: Yeah. Yeah. Yeah. I I do think by the way that there
• 55:03
  Paul Frazee: Go ahead. Bro, I actually
• 55:05
  Bryan Cantrill: do think that there are a lot of opportunities for I think people will pay for things that people will will pay for things that add value, and I think there is opportunity to monetize a bunch of the stuff. Not not to boy, this is, like, our Adam, this is where we our our inner VCs are coming out. What's what's wrong with us? We're broken human beings.
• 55:22
  Paul Frazee: That's right.
• 55:24
  Bryan Cantrill: Oh, you do have to think about
• 55:25
  Paul Frazee: that stuff, especially, you know, when you have other people making this thing, you wanna think about what the incentives are, you know? So that and and if they can make money, that'd be great. You know, I'm I we we wanna make sure that that the whole thing sustains itself.
• 55:37
  Bryan Cantrill: Yeah. Totally. So, August, you mentioned that so feeds, and then the other thing that you wanted to real that that you were rethinking kinda at the same time was moderation. And what was your, obviously, another, I mean, just age old issue in social networking and now a hotter issue than ever, just with our Okay. Direct device of error.
• 55:57
  What was your thinking on moderation? And next thing, when when you rolled out the beta, there was, like, moderation was still pretty primitive in terms of the tools that were offered for folks.
• 56:05
  Paul Frazee: Yeah. This one is hard, and it's gonna be probably a very, very long time before I'm willing to say, yeah. We totally nailed that one. There's gonna be a lot of trying different things to find out where it can go. We've gone I think we've we've taken some interesting stabs at it, and I'll I'll kinda get through, all that in a second.
• 56:25
  But the, I think there's probably many more, things that we have to play with. Probably speaking, the the core of it, I think, the thing you could just gotta realize is that you really you cannot make the world happy with a one size fits all moderation at this point. It's just it just can't be done. And I think a big part of it is that, like, people come into the world with different values and, expectations about how those values should be upheld. And, and I think it's very, painful to see, those values not being reflected back in the spaces that you're in.
• 57:00
  And it's a you know, I think it's quite important that we find a way to to get there so that communities can you know, I kinda really feel this is really at the heart of, like, why social media can be so painful for in general is that we just haven't found the right way to do this for large, scale, you know, public spaces like this and what you might, you know, in the kind of public square mentality about social networking. So anything we can do to get us closer to that is, I think, kind of the right sort of challenge to be trying to dive into with this sort of stuff. What where can you move forward the experience of, really, the Internet at large would be finding a way to create these these spaces that I think, give people people a better experience with that element of it, the the moderation, and then, in some ways, curatorial or editorial side of things.
• 57:48
  Bryan Cantrill: Would but I also like I mean, like like feeds, you're saying, like, look, we're not gonna there's not gonna be one way to moderate. There's gonna be Right. Can you talk about that approach a little? Because I think that's a really interesting approach. We're, like, we're going to allow people to moderate in different ways and allow people to give people some some transparency and some choice among different approaches to moderation.
• 58:07
  Am I I mean, probably being overly reductive there. But
• 58:11
  Paul Frazee: I mean, that that's that's probably right. I mean, the the basic thing is, first of all, we start from the assumption that even if we did come up with, like, a really great execution of, an algorithm or moderation or things like that, You have the long term to consider and, like, do organizations stay effective at what made them what made them successes in the first place? And so coming into this, we have always gone gone in with that belief that the the answer is no. Right? That you kinda have to be planning for when our organization or any other organization starts to, wither from its original purpose or or just maybe not be fit for the time anymore and be ready for for, like, the world to be able to change without too much loss.
• 58:52
  Right? That's really deeply embedded in everything that we do from the application side of it to the, you know, to hosting side to the moderation of the algorithm. So, as much as we do our best to try to create good versions of, like, algorithms and good versions of moderation, we just kind of know that, first of all, long term systemically, you gotta be ready for that to not always be the case. Beyond that, I think it's just quite clear that, like, people have very different tastes when it comes to both algorithms and moderation. And people have different needs for this stuff.
• 59:27
  And so, how can we get everybody to a place where they are experiencing the life that they wanna live on the Internet? And that's only gonna happen if we start to allow, you know, a a company like ours where it's kinda challenging position of trying to, of balancing a lot of different voices. Users don't have the same challenge. They can just choose to be a parts of communities that actually more connected to what they personally want as opposed to what an organization wants. So we should enable those groups to be able to get that specific with what their with what their communities are like online.
• 1:00:03
  That's the
• 1:00:04
  Bryan Cantrill: basic concept. I have to say it it's kinda remarkable. And this this as you said, it permeates your thinking of, like, thinking and you call it the credible exit in, in your paper in terms of thinking beyond blue sky organizationally, I think is is, kind of remarkable and unusual, I think, for an organization to think, we need to build in the things that will survive an arbitrary change in management and arbit I mean, which is kinda I mean, it reminds me of open source in that regard that where, you know, we are open sourcing this to assure basically that it it that it can endure. It can endure beyond its corporate vessel. And, you know, not that Adam and I might be speaking from personal experience a couple times over, but, the corporate corporate vessels don't endure, folks.
• 1:00:52
  The, except for this one, of course, for whatever one you happen to find.
• 1:00:56
  Adam Leventhal: For eternity. Right.
• 1:00:57
  Bryan Cantrill: For eternity. Exactly. So I I I think that that that is, the the how do I boot some model off of a discord? Larry Ellison buys blue sky. Who would say such a terrible thing?
• 1:01:09
  I, unfortunately oh, my I'm I'm so sorry, Paul. This is the the the demographic we attract. You know, these are the, but the I think that, like, being able to kinda think beyond yourselves in that way is is really pretty interesting. Can you speak to some of the specific tools that you booked? Because I I mean, in particular, like, I love the ability to detach a quote tweet.
• 1:01:31
  I think it's a really I mean, that's not one that I've seen before. Yeah. Could you talk about some of the stuff that you've done to just give some tools out there for folks?
• 1:01:40
  Paul Frazee: Yeah. I mean, generally speaking, it's it's funny. Honestly, like, moderation is a tough job as you might expect. And things like that, I don't know why nobody else has done it before because, gosh, anytime people can fix the problem for themselves, that actually makes our lives easier. So we have a pretty good incentive to do this sort of stuff.
• 1:02:00
  Removing the quote posts, that one was like, we didn't do it straight away, but that was one that I had on my list personally right from the get go. Because let's just we all know that the quote quote dunking is one of the big generators of just some of the most painful experiences on Twitter at least. And I imagine anywhere else that has a similar dynamic. And so the detaching of quote post is essentially, like, it it's it just takes away whatever the quote quoting post is. It just detaches it.
• 1:02:28
  It just whatever they've quoted, it now just says content no longer available. And, so now, you know, of course, what people do instead is they'll screenshot whatever it was that they originally
• 1:02:37
  Bryan Cantrill: Of course. Wait. Yes.
• 1:02:37
  Paul Frazee: Of course they will. But that's actually still better because
• 1:02:41
  Bryan Cantrill: one of
• 1:02:42
  Paul Frazee: the things that you gotta think about with large scale social spaces is friction. Friction matters so much. It matters so much. And when you can introduce points of friction, you're actually able to because it is such a game of numbers, you actually will dramatically reduce bad outcomes if you can just slow people down. And the reason that the quote posts are so painful whenever their quote dunks is that it sends a hoard of people with one tap into the replies to start dogging you for whatever it is that this you're getting dunked for.
• 1:03:11
  And, honestly, if you've ever I've I've witnessed a a friend of mine get a really aggressive, like, raid army kinda directed at them with a quote tweet back in, like, I wanna say, 2017 or 2019, somewhere around there. And, like, you experienced that once, and I think you'll realize, like, oh, this is you gotta have something to slow this down. Yeah. So yeah.
• 1:03:37
  Bryan Cantrill: Adam actually once gave me the book, so you've been publicly shamed, which I would I it's true. Highly recommend it. Not necessarily being publicly shamed, but, Adam obviously thought of me immediately when he saw the book. So I I think that's, yeah.
• 1:03:50
  Paul Frazee: Nobody on this voice call
• 1:03:51
  Steve Klabnik: has been publicly shamed at all. Definitely not. Never heard of that before.
• 1:03:57
  Paul Frazee: Yeah. I mean
• 1:03:59
  Bryan Cantrill: It it it's so are there other things that, like, you all did like that? I mean, I think it's, like, one of the things that I think is interesting is that, mute lists and block lists are public, and I I I'm sure this is an issue that is controversial. Oh, yeah. I not not to have you weighted to every issue of controversy, but I, I mean, one of the things I really didn't like or I thought was lame about Twitter is when you had very powerful people that would block people that they disagreed with. And I was like, that's not kinda not what it's for.
• 1:04:30
  It's a Marc Andreessen, the blocking functionality is not so you can block me. Sorry. This is like, not to not to overly personalize it, but I've always been like, I've always wanted to be like, hey, so I find Marc Andreessen, you get to block little old me. I also like I kinda wanna follow everyone that you're blocking, so because I think I would like like them a lot. So I just wanna, like, have a feed of people that Marc Andreessen blocked.
• 1:04:55
  And that requires public block list. There's nothing I can feed. I know
• 1:04:58
  Paul Frazee: the Oh, I'm I'm in
• 1:05:00
  Adam Leventhal: the brotherhood too, and I have no idea why. But, yeah, I'd love to I'd love to get together for meetups and stuff too.
• 1:05:06
  Paul Frazee: Oh, but no, he no longer That's not why we
• 1:05:09
  Bryan Cantrill: oh. Why? Yeah. I I know. So do I get do I get kicked out?
• 1:05:13
  Because I actually
• 1:05:14
  Adam Leventhal: You're out of
• 1:05:14
  Bryan Cantrill: here. I'm out. Yeah. No. It's simple.
• 1:05:15
  Yeah. Of course, you're out. Like, we can't you can't you read the sign?
• 1:05:18
  Adam Leventhal: So, like, I'm nervous. Right?
• 1:05:20
  Bryan Cantrill: It's like so but, yeah, get out of here.
• 1:05:25
  Paul Frazee: Well, we didn't make them public for that reason. I'll I'll start. But they they actually they're the blocks are mutes are private, blocks are public. The reason that is is actually kind of a limitation of the design of the protocol because you need the blocks to be public for the applications and really all applications to be able to respect them.
• 1:05:44
  Bryan Cantrill: Right.
• 1:05:45
  Paul Frazee: And, you know, so we we you can do the mutes privately because the only the application needs to know. It's just filtering things to the recipient, but the blocks kinda had to get broadcast. And you could come up with a privacy scheme for that where you just give the blocks to the applications in the world, but there's no well defined set of applications without creating some kind of a organization which has, like, membership. And at that point, you're essentially starting to create a cartel around the protocol. And maybe that's where all this goes someday, but for now we're not about to start that action.
• 1:06:17
  So we just said, you know what? We can't find a better way to do blocks. They're gonna have to be public. So that's why they're public. That said, there's a lot of other interesting things about blocks and and the fact that we act we, you know, we implemented what we call moderation lists.
• 1:06:31
  So these are lists that you can mute or block. Implemented that pretty early on, again, as a part of the, of the the applications kind of, decentralized moderation. And we also made blocks really aggressive. Sometimes people call them the nuclear block. Mhmm.
• 1:06:48
  Where not only do they sort of like in Twitter as I recall, it would sort of freeze the interactions between you 2. So you would no longer see each other's stuff, but, like, your previous interactions would just kinda be frozen in time and public. Blue sky, it actually causes all interactions previously between you to also essentially go away. They just show content not available. And so we actually have a very sort of, really aggressive, toolkit for blocking, and we're encouraging people to put together a block list and to subscribe to them.
• 1:07:24
  And, again, it kinda gets into the point of view that we have, which is just that, there are just some people who can't coexist very well. And, I don't think trying to force them to do that is a great idea. I think that's why you get so much toxicity in social networks. And it felt even more pressing when we're creating this open design and sort of, like, loosening up the control that we have over participation. We wanted a way to articulate really aggressively, almost like your own perimeter since the website is no longer a perimeter for the social network.
• 1:07:59
  Being able to say, no. This is kind of a group that I'm keen to be hanging out with. And if you're in this list, I'm not I just don't wanna be in the same public space as you. And it comes with pros and cons. You know?
• 1:08:11
  The I won't say that it's like a pure win. Sometimes it could be really frustrating for people. I'm quite sure. But I think it's important that people are able to make that choice for themselves. I don't think you I don't really like the idea that people should be forced to share space with each other.
• 1:08:27
  So that's the mentality.
• 1:08:30
  Bryan Cantrill: And I mean, obviously, you've seen that. I mean, as the popularity and it's just it must be still a bit surreal to watch the popularity of Blue Sky really stumble into, like I mean, not stumble, but really become I mean, to like, for whatever reason, Adam, you said ESPN has got a Blue Sky account now. I mean, I just felt like, wow, that is Blue Sky has really arrived. And I but I presumably with that has become I mean, they're just you get a lot of people who are trolls in the in in the the the kind of the oldest sense of really deliberately wanting to get a rise out of people who are now attracted to the social network because of its very success. I mean, I assume is that fair that you've seen, more users, more problems, I assume?
• 1:09:18
  Or, oh, yeah. Oh, yeah. Yeah. Yeah.
• 1:09:22
  Paul Frazee: Now now we've we're tracking grades. Right? Like, the persistent automated, like, harassment campaigns. And so we've had to really aggressively scale scale up our tooling to deal with that. And, you know, that is, thankfully, we designed the whole system to be prepared for that.
• 1:09:40
  We just had to get the tooling in place, but then you, you know, on a long enough timeline, we all know the game. Like, the bot fight is hard, and it just got a whole lot harder with AI. And we're starting to putts around with a couple of things that might help with that. But, yeah, somebody some jerks made, AI bot army that just would disagree a little bit with people. That was their angle.
• 1:10:03
  Bryan Cantrill: Oh my
• 1:10:03
  Paul Frazee: god. That's Oh my god. Thanks a lot. You know? You don't
• 1:10:07
  Steve Klabnik: it's some real stick o stuff. Like, they know how to make people upset, which is not like a giant, like, fuck you at somebody just to be like, you know, you make good points, but I think you're just a little wrong about
• 1:10:17
  Paul Frazee: this.
• 1:10:17
  Steve Klabnik: And it's like, oh, man.
• 1:10:19
  Bryan Cantrill: Yeah. Yeah. Is is it possible that their training data is only older siblings in the are they are they God, I know.
• 1:10:26
  Adam Leventhal: They've tapped into something deep in the older sibling brain brain.
• 1:10:29
  Bryan Cantrill: The older siblings like, it would mean because older siblings do and I look, I say this as an older sibling. I mean, you just like, you optimize the least amount of effort for the most amount of of of reaction, and it's like yeah. I mean, it's a they're very, very good at it. Especially, you know, when the I when my oldest has, like, the countdown to when my 17 year old is going to crash out is the new term for, like, going it's just like, it's, you know, it, so I think this is very deep in the human psyche is what I'm saying. It's like you're trying to contend with something that is, very, very deep.
• 1:11:05
  Oh, yeah. And so, I mean, how do you I mean, you said that, like, the the the things that you've deployed seem to be so far holding up. Have there been any surprises there in terms of or maybe you've just been surprised by the degree by the kind of the scope and depth of some of these efforts?
• 1:11:21
  Paul Frazee: Oh, gosh. I wouldn't say it's terribly surprised. I mean, it's discouraging. You know? It definitely makes you go, I I wish better of the world, but we all know.
• 1:11:30
  It's a bit of a dark forest. So, no. I don't think I'm terribly surprised right now. I I I'm still, I think, in the zone of, like, yeah, I think we could do novel things. And I'm, like, it's it's only speaking somebody that that builds things, like, it's only once I'm out of ideas completely and, like, feeling defeat that I'm really, like, bummed out.
• 1:11:51
  But I'm still I still see some moves that we can try. So I'm I'm I'm feeling good.
• 1:11:56
  Bryan Cantrill: Well, that's great. And I we think also is this where I I presumably, the at proto is helpful where you can think, like, this is not, like, your fight alone that there are other folks that can, like, start implementing some of their own ideas. I mean, if you in terms of, like, getting into the the, like, the the the technical details of how those moderation servers actually work. So I think one of the interesting things the observations you had is that, the skill set to, like, run one's own server and the skill set to moderate are don't overlap very much. Right.
• 1:12:29
  Yeah.
• 1:12:32
  Paul Frazee: Yeah. I mean Can you
• 1:12:33
  Bryan Cantrill: speak to some of the the innovation that you're kind of, I mean, are you seeing some innovation there as as kinda third parties experiment with different ways to to moderate?
• 1:12:43
  Paul Frazee: Yeah. A bit. You know, the, we are starting to see researchers just sit on the fire hose, and there have been a couple of catches they've made that we didn't make, and that's awesome. That's, like, you know, that's that's a a a great benefit. Hasn't gone, like, crazy on that front yet because we're we're pretty active in there.
• 1:13:02
  So we're, you know, thankfully, it's not like we're getting a ton of, like, oh my gosh. I had no idea. But we are starting to see some folks do it. In terms of actual application, interestingly, I don't think I've seen anybody deploy a bot labeler that I think is, like, totally crushing it yet. But one of the things that we did put in is the ability to to run what we call labelers.
• 1:13:26
  And labelers are essentially moderation services on the network. It is how our moderation works. When somebody else is running it, the application privileges are moderation so that, like, only ours is able to do what the application considers to be a full takedown. But the, other the user run or what we call community labelers are still able to do some pretty direct interventions into the application in terms of hiding, and filtering out content. And where we do see, like, really good pickups on that, I think, or, like, for instance, the screenshot labelers for, screenshots from, like, other social networks.
• 1:14:06
  Oh, excellent. Yeah. That was a good one. Right? Because, like, sometimes you really get I I I it's kinda like whenever a cat keeps showing up with a dead bird on the doorstep, you know, to show you the dead bird.
• 1:14:16
  Hey. So there that's dead birding. And, so a lot of lot of content from the dead birds coming on to the the website, and, you know, sometimes you're like, you know, I really don't see that. Or at least I want, like, a little warning so I can kinda check-in. Like, am I in the mood for that?
• 1:14:33
  So somebody was able to apply just a nice little AI and modeler to be able to detect those things and then label them. That is pretty great for improving the experience. Another one that somebody put out there that I I don't think I've I haven't checked in on it to see how it's doing, but, like an AI slop detector. Right? Like, are you tired of AI generated images?
• 1:14:56
  Here's a little labeler. Then at least, you know, you can filter it or you could just give you a little warning, notice so that you could just decide. Or, you know, if anything, in some ways that kind of works well. It's just, in, like, a protection against misinformation if it's, you know, I I assume if the AI model can figure out that it's AI slop, it's probably pretty obvious. But still, you know, it's in the zone.
• 1:15:18
  Another one, this is not automated. But actually a decent one is a, a politics labeler. And this one's spicy because, like yeah. Because, like, what gets characterized as politics is,
• 1:15:31
  Bryan Cantrill: you know, it's a little bit a little
• 1:15:33
  Paul Frazee: bit spicy, but, definitely not the kind of thing that we as a company would necessarily wanna be doing, for because it is spicy. But for somebody else out there, they'd be doing their best. You know? Like, yeah, we think this is kind of political. And, like, it's not actually you can have it just, like, completely hide this stuff, but, honestly, just had to get behind a little interstitial.
• 1:15:50
  And so you could just decide, do I really wanna read something about the poll like, the election going on right now if I'm not
• 1:15:55
  Bryan Cantrill: Right. People need
• 1:15:56
  Paul Frazee: a break.
• 1:15:57
  Bryan Cantrill: You know what I mean?
• 1:15:58
  Paul Frazee: So, like, being able to just have this kind of thing to you know what? I need a break. I'm turning this on. Let's cool down the politics for a little bit. I that's the kind of stuff that I I get pretty excited about because these things do psychic damage.
• 1:16:12
  So any tools to kinda, like, reduce it.
• 1:16:15
  Bryan Cantrill: Totally. And then as how do these labelers work? Are the labels where are the labels actually stored?
• 1:16:21
  Paul Frazee: Yeah. That works on a actually, a completely separate synchronization. So it like, we have 2, I guess, main kind of data structures in the app protocol. The first one is what we call the repositories. Those are the what represent users, and that's where all your data goes.
• 1:16:37
  So repository like a Git repository, but in this case for, like, JSON. And then we have the labelers, and the labels. And so those exist kind of in their own, system that we didn't put them in repositories because it's kind of a giant long list of just strings. So it does not work well with the cryptographic structure that under resides underneath. Like, the, the the repositories are are a variation of a Merkle tree.
• 1:17:03
  And, we with the labels, we're like, you know, it's just not great. So we just decided to have them in their own system. So they kinda synchronize between, the, labeler and the any kind of services that are interested in subscribing to them in a much slightly dumber way, to be honest. Yeah. That's it.
• 1:17:20
  That's how it works. So, our app view, our our application, whatever a user, like, subscribes to a labeler, our application, if it isn't already subscribed to the labels list, we'll find it and start syncing it. And then we do a little runtime join based on your currently actually, it's a header in the request. We we look at a header of the which labelers you wanna apply labels for, and then we just pull them in and attach them through the response. And then the client observed them based on the user settings, and that's it.
• 1:17:49
  I mean, a label really is just a label. It's just a string saying, hey. This is like NSFW or this is like a spider. Are you arachnophobic? Great.
• 1:17:56
  Don't show the spider. It's that's about it.
• 1:18:00
  Bryan Cantrill: And then so I subscribe though to these labeling services. So I get to choose, like, who if I there's a labeler that I really like, I I can subscribe to that. Is that right?
• 1:18:08
  Paul Frazee: Yeah. Yeah. The way they right now, the way that it feels in the application, I don't I wouldn't say that we have, like, nailed this UX. But, the way they work is they're kind of like special accounts. Like, you can kinda upgrade an account into becoming a labeler, at which point the labels that it offers get turned into the primary profile screen.
• 1:18:33
  And rather than the primary action being, follow, you know, the primary action is subscribe, and you subscribe to it. It gets added to your moderation screen, and you do then configure each of the labels on how they're supposed to operate.
• 1:18:44
  Bryan Cantrill: Well, that UX definitely doesn't clutter anything else in that it is, it is definite That's great. I actually did not know this at all and going to like the, there's someone in the chat had linked to the U. S. Government contributions labeler. Oh, yeah.
• 1:18:58
  That's a good one. And this seems like, I mean, it I just this seems another like a really good experiment, and a a because then, like, of course, I wanna know, okay, what are the labelers that Adam subscribes to? Because I I may want those same or maybe those are labelers just to avoid because, you know, that guy. I don't know. I mean, he just kicked me out of the block by p market club because I I'm no longer blocked by p market, so I don't know.
• 1:19:25
  I've got kind of some axes to grind over there. But I mean, I don't know. It just feels like you're you're giving people kind of another tool to go experiment with this problem that you did not create called bad humans. And but also, like, good humans that actually wanna come in and and help offer some clarity on some of the stuff. I think I don't know.
• 1:19:45
  I think it's really to me, it's really inspiring. I know that's it it's kind of strange to be inspired by the encroaching 8 channel of the Internet, but I I I really do think it's it's inspiring.
• 1:19:56
  Paul Frazee: Well, I hope that we can get there, like, as far as we want to with it. You know, I think there's a lot of good things that are happening with it, and there are aspects of it that I'm really satisfied with. I will say that the labelers, so far had been a bit of a mixed bag in terms of their practical reality because it's pretty hard to run one, for the non trivial use cases. If you're trying to actually do what you might classically consider to be moderation, I think we've that that part's been a little bit tough, and that that is one of the things that we've been sort of sitting back to evaluate before we kinda push on recommending them more is to see what this is how this is affecting communities. And I I don't think I would give that 1 10 out of 10 marks.
• 1:20:41
  We're here in in some cases, it's gone pretty well, but in some cases, we're hearing that it's a pretty stressful position. There's a lot of responsibility that comes along with it. And, and I don't know if they're empowered in all the ways they need to be to handle that responsibility or the expectations that come along with it. So, yeah, I don't wanna I don't wanna, walk in here and and ignore that. You know what I'm saying?
• 1:21:03
  Like, I I do wanna call up to that. It's it's not quite 100%, I think, at the place that I'm I would love for it to be. But I am glad that we're giving it a shot. You know what I mean? And I'm glad that, like, we're learning off of it.
• 1:21:15
  And I think you'll probably see more updates to it based on some of the challenges that we've had over the coming year.
• 1:21:21
  Bryan Cantrill: Well, that's exactly it. I I feel like it's it's the giving it a shot bit, and I feel that, like, learning from so many previous experiences, and then also doing it all the open. Because this is all I mean, you you're all open source. Right? In addition to having, like, these open specifications, you've got Yeah.
• 1:21:38
  All or much of your software is actually out there in the open.
• 1:21:43
  Paul Frazee: Yeah. I would actually like to for it to be more of it. There's still a couple of things that we've still got closed, like, our main algorithm is still closed. We'd love to get that open at some point. We have some of our back end that's, you know, you can get the back end running, but it's not to the 100% ready to go place.
• 1:21:59
  And so I'd love to get all that, totally open and in a good package, so that people can run it. But, a lot of it is open source. It was really fun to be able to share, you know, when we got to the we we got we managed to hit, you know, number 1 in the app store. Really proud moment. And
• 1:22:19
  Bryan Cantrill: That's awesome. Was able
• 1:22:20
  Paul Frazee: to share the source code and say, here it is. Here's the source code. If you wanna do it, you're you have one yourself. So that I'm really proud of that. You know what I mean?
• 1:22:28
  Like, that we were able to take an open source application to this place, and I really like that we have stayed with that throughout.
• 1:22:36
  Bryan Cantrill: That is awesome. And congratulations on that. I mean, that is that is really terrific. I close to without precedent, I assume. I mean, it's certainly certainly for a social a social app, it feels like it's without precedent.
• 1:22:47
  Paul Frazee: But, You know, I'm gonna I'm gonna Steve Jobs it and say, yes. It's never happened before.
• 1:22:51
  Bryan Cantrill: This is the very first time. That is well, that well, that is just gonna ask me is, you know, kind of like I mean, you've obviously got, a lot of challenges still ahead in part because of the popularity. What happened some of the what have been some of the surprises along the way of things that maybe have worked better than you thought or works much worse than you thought. I don't know. Maybe you were putting scenes into that category, at least in terms of nomenclature.
• 1:23:18
  But with the what are what are some of the things that that that, you were or are there things that that you've been kinda surprised by along the way?
• 1:23:28
  Paul Frazee: Yeah. I may only restrict this to the things that I'm proud of, to be honest. But,
• 1:23:33
  Bryan Cantrill: Sure.
• 1:23:33
  Paul Frazee: The one I'll say the one that comes to mind that actually works so much better than I ever expected was the domain names for usernames.
• 1:23:41
  Bryan Cantrill: I'm glad to bring
• 1:23:42
  Paul Frazee: that up. Yeah. Yeah. Yeah. We we put that in there, honestly, for a very pragmatic reason.
• 1:23:47
  We just needed a namespace that was decentralized. Right? Yeah. And which, you know, which would work for the open network and, in which, like, had the affordances that we needed. And so I was like, well, okay.
• 1:23:59
  Domains. Right? Like, what what else are you gonna use? I guess, like,
• 1:24:03
  Bryan Cantrill: we could have gone with the kind
• 1:24:04
  Paul Frazee: of, the activity pub style, like, the email sort of thing. But we just didn't. There was I wish we were so brilliant that we said, oh, we also want the website correlation, but actually it was more that we just didn't like having 2 at symbols in the syntax. So it's as petty as that.
• 1:24:18
  Bryan Cantrill: On you. Absolutely. Yeah. Yeah. Cool.
• 1:24:21
  It's come on. It's all about aesthetic. Please. Yeah. I
• 1:24:24
  Paul Frazee: mean, there is something kinda cool about that. I can't yeah. Believe it or not, I I do really like you could just, like, sort of, like, toe turn turn the host into a pill and then just have the username, and you could do maybe some nice things with the UX. But, anyway, I'm glad we've made this choice. Where I was really shocked when it came to the domain names was the uptake of the custom domain names and how that has played out has been way beyond expectations.
• 1:24:49
  Because when we did that, we would put together, like, little mock ups with, like, the Washington Post using that as their handle. And it was kinda like the again, it was one of those things that, like, yeah. Right? You know, someday, maybe, you know, who knows? And then they did.
• 1:25:04
  Like, when ESPN came on, they use
• 1:25:07
  Bryan Cantrill: With ESPN. Yeah. Com. Yeah. That is so cool.
• 1:25:10
  Like, I can't believe that these organizations are going to the trouble of configuring DNS for us now. Well, it was even better than that. It was when ESPN came on, they were espn.bscout.social for, like, a hot second. And they're and literally their first post is like, no. No.
• 1:25:24
  No. Like, we know we know we know. Like, we're changing. I mean, everyone is dog piling them. Be like, come on.
• 1:25:28
  You gotta, like, change the domain name or you're espn.com. And Yeah.
• 1:25:33
  Paul Frazee: Yeah. Which I thought were these like I was I mean, I
• 1:25:36
  Steve Klabnik: thought it was cool to use the the main names, but at one point, it really hit me. I don't remember which account it was, but, like, you know, during the massive uptick in growth, you know, people join. You always wonder if it's really them or not. And I remember seeing a post that was, like, by the domain name of some it was like a news organization or something probably, but just being like, oh, I, like, know that's their account. And, like, that's actually really, really cool.
• 1:25:59
  And obviously, like, I don't know how lay people, you know, know about that kind of thing or whatever, but it definitely, like, really there was a specific moment where it really sunk in for me as, like, this is more than just, you know, piggybacking off of another Internet protocol to, you know, name things.
• 1:26:16
  Bryan Cantrill: That yeah. Very elegant solution to the problem. And I I mean and and people are obviously calling out, like, the fact that you get because you're getting this verification, basically, and you're verifying it the way we verify the Internet, which is, like, why wouldn't you do it that way?
• 1:26:30
  Paul Frazee: Yeah. Yeah. I mean, I'd again, that wasn't, like, the driving decider on that. Right? It was really, oh, we need something, you know.
• 1:26:38
  But then I think we you know, it does it has helped us a lot. I think we're probably gonna need something more, both because it is a little bit wonky. There you know, I don't wanna move too quickly on that on that aspect of it, but there's definitely some folks that may not totally understand it. But then you also just have folks that don't have, like, high reputation domain names. A good example of that would be, Flavor Flav came on, a bit ago.
• 1:27:04
  And got so it was so enthusiastic. Love it. And he'd like, he doesn't have a website. You know? He just doesn't.
• 1:27:10
  And you also start to run into the okay. He could set 1 up. He could set up flavorflave.com, but then how do you know that that's the right one versus flavorflave.netor.ioorwhatever? So there's gonna probably need to be some layer in there at some point. But no matter what it is that we do, the well known domain names are really fantastic anchor for it.
• 1:27:33
  And boy is it great for the government stuff because they've all started to use, like, dot senate.govand.house.gov. Like, Tim Walz just came on and became, I believe, governor.mn.gov. Just, you know, totally totally solving that problem for some of the more high stakes ones, which is, you know, political leaders. And those are under that tight governance. Right?
• 1:27:56
  Like, the dot gov domain name is the US government highly regimented, and it's so much better than anything that we as an organization could be doing. So definitely some wins in there.
• 1:28:08
  Bryan Cantrill: Big win, I think. Do can I ask you just a very pedestrian question? Because this is the only thing that has kept me. So I'm currently, be canceled, be scanned at social, but I I definitely want to I I actually, I don't know if I wanna be canceled. Detrace.orgorbecancel.oxide.computer.
• 1:28:23
  So this is my own thing I need to go grapple with. Matthew Piep (3five forty six):
• 1:28:26
  Adam Leventhal: I'm sure Paul can tell you this, but you could move it. You could change it later. Right?
• 1:28:30
  Bryan Cantrill: This is
• 1:28:30
  Adam Leventhal: one of the frustrations with the Fediverse and one of the liberating things about Blue Sky.
• 1:28:36
  Bryan Cantrill: You can. The only thing that's hung me up is then I need to go immediately squat on my old name. Right?
• 1:28:43
  Paul Frazee: Oh, I got good news. We fixed that. We just
• 1:28:46
  Bryan Cantrill: fixed that.
• 1:28:47
  Paul Frazee: So Oh, boy. Yeah. Yeah. Yeah. Yeah.
• 1:28:49
  We we haven't done comms on it yet, but we, are, planning to to wait. We're we're scrunch together those comms. But, yeah, you now if you I got these for a piece of just comms
• 1:29:01
  Bryan Cantrill: on it, my friend.
• 1:29:04
  Paul Frazee: Yeah. And it it's out. So, like, you you actually, ironically, somebody ran into it in the reverse because they they were doing the squatting move that people have been doing as a patch. We're like, I can't register it. It's probably no.
• 1:29:17
  No. No. No. It's working right. So, yeah, what it does now is we just keep in our database of the bsky.social stuff.
• 1:29:23
  We just remember it when somebody moves off what your original one was. Yeah. And then if you you can go back to it. So if you were the original holder of it, you can go back. Nobody else can, though.
• 1:29:34
  And so and I guess if you delete your account, then it gets relinquished. But yeah. Yeah. So that is now fixed. Go wherever you like.
• 1:29:42
  Bryan Cantrill: Oh my gosh. That is great. That is that is terrific. And I'm sorry to disrupt your comms plan on that. Thank you very much for you you breaking news here on oxide and friends.
• 1:29:51
  That that that that that
• 1:29:52
  Paul Frazee: that is just being handed
• 1:29:53
  Bryan Cantrill: to me. That is, that's but that's that's terrific. And I mean and I you know, one of the things that you all are doing so, so, so well, and I I just cannot thank you for enough, is the degree to which you're listening to people who use the product and you're using the product yourselves. It's like Yeah. Feels basic, but it's amazing how often this doesn't happen.
• 1:30:17
  And, I just I mean, that's a great example of, like, a I mean, I know you've got so many terrific ideas for things that can be done, but that's just like that's a really kind of like a basic simple kind of pain point that is not debilitating, but, boy, thank you so much for for fixing it. I think it's just it's part of what makes it as a user a Blue Sky. It makes me so enthusiastic for the future of this platform. One thing I do I wanna be mindful of time because I know, Adam is gonna have to bolt here. He's, no longer toddler, is now, is that kid in high school yet?
• 1:30:53
  1st grade? What is he in?
• 1:30:55
  Paul Frazee: 2nd grade.
• 1:30:56
  Bryan Cantrill: 2nd grade. God. Yeah. Built like a high schooler. Will, but the I do there's one last feature I wanted to ask you about and that is starter packs.
• 1:31:04
  Paul Frazee: Oh, yeah.
• 1:31:05
  Bryan Cantrill: Because another thing that was I I I know is basic, it was really important, I feel. Could you just describe a little bit, like, kind of the the the thinking behind starter packs?
• 1:31:18
  Paul Frazee: Okay. This one's pretty fun for a couple of reasons. So first of all, let me start with the context of how that got built. We, the full context, you have to start with the fact that we are not an algorithm company. And so our initial algorithm that people are getting landed in, we're working on it all the time, and we're making some great progress on it.
• 1:31:41
  And I don't wanna harsh the the team that's working on it. They're really cruising. But it is not great yet. And so new users are coming in. It's really important that they are able to get their social graph built so that the following feed can start working for them.
• 1:31:58
  Yeah. That's point number 1. Point number 2 is building, your social graph on the app. Also a bit challenging. Getting good recommendations for follows, that is quite hard, especially when you're seeding communities.
• 1:32:13
  Like, here's a design challenge for you. Make a good recommender when you don't have social clusters on there yet. How do you even evaluate if this thing is doing its job correctly? You kinda can't do it. Like, we're learning a lot about algorithm design in without content to work against.
• 1:32:27
  Thankfully, that is less a problem now, but that was the kind of we have a real chicken and egg problem. So okay. You're obvious thinking is yeah. So so okay. So you're you're obvious thinking you're you've got this design problem.
• 1:32:39
  Well, let's do, like, a contact import of some kind either from an existing social network or, like, from the address book. Well, you can't do the address book because that has privacy problems that may not be fixable. As far as I know, last I heard, you can't come up with an anonymized upload your address book thing without creating triangulation of people's contacts. Talking
• 1:33:01
  Bryan Cantrill: to dirty tactic anyway. I think that is so damn dirty.
• 1:33:05
  Paul Frazee: I I mean, I I agree. Yeah. Well, and, like, it it it gives an eck vibe that, like, anytime I get into an app now. I and I think everybody has it. You know?
• 1:33:12
  And he's like, oh, no. I'm not gonna give you my con actually, you have your mind. Out
• 1:33:15
  Bryan Cantrill: of there. No. Don't even ask. It's like, like, I I don't wanna I don't wanna peek. Like, no.
• 1:33:20
  No peeking. Gross. Get out of here.
• 1:33:21
  Paul Frazee: Yeah. It's it's a real shame. Now that I'm on the other side of that, I get why people go ahead and do it. But we still weren't gonna cross that line.
• 1:33:30
  Bryan Cantrill: Right. Yeah. Yeah.
• 1:33:31
  Paul Frazee: It's yeah. It it was it was a shame because we're sitting there going, like, wait. We're not trying
• 1:33:34
  Bryan Cantrill: to be jerks. We're actually trying
• 1:33:35
  Paul Frazee: to help you. But, like, you know, here you are. Right? So we didn't cook up with some kind of breakthrough on the privacy solution there, so we didn't bother. So okay.
• 1:33:42
  That's out. Well, then you can get into can you try to get the social graph from another social network? And the reality is none of them wanna give that up. And we're really in an era where, like, these APIs are getting closed down. You can't do it off of, the meta products and you sure as hell can't do it off of Twitter.
• 1:34:01
  I guarantee I know how that would go. So we that was a nonstarter too. Right? So we're sitting around going, how the heck do we solve this problem of getting people's social graph seeded? The original idea for starter packs was actually not just to have, like, a list of people to follow.
• 1:34:16
  It was actually to create an invite link because we have had a lot of success seeding the social network back when we were invite only. And Yeah. Ironically, when we switched to the public thing, we lost one of the best organic mechanisms for people to get worked into their social grasp whenever they joined. So we were looking at that and asking how can we make that better. And the initial impetus for this was actually just to create an invite link that you could share with friends.
• 1:34:42
  They would take them through the install flow and then get you into a bidirectional, like, follow relationship with whoever it was that was invite that invited you. And then that would help seed the network. And then we were like, okay. Well, let's go ahead and add in, suggested follows and some suggested feeds and things like that so that you could actually, like, hydrate in the experience. Well, we launched this thing and, we also included the ability to just, like, follow all of them from within the application.
• 1:35:07
  So it would work also if you were an existing user, but it was just kinda like a, yeah, just in case kinda thing. We launched starter packs and people were like, oh, yeah. That's really cool. But, like, they just didn't get used. So we ended up spending, like, 2 months going, like, there was so much promise with starter packs, but that really didn't land with folks.
• 1:35:26
  And then the big influx came in November, and man, did they pop off. They ended up becoming the absolute most potent mechanism for getting people onboarded into the application. And, again, rarely through the install flow. It's very rare that people were using, like, clicking into them and then, like, kinda having the app and we we would use an app clip to, like, do the install flow and then get you creating your account and stuff like that. That was very rare.
• 1:35:51
  It was almost always people just sharing them in, in feeds or, like, installing the app person, like, you know, messaging each other with these things or however it was that they were doing it. And that ended up helping the social graphs and all the clusters form really rapidly and get people's following feeds into an interesting place. And being really strong proxies to topic interest, which is an area that I think we're still pretty weak on is giving new users a fast access to a topic. So you would find, like, a journalism starter pack or an art starter pack, and that would boom. Now your following feed has given you some great stuff on that front.
• 1:36:27
  So, yeah, it was, not a linear, like, a a to b thing with starter packs. The core idea, I think, maybe there was but in terms of, like, how it got used and and what we thought we were going to be pulling off of it, it it had some twists and turns.
• 1:36:42
  Bryan Cantrill: And well, and for whatever it's worth, just my own personal experience with starter packs, I was trying to get so the I'm I'm an Oakland A's fan, the hashtag sell the team, the and we kind of got this now, beleaguered lost tribe that is still very much on Twitter, was very much on Twitter. And talking to the folks who are kind of like the leaders of that community being like, we gotta get one of the starter pack because we like, the entire community wanted to leave, but we you kinda needed everyone to go at once in order for and that is what starter packs starter packs were a lifeboat for people who wanted to And I saw this in various scientific communities. I saw it in, certainly, in Oakland A's Twitter, where it's like, okay, this is now the opportunity. And I think actually, another thing that I that I love about just, like, the the dynamic was people would start a starter pack with, you know, 60 people. And those people would kinda come on and the person who started the starter pack would always say, hey, like, let me know if I forgot anybody.
• 1:37:41
  And you've got, like, a bunch of people raising their hand being, like, oh, god. I'd love to be included in that starter pack. And Yeah. They ended up being very inclusive and uplifting and a great way to move communities across. So I, I it it doesn't surprise me because in that multi month gap where you're like, why isn't this working?
• 1:37:59
  I, and I'm sure other people in various other communities, are kinda lobbying people who are still on Twitter to be like, no. No. Like, let's get one of these together so we can get everyone over at once. Yeah. It it's been really great.
• 1:38:10
  So thank you very much for that. It's a it's been a it's been a great feature.
• 1:38:14
  Paul Frazee: Oh, really happy to hear it. Yeah.
• 1:38:17
  Bryan Cantrill: And and just again, thank you, and, thank you, Paul, for for taking all of the time with us tonight. And, I mean, we are are huge fans, love what you're doing, what you and the team have done, love that I mean, we feel like very much kindred spirits and that, like, this is a problem that you all have as a team have been thinking about for many, many, many years and watching all of that wisdom get expressed in in what you've done. And, you've created something. I know it's challenging and the, you know, more people you get in there, that there's gonna be challenges that are gonna rise with that. But, boy, what you're doing is so important and special, and we're really deeply, deeply, deeply appreciative.
• 1:39:03
  You've you've given, I think, a lot of people, certainly me anyway, real hope in what we can have out of social networking. So really deeply appreciated.
• 1:39:13
  Paul Frazee: Well, thanks. I mean, I gotta be honest, the that level of, like, supportiveness that you're you're showing here really does help, like, at any given moment. Like, it's been really nice to to see people get that level of interest in it too. So the feeling is very much mutual. The when it when it gets tough, the the folks that, like, are there and, like, yes, let's keep moving, that that kind of buoys us up for those tough points too.
• 1:39:38
  Bryan Cantrill: Absolutely. Well, and I'm sure, again, more humans are tough, so more more challenges to come. But, really appreciate it, and thank you again for for taking the time to to come here. And, we'll we'll be, glad that we don't no one has to hit up Klavnik for invites anymore. Everyone can hop on there.
• 1:39:57
  And, Adam dropped an oxide computer starter pack if you wanna follow folks who are we got a lot of folks from oxide, obviously, who are over there. Paul, thanks again. Thank you very much. Welcome back anytime, obviously, and, keep up the great work. We'll be, we'll be there and in the thick of it.
• 1:40:16
  So thanks again.
• 1:40:17
  Paul Frazee: Sounds great. Thanks for having me.
• 1:40:20
  Bryan Cantrill: Alright. Thanks, everybody. And next time, we're gonna be doing our our year end wrap up. So, bring your your your your your best and worst from, from 2024. So Adam, looking forward to that.
• 1:40:32
  It's going to be fun.
• 1:40:33
  Paul Frazee: Should be great.
• 1:40:35
  Bryan Cantrill: All right. Thanks everyone. See you next time.