Latest / Oxide and Friends / Get You a State Machine for Great Good

Transcript

Get You a State Machine for Great Good · Oxide and Friends

• 0:00
  Speaker 1: I will let's see maybe describe how we got here a little bit. So, we I think you made Adam, I know we have mentioned demo Friday the past here, that we and and our credo around demo based development here at Oxide.
• 0:13
  Speaker 2: Yeah. I mean, it's I feel like bordering on extensively. But, yeah, like, demo day is prominent feature.
• 0:20
  Speaker 1: And it is deliberately unstructured, so it's kind of a it's it's a form for people to demo stuff for their peers. Our kind of you is that, your peers are I mean, you get the biggest shot in your arm, I think. You get big shots in the arm, obviously, from customers and from from folks that are looking at your work, but especially your peers. It's a big shot in the arm when you're when you can show your work off to your peers because they can appreciate some things that are can be hard to appreciate. So we've had a lot of fun, this every week, and, we it's it's always been very uplifting and interesting.
• 0:51
  You kinda never know what's gonna happen. I'm always worried we're gonna have demo Friday and nobody shows up, but I think it's happened once. I think that actually has Like, alright. No demos today. End of demo Friday.
• 0:58
  But, basically, it's, there's always something going on. And so, Andrew, I did not know you had this one coming at all. And, could you describe first the body of software that you that you have been working on, kind of what the this body of software does, and then maybe we'll get to describe a little bit what the actual demonstration was.
• 1:18
  Speaker 3: Sure. So, I mean, I'm with you. I actually didn't know this demo was coming about a week and a half ago, and then I started working on it. But it was planned, as possible from the beginning. And so, like, what what we have, is this giant rack, right, that users are gonna plug into their data center and they need to connect it to their network, and they need to connect it to an identity provider, and they need to do all sorts of initial configuration so the the control plane will actually boot and be reachable, from their internal network.
• 1:50
  And so they can deploy VMs and do whatever else they need to do on the rack. There needs to be some if the control plane is not up, there needs to be some way to actually log in and configure the rack, in the first place so we can set up the control plane. And so Adam and I and another colleague of ours, John, we sat down, last February or so and started working out, well, what's what's that configuration actually gonna look like? And we talked about using, you know, various ways to do it with an API, through the web, just have a CLI. We basically landed on some sort of CLI, that would where a user would plug their laptop into the technician port, and that would allow them to do the low level configuration.
• 2:30
  So go ahead,
• 2:31
  Speaker 1: How do we configure the well, how do we configure the network without the network? I mean, it feels like we we've got a bit of an inception problem. So how do we just physically, how do we connect into this thing? Yeah.
• 2:39
  Speaker 3: So I am probably not the best person to explain how the switch works, but there are some, there's there's basically 3 sets of ports. There's the front facing ports, the rear facing ports, and these and 2 technician ports. And so the front facing ports are the ones that connect, to the user's data center. Right? And, like, that's what gives them their fast 100 gig networking.
• 3:03
  And then the rear facing ports are the ones that I believe are just for inside the rack to allow slides to talk to each other over the rack. And then there's these 2 technician ports, and these have, you know, regular old RJ 45 cables, you know, endpoints you can connect, an Ethernet cable into. So that's, those basically open up into some sort of environment, that allows you to configure do something to the rack. Right? And, they plug in those those, technician ports, are reachable on the management network, I believe.
• 3:36
  Or is it the bootstrap network?
• 3:38
  Speaker 2: I'm It's the management network. Yeah. And and we need to do a a whole show on the management network, because it's I think it's pretty outstanding. But we've talked in the past about the the data network, like how all the nodes talk and how it talks you know, how we do, you know, multipathing in in a in a smart way and so forth. But there's another network, and, actually, there's yet another network, but the management network to control all these nodes out of band.
• 4:05
  And and that's the that's the network that this technician port is plugging into. So, the rack rolls into your data center. You plug in to with an rj 45 connector into, you know, to your laptop, do a, you know, LLDP to discover the link local address and SSH in, and then we're in Andrew's body of software.
• 4:27
  Speaker 3: Yeah.
• 4:28
  Speaker 1: And this is the this is the body of software that's gonna basically configure this thing, and it's gonna be effectively, it's a terminal it's gonna be a terminal UI. And, you know, these are kind of these are the apps that often these are essential, but can often get neglected. How dare you say it, Andrew?
• 4:44
  Speaker 3: Yeah. So I certainly neglected it. I didn't think I would be the run to write it. I have many other pieces of code, required to run on the rack. And I thought at one point, like, we're talking about making a web a web UI or something, but I figured there'd be somebody with some sort of user experience knowledge who would be building this.
• 5:05
  But apparently, that is not the case. Although to be fair, I have worked with an NGR designer to correct some early mistakes.
• 5:12
  Speaker 2: It it it should be said that, you know, we we chose a CLI because we really wanted it to work for everyone in every environment, and we didn't you know, this is you're setting up just the earliest earliest small amount of configurations so that you can plug into the rest of the network, and we just wanted to keep it simple. From from a sort of connectivity perspective and the technologies involved, we didn't want people, like, debugging JavaScript or whatever. We just want to keep it simple. And I say
• 5:41
  Speaker 1: not browser deniers is what you're saying.
• 5:42
  Speaker 2: No. Totally. We're we're we're browser maximalists for sure, but we just wanted something simple to get to the browser.
• 5:47
  Speaker 1: That's right.
• 5:48
  Speaker 2: I would also say, you know, Andrew, you talk about this being neglected software. The the, you know, what I sort of imagined was the most most basic user interface conceivable of, like, you know, something prompting you 1, 2, 3, or 4. But Andrew has discovered an entire universe of Chewy's, of, like, text based user interfaces and has built something unbelievably beautiful.
• 6:15
  Speaker 3: Yeah. So it was very much by accident because, Adam, I I figured it would be just like what you said as well, especially if I was gonna be the one building it. Like anything else, you you find, you know, an interesting niche and you start digging in and having fun. And, boy, was it a lot of fun. When you start having to make things real and get to the production engineering side, you know, things get can get less fun.
• 6:38
  But it's certainly a fun thing, to to dig into. So so we do have, there's a text, a text user interface, and we talk to it through this technician port. And so there's a there's a whole subset of software that runs here. Right? And so where is the software running?
• 6:53
  It's running on what we call the streamlet, which is a server sled that is directly connected to the switch. And there is some magic routing, I am not gonna discuss because I can probably not explain it well, but that essentially allows, the technician port plug in all that data to come over, out a port on the technician switch. And so when a user does SSH in, we can it ends up in a Lumos land and the SSH command runs, our shell, which is called Wicket. So it's a captive shell. It's a text user interface.
• 7:26
  It's built on top of, 2ERS and cross term and Rust. And it presents the user with essentially, like, a graphical interface inside the terminal. So they're SSH ing in and this is their their login entry point to allow them to configure the system. And so should be
• 7:44
  Speaker 1: said that just in general, these are the kind of interfaces that Rust makes really easy and delightful to write. Honestly, there's a good, great support. It's been really fun to write these kind of applications.
• 7:57
  Speaker 3: Yeah. Totally. And so, like, credit where credit's due, like, I would not have even remotely thought of doing anything like this, but we have a manufacturing station that we also built, like Josh Clulo, and he, implemented a text user interface for it. And I was literally about to start working on this, and I was like, He he it was a demo Friday thing. He demoed it, and I was like, well, why am I gonna do, like, a CLI where I can have this beautiful captive shell?
• 8:23
  Because Josh does have some aesthetic sense, and, it was nice. And so I initially copied him, which it got changed later for various reasons.
• 8:30
  Speaker 1: Okay. I cannot risk this pointing out an implementation detail of Josh's manufacturing software, which is all, which honestly, on the one hand, no customer sees the other hand. Runs on the manufacturing line. It's very important. The font that he uses is the same font that Sun had on the open boot prom.
• 8:49
  And, I mean, Adam, I felt like I I teared up when I saw that. I was just like,
• 8:53
  Speaker 3: we're home. We're home. Throwback. Yeah. Totally.
• 8:56
  Awesome throwback. Anyway, sorry. Andrew No. No. No.
• 8:59
  It's like it's super it's super impressive. And, like, at that manufacturing station, like, Josh, I actually talked to him and I I was like, walk me through your your mental state and, like, what is your point of view on this type of software? Because, like, if you know Josh, yes, and that, like, he definitely has a point of view and, like, specifics specific reasons for why he did something. And so I wanted to understand it and see, like, if I wanted to go down exactly the same path or something different. And so I did do something slightly different, but it was more because I knew this was going into, going to live on the rack, and it wasn't gonna be just me operating on it.
• 9:30
  And I didn't so I didn't wanna, like, write a terminal library from scratch. I want to reuse some available widgets, and there are third party widgets. We're using some of them. So anyways, we have this next user interface, and it is mostly stateless. And what I mean by mostly stateless is that it talks to a downstream service called WICD, and that is what talks to other services on the rack, including our management gateway service, which allows us to talk over the service processors on each switch and configure the rest of the rack.
• 10:01
  And I don't know if Brian wants to get more into the service processors.
• 10:04
  Speaker 1: Well, so so at the moment, like, so now I wanna jump actually kind of forward in time to your actual demo. So this is kind of all background to under to appreciate. And you you had built Wicket and had demonstrated in the past and would make great progress on that in terms of walking the this kind of the the initial application effectively to configure the rack. But and so now maybe fast forward to the demo. Also, the way you did it, you little you little sneaky guy.
• 10:30
  It was very you I think you got an inner showman in you, Andrew. I was very, in terms of the way you demoed it. So maybe you wanna describe the demo a little bit.
• 10:39
  Speaker 3: Yeah. So unlike PT Barnum, we at Oxide are transparent in what we are selling, but That is not always true for internal demos. And, like and so, I I
• 10:49
  Speaker 1: feel that transparency does not apply in the first two minutes of a demo of a demo Friday.
• 10:53
  Speaker 3: Yeah. And and and apparently yeah. Yeah. And, you know, secrecy secrecy also for keeping things fun. Those are only for fun reasons, not for any various reasons, I must say.
• 11:05
  Okay. So I started the demo, and, like, this is, like, the 3rd time I've demoed Wicket for various things. One was the initial like, after the initial prototype. 2 was after a redesign and when, like, we could do some emergency updates where we could actually blast some could actually update service processors and ROTs, through some some software. So, like, that was, the second demo.
• 11:28
  And so the third one, I was like, okay. I have some new functionality. I wanna show you. And so I essentially, said, like, watch closely, watch what I'm dealing, watch the screen. And everybody's watching and probably going, okay.
• 11:37
  Like, what is this? I see what's going on here. It it makes a lot it it makes sense, but, like, I've seen this before. You're just walking through the UI, and then I just lifted my hands up and the UI kept moving. And so, it turns out what was going on was a recording, but it wasn't a screen recording.
• 11:53
  It was a, recording of a live Wicket session that I was able to capture with a magic key press that I'm sure Brian's cat could learn. And that recording was being replayed. And then I went to see how it works. Yeah.
• 12:10
  Speaker 1: Okay. Well, that's not true.
• 12:13
  Speaker 3: K. Well, my memory is terrible, so whatever.
• 12:15
  Speaker 1: But you you didn't go through no. Then it's, like, not and then, of course, everyone's, like, wait a minute. What? And then you'd be proceeded to show this debugger that you'd written. Do you not recall the It's right.
• 12:26
  Speaker 3: I mean, so so the debugger is what does the replay. So okay. So I did, I did But then you also
• 12:31
  Speaker 1: you're going backwards.
• 12:34
  Speaker 3: Yeah.
• 12:34
  Speaker 1: Go
• 12:35
  Speaker 3: ahead. It did not go I the still bugger is not reversible.
• 12:38
  Speaker 1: For the go backwards. Those screw.
• 12:39
  Speaker 3: It did not go backwards. It can be made to go backwards. I have built things that have gone backwards in the past. We are not using immutable, like, functional data structures. And so it is very cheap to capture the state or I mean, it is very expensive in the current system to capture all the state because we're making a copy of it all in memory, for every step through a UI screen.
• 12:59
  Really not the way you wanna build it. And I'll I'll talk about the internals. But, yeah, what I showed was that you can reset this. You can then, kick off the you can kick off the player recording. You load the recording, through, like, a file, and it's a there is, like, a little CLI.
• 13:15
  That's probably what Adam and I were envisioning for, like, a normal captive CLI built on, something called Readline, which is a tool for New Shell. And that that's just, like, allows you to give you interactive, command line driven, abilities. And so what I showed was that you can load a recording from a file, hit play on it, and then as the recording is playing, you can also pause it and you can resume, and you can show, like, what the current event that's being executed is and then, the internal state of the system.
• 13:50
  Speaker 1: At that point It's about now when Jordan is, like, did I miss what what debugger are we talking? Did he did I, like, miss something? I mean, it's like, no, no, no one knows anything about this. But this so everyone is seeing this for the very first time. You can't like it over.
• 14:05
  So it was it was pretty neat. So, Andrew, I think then you, at that point, you you relented and explained kinda how you had implemented this that allowed for the this kind of replay debugging.
• 14:19
  Speaker 3: Yeah. So this is a technique that is fairly widely known, but unfortunately, like, not widely known enough, I would say. And it's, it's really I mean, there there's many different terms for it and and different techniques, but it's essentially, if you can maintain if you can separate your IO your your your IO from your state from your mutable state, you can record you can basically treat all mutations of the state as individual operations, and then you can record those operations. And you can replay your entire history by just, passing the operations through the input as the input to a state machine. So you have this mutable state and the state takes a message essentially, and it updates its own internal state and possibly output something some some output messages.
• 15:13
  Right? And so, that's how WICkit is is built essentially. There's an event loop. There's a global state, and then there are a bunch of messages coming in. And all this, all that makes this possible is that we were those messages are totally ordered, and we are keep we're able to serialize them and save them to a file.
• 15:35
  And then with that, you can take a live dump, and replay all those events in order.
• 15:43
  Speaker 2: Yep. Andrew, I don't know if this is describing too much foresight to you, but I really got the sense when you were talking about this for the first time that you sort of had this in mind from the beginning. Right? That as you were designing, Wicket, that you were thinking about maybe not this specific debugger, but debugability in its mastermind. Okay.
• 16:08
  Speaker 3: You have to do this ahead of time. Right? Like, if you start so for instance, like, Rust makes this it's easy to build this. Like, it's easy to build many things in Rust, but you can't, like, do it after the fact. You you have to separate your IO, like, in any language.
• 16:22
  You have to, like, you have to do the do this purposefully. And so if you're writing, if you started out with a bunch of, like, Tokyo tasks, right, and those tasks were receiving messages and mutating local state and, like, spawning things and Done. The state was spread all over the place, you could never do that. Right? You can never Yeah.
• 16:43
  You can never or totally order those operations. And so I knew from the beginning that I wanted to totally order things because, a, it makes the system easier to understand. But, like, that's that's what a UI is. Like, UIs have been built this way for a long time, as far as I know. Right?
• 16:59
  At least I mean, it's possible that I've only read, like, UI papers from the seventies eighties. But, like, essentially, like, there's a there's some sort of main event loop and interactions are getting pumped through that. I think this is, like, somewhat how, like, GTK works and and other things like that. The the the main thing is that you separate all kind of mutating, states separate all your state mutation from, like, any nondeterministic operations. So one other thing you have to do to make this work is that, there's 2 real in this in this scenario, there's 2 main case of nondeterminism.
• 17:37
  3. So there's events flowing from the downstream system such as, like, a sled gets pulled and the event fires up, right, saying, okay. Here's an alert. This sled is no longer there. You wanna update the UI.
• 17:49
  There's also, a user pressing a key press to say do something, and that should trigger a redraw on the UI. And then there's, the third thing, which is just time. Right? If we wanna animate things, and there are some animations in this UI, we need to have, like, a regularly timed tick operation. Like, you don't wanna just have the main loop, like, setting a timer and then processing that concurrently because now those ticks are not ordered with respect to the other events coming into the system.
• 18:20
  And so instead, we have this channel, really a queue, which allows us to totally order all those events. So anything coming in, user key presses, downstream events from the from the rack, and timer events are all normalized into a Rust enum and then popped on this channel. And so now you have a total order. That total order just gets replayed through the state machine, and the state machine looks at what's happening and says, okay. What do I need to draw now?
• 18:44
  There are some optimizations there just to, like, not draw when we don't have to. Another idea I got from Josh, but it's essentially for feeding all these things into a single threaded event loop. And there there is Tokyo, in the works, but it's but but that IO handling and whether it's asynchronous or synchronous is is totally extracted out from the main loop, which does the drawing. It allows us to replay that that drawing, in a in a debugger later on.
• 19:12
  Speaker 1: Yes. I mean, a bunch of important stuff there. One, I think just and you said this, but just to really underscore the point, like separating out your IO from your logic is really important, And it allows that because it there's just so many positive things that come from that. And, conversely, when your IO is convoluted with your logic, there are so many things that become just much more difficult. It's like having a 2 pack a day smoking habit.
• 19:35
  It's just like, oh, man. There's gonna be so many so many health problems now that are just gonna be really hard. And it's, like, it actually if you could separate this out, there's a lot of things that that positive things out of it. One question that that was in the channel intro was, were you inspired at all by by Elm, which I know is taken it's I the the program I have for web browsers. I don't know if you you, you talked to Crespo about his, his affair with Elm, but, really interesting language.
• 20:00
  Speaker 3: No. So, like, you know, I am not a UI person. Like, this was, I I won't wanna call it my first, like, terminal UI or whatever. Like, I've worked on a few a few things, a few toys before, like, in college and elsewhere. But but no, I haven't done any real UI work and I was not super familiar with Elm or how it worked.
• 20:23
  I learned about that actually while I was was doing this. I was I was trying to dig up papers, and I learned that this is very similar to the Elm architecture, but it was after I kinda made this decision. So, like, this decision really comes from my background, which is building, like, distributed systems and, in particular, like, consensus systems. And so if you wanna model, like, a consensus protocol, you have a bunch of independent actors and they all maintain local state, and you can't in reality, you can't get a total order of overall events because they're remotely located whatever. In a simulation, you can certainly do that.
• 20:58
  And so if you model it the same way where each replica or actor in the system has its own state and is just getting fed events, no matter if those events are coming from the network or from a debugger, you can write property based tests and you can do all sorts of things to ensure that your each independent system maintains as you want it as a whole. And then you can, also pass output events and messages between those replicas all single threaded. Right? You you can you can basically run your whole system, your whole, consensus network in a single thread and total order everything and make sure that global invariance are upheld. Right?
• 21:33
  This is not a real real real test. Right? You're leaving out the network. You're leaving out a lot of non determinism. But, like, if you could get that things working and the inputs the only way you can get an input into these actors is through a message, then you're pretty confident that, like, barring stupid, like, stupid bugs, that your system's gonna behave the way you you want it to behave.
• 21:54
  And so I just built it like that. Like, I this is the I said this is the second time I built a a debugger like this. And the first time was for another project I was working on at VMware, that sadly got canceled for, you know, unknown reasons.
• 22:10
  Speaker 1: I'm sure, like, really awesome well founded reasons had nothing to do with
• 22:14
  Speaker 3: Always a good decision. Yeah.
• 22:15
  Speaker 2: It always and articulated to everyone.
• 22:17
  Speaker 1: Yes. That's right.
• 22:19
  Speaker 3: Very clearly spelled out. Not I I have no confusion about it whatsoever.
• 22:26
  Speaker 1: And and there's no lingering resentment, so it it all worked out.
• 22:29
  Speaker 3: Yeah. So I started building that from yeah. No lingering resentment at all. I didn't not every other every other position I took after that was was not a slight disappointment. It's fine.
• 22:40
  No. I I don't get me wrong. My time at VMware was great for the most part, and I worked on a bunch of interesting projects. But the thing I was hired to work on lasted much, a much shorter time than I expected it last. And so it was implementing the that project was something called Herit, and it was I was implementing, view stamp replication.
• 23:00
  The view stamp replication or visited paper from scratch, and I was doing it in Rust. And this was June of 2015. So if you know your timeline, Rust had just gone 1.0. There certainly was no Tokyo.
• 23:13
  Speaker 1: I don't know. When is that relative to your blog entry on Rust? That is about the same time, isn't it?
• 23:18
  Speaker 2: Oh, jeez. Yeah. I gotta look that up. What what month in 2015 was?
• 23:21
  Speaker 3: June.
• 23:23
  Speaker 1: Yeah. That's that's
• 23:23
  Speaker 2: right around that time. Yeah. Right.
• 23:25
  Speaker 1: Right. This is my first introduction to Rust is reading Adam's blog entry, and the well, I was, like, okay. Well, in conclusion, I will never use this language. This sounds extreme seemingly painful. And Adam's last line was like, in conclusion, this is kind of interesting.
• 23:38
  I'm I'm I really enjoyed this. And I'm like, what is you read your own blog entry. But, I mean, it was a I think that it was a painful time in a lot of I mean, it was a there were things that that you could tell when I arrived in 2018 that, a just gone through a big step function in terms of reducing pain, but I think in 2015, intro, Rust was raw.
• 23:59
  Speaker 2: Yeah. Yeah. It was to really you had to really want it. I mean, there were lots of times when the compiler would would tell you to go fetch rocks and then tell you those aren't the right rocks. Go get some other rocks.
• 24:10
  Speaker 3: That is very accurate.
• 24:12
  Speaker 1: Did the Rust compiler from 2015, did that prepare you for being, for being a parent of a toddler? Adam, just like, you know what? I was a Rust user in 2015. Like, you're gonna have to do more than this kid.
• 24:22
  Speaker 2: I would say better certainly better than any other programming language. I mean, your c plus plus is, like, close, but a a little more psychotic and and less sort of arbitrary.
• 24:33
  Speaker 1: Well and this is where you also have the, you know, the, certainly, the ownership model is a very is is a real is certainly new to was new to me with Rasta. New to
• 24:42
  Speaker 2: me as well.
• 24:43
  Speaker 1: And did
• 24:44
  Speaker 3: you just say c plus plus was less psychotic?
• 24:46
  Speaker 2: No. No. No. Pardon me. C plus plus more psychotic, kinda less arbitrary in its whims.
• 24:53
  I'm just thinking about which language prepares one better for having a child. It's better.
• 25:01
  Speaker 1: Yeah. I mean, the the the memory on safety and kind of ramp of data corruption definitely helps you prepare for for being a a parent, I think.
• 25:06
  Speaker 2: And Yeah. Every child I think it's the short of it.
• 25:09
  Speaker 1: Exactly. But so so, Andrew, you had been a, a Rust in the Rust user in those early days. And so I go because one thing I I would love to understand is building this debugger was super fast once you had this whole thing built out. It was, like, days is my understanding.
• 25:29
  Speaker 3: Yeah. It was about 2 days, which, like, it was just started. Right? It it's actually at a point today, so it's been a week ish. And now it has, you know, like, single stepping and, you know, you can do a few fancier things.
• 25:44
  You can you can play at different speeds by manipulating the the tick time. Since we're just sending messages, we can choose how long to sleep when we see a tick message in the debugger.
• 25:54
  Speaker 2: And has it has this been useful for for debugging some some problems that you've encountered while developing this?
• 26:00
  Speaker 3: Of course not. It's a total toy right now. No. Like, no. The real purpose so so part of the reason I did it was because I knew we could, and it was a weekend.
• 26:09
  And I thought I could get it mostly written over the weekend, because I just I don't know. I was bored, and I didn't wanna do real work. I had started working on, like, a real thing for work and decided I wanted to do something else that was tangentially work related, which is honestly how most of my projects go. Like, most interesting things that come out of my head are very, like, immediately relevant to the work I'm doing, but not strictly on the path to shipping a product. And so that's that's how this popped out.
• 26:35
  But, like, I had the idea that I wanted something like this. I wasn't sure it was necessary, but then I get excited. And so, like, the real driving motivation besides just me being bored and, like, really want to build this because I thought it would be cool to show visual state, was that we have to add we were talking about, so a coworker and I were talking about adding progress bars. And, also, like, we're trying to get this system running on the what we're calling the dog food rack. And I don't have full access to that rack at all points in time.
• 27:10
  And so I was thinking, well, it'd be nice if I could just simulate all the events I want in the UI. So when I add animations or change styling, I don't have to walk through and be plugged into a real rack to do this. But I wanna look at it as a human. I'm not just looking to, like, run a test and see if it pass or fails. Like, I wanna see how it looks.
• 27:26
  I was like, oh, would it be neat if I could just, you know, step through all the important parts of the UI that I care about while I'm making changes and then just run the recording after I make a change and see what it
• 27:36
  Speaker 1: looks like.
• 27:36
  Speaker 2: That's what
• 27:37
  Speaker 1: and you demonstrated that. That's what I actually love. It's like, alright. So let's, you know, let's see if we if we, you know, change this color or we change the way this thing looks or just you know, you're gonna make some cosmetic change, and we wanna, like, really understand quickly how that's gonna look. And now we can just go replay it and see and Yeah.
• 27:51
  Get this, which I thought was really neat.
• 27:53
  Speaker 3: Yeah. And so, like, you can see. So I don't think I did that on the demo. I mentioned it. But, like, I did do it for Benjie the other day.
• 27:59
  Benjie's our designer. But, like, you can, like, you can see where there's bugs. Like okay. So for instance, I created a little, like, style module, and it just it just contains some colors and, like, names them through functions. Right?
• 28:16
  It just says, like, for plain text, make it off white, whatever our specific off white color is. And so, like, you change that. And if you change that, you're gonna see all the places you're using that as you run through the recording. And, like, when I first made the change, I noticed that it did not change where I thought it was gonna change, meaning that something was hard coded in that place. Right?
• 28:35
  And so that was, like, really nice. That would be a really hard bug to catch. But, like, when you're just walk watching it and you see all the colors change, except there's one thing you expect to change, doesn't change, then you know you gotta go fix that spot. So, like, there's unexpected, like, serendipitous, bugs revealed to you that you probably wouldn't even notice if you were slowly, like, manually walking through things unless you were looking for them. Well, so there there there we go.
• 28:56
  Speaker 1: That's your answer to Adam's question. How's it help find bugs? That sounds great.
• 28:59
  Speaker 2: Yeah. And and could these could these event streams also form the basis of, like, CI where you're, kind of taking some recordings and replaying them into the new code to validate that everything's working properly?
• 29:12
  Speaker 3: Yeah. So this was a larger idea I had that I'm not sure I wanna build in particular, or at least not right now. But you could so you can imagine that these events are coming through and they're manipulating what is essentially a bunch of rectangles on your screen. And you can write some tests that say, when these mutations come in, I expect this rectangle to look like this, or I expect the modal to be up. If a modal's up, and, like, the background should be this color.
• 29:36
  And you can actually copy out the buffers. Right? And then check after you run it through up run some operations that the buffer mutations look as you expect. So you can totally automate, like, what the CLI is supposed to be. With 2 ERS, like, you have full access to the terminal buffer.
• 29:50
  It's not that big because it's just, like, some some characters on a screen. Right? It's not a full, like, 4 k resolution thing. So, yeah, you you can you can write tests through that. And, like, I thought about it.
• 30:06
  And so this system is that we've talked about recordings and how you can play your recordings. Well, so 2 ARS and Wicket allow you to resize the screen. So I operate in TMUX a lot, and I'm I resize the screen so it's, like, small like a laptop screen, like it will be in the data center. And when you resize, the placement of everything changes. And so if you were to use a, say, take your recording in a large screen, and then without doing any resizing and then run it on a small, like, debugger window, it's probably gonna crash.
• 30:44
  Just because the resize events are ignored right now in the debugger, and it's it's gonna try to place things outside your your terminal window, like, if your window is too small. But that is just like an artifact of not implementing it implementing that functionality yet. You could imagine that if we know we're running a TMUX, we can automatically resize the window. And, Rain, my colleague, they pointed out that, you could actually use, like, a PTY and so have, like, a virtual screen that's always the right size and then process resize events that way. So that would be a cool way to do it, but I haven't dug into that yet.
• 31:23
  And so right now, we're just using this to iterate fast on debugging. But it also is running in production or well, nothing is running in production now. But, like, it could be. It's enabled so that you can store up a certain number of events and then dump them, if you see an anomaly.
• 31:39
  Speaker 2: So, Andrew, if I'm if we're in the data center, you know, sweating next to the machine and something goes wonky, we can hit a keystroke, and it'll emit the stream of events to, like, bring back home to diagnose what went wrong?
• 31:50
  Speaker 3: Yeah. Yeah. Exactly. So it just dumps it to a text file, which you can configure a location. Oh, it's not text.
• 31:56
  It's serialized. What am I using? CBOR? A number of reasons.
• 32:00
  Speaker 2: Oh, gotcha.
• 32:02
  Speaker 3: But yeah. Like, what there's another interesting component there, which is, like, okay. You have the stream events. Well, how long are you gonna be sitting at this screen messing with it and ticks going off? Like, you can't let that stream of events build up indefinitely.
• 32:14
  You will take up a lot of memory, and your files will get quite big. And so, like, you typically wanna constrain the history of events. And again, because we have a state machine and we have access to our global state, we can take a snapshot of that state, when we have, you know, say, a max number of events or we've used a certain amount of memory. And that that allows us to keep, like, a a fixed memory usage while this is running and not kinda destroy the machine, which which lets us use it in production. And so, like, let's say you have a 1,000 events, and that's all you're allowed to record, and that'll give you whatever, 10, 15 seconds of recording time.
• 32:46
  Once that initial 10, 15 seconds is up, you take a snapshot, you clear out your vector of events, and you start building up from there. And so what that means is when you take a dump, you're not gonna start from the beginning. You're not gonna start from the oxide splash screen. You're gonna start from wherever, you know, within that last event window. But everything will still be consistent.
• 33:04
  And so if you have a couple minutes worth, and you see a problem, and you wanna hit a few keys and see what happens, you can just do that and then know that, like, 30 seconds later when you when you hit the hot key to take the dump that you'll have the the anomaly that you wanna capture.
• 33:19
  Speaker 2: Well and that seems huge for the kind of deep menued system that we're building there. Or if you wanna debug, you know, the the 15th step that someone's taking, it's or or not even debug it, but if you just wanna kinda keep on building on it, it's such a pain in the neck to have to start from the beginning each time. So this thing fast forward you to the part that you're interested in, that seems like a huge time saver.
• 33:40
  Speaker 1: Yeah. Well and, also yeah. And I love just the ability, obviously, to debug postmortem. I mean, as as our colleague Cliff said, shortly after he arrived at Oxide, y'all are really into postmortem bugging. We are we are really into postmortem debugging.
• 33:54
  But just the ability to to have that state that can then be sent elsewhere for purposes of understanding the system is just extraordinary. Andrew, something you said briefly that I wanna go back to because I think it's I mean, that this was in Rust allowed you to extend it in this new way or or develop this kind of extension from Citibank really quickly. And, I mean, I know yeah.
• 34:16
  Speaker 3: I don't I mean alright. Continue. Sorry. Continue the question. I mean
• 34:19
  Speaker 1: Well, I was just gonna say and I mean, you know, for anyone who influenced Rust, this will be old hat. But for folks that that, are new to Rust or haven't implemented in Rust, they may not realize the load bearing role that it has here as it has in so many things that Cerdee is playing here. Serde makes us super quick to to implement.
• 34:40
  Speaker 3: Yeah. So that is that is very much a key. Right? Like, I I cheated. Like, I have on so much other software.
• 34:48
  Right? Like, if you were working in c plus plus or other languages, there may be similar functionality. Right? But, like, Serde is the serialization and deserialization library built into Rust. So I was talking about Herit before.
• 35:04
  Serde also didn't exist when I was writing Herit. There was something called Rust
• 35:07
  Speaker 2: I was just saying that. Yeah.
• 35:09
  Speaker 3: It was the precursor, and it was okay. It was much slower, and it was definitely not as well documented. But, like, it works. So I was able to cheat back then as well, just not as well. Like, the cheating wasn't as effective.
• 35:20
  I got, like, an 89 on the test.
• 35:23
  Speaker 1: But so and describe a little bit how one uses Ceredex. This is one thing remarkable about it is how
• 35:29
  Speaker 3: Yeah. This is so this is, like, just amazing to me. Like, this blew me away. So, like, I've never personally written a procedural macro. I have some knowledge of how they're built.
• 35:37
  I've if you looked into it and always gotten distracted and gone on to something else. But I have used procedural macros in pretty much all my code, since I started with Rust. And so what a procedural macro does is it allows you to essentially give an annotation to some part of your code base, and it will automatically generate code, by reading that code and modifying it modifying it according to what you instructed to modify with your annotation. And so for Serde, what it does is it takes a structure or any sort of data structure in Rust, and you can say you can annotate it with the SERDESerialize and deserialize tags, and that will automatically generate serialization and deserialization code. And so that, is what allowed me to very quickly hook this up to just writing things to a file.
• 36:25
  So one question that arrives, okay, like, well, what serialization format? Well, that's the great thing about SERDES, it's pluggable. And so it defines, like, a visitor pattern, and then there are different, serialization formats that implement the SERDES, callbacks, essentially. They they they implement the serialized call for and so the macro uses them or generates enough information. I'm not fully aware of, like, what the path here is.
• 36:51
  You can you can you can combine various formats and SERDHE itself, and then all the user has to do is say use this format and use SERD A, and it generates serialization code of the right format for you.
• 37:03
  Speaker 1: Well and importantly, deserialization code. Because serialization code is I mean, it
• 37:08
  Speaker 3: can be True.
• 37:08
  Speaker 1: Yes, but it's actually like, it it it is much easier to write serialization code than deserialization code.
• 37:14
  Speaker 3: Yes.
• 37:15
  Speaker 1: And, rarely is one good without the other. So you you needed to both serialize and deserialize here. And, I mean, I've just done this over and over and over again, where the ability to just literally annotate a structure by denoting that it's gonna derive the serialization and deserialization traits, and then, like, it all happens just magically, and you get this really robust deserialization, which is not which is I feel that that certainly speaking from my past self, mean, I would YOLO the serialization, and the deserialization would be, for something like this, would be, just the max power way, as we say. Making a Simpsons reference, the wrong way, but faster. And whereas when you insert it, like, you're actually getting, like, really robust deserialization here as well.
• 38:00
  Speaker 3: Yeah. It will not deserial so, I mean, it depends what format you're using, etcetera. But, like, when you say deserialize this struct, you know essentially what type you're gonna be deserializing into, and SIRnet will attempt to deserialize into that format. And if it doesn't, it will give you an error, and it'll tell you where it errored out. That's not true for every, like, limitation of every format and and inserted, but it it works pretty well.
• 38:23
  And, like, the way, Wixit is using it and the and the Wixit debugger is using it is it's kind of the interface between the 2. Right? Like Wixit, like, I these were separate commits. Right? Like, Wicket, I I had the serialization code.
• 38:37
  I dumped out all the events of the file. I created the snapshot format, which has that state that we talked about and the events. And I dumped that out to a file and then the debugger came later. It just read that file, deserialized those events, and and played them replayed them. So there were some abstracting of shared codes so that we would play things the same way we do an actual Wicket, so that, like, you're not not writing 2 separate implementations that can diverge.
• 39:02
  And so there was a little trickiness there to get that how I wanted it to. But for the most part, like, the the the interface layer between those two things is just serialized events, and it's automatically serialized. Like, I didn't do any work to, like, pick you know, you know, do anything with that format.
• 39:20
  Speaker 1: Yeah. When I hear the it feel this is, like, one of these examples. Right? And I know there's there's a lot of, you know, discussion happening out there about where is Rust a good fit, where's Rust not a good fit. But it's the presence of this kind of stuff that that, at least at Oxide, we see a really broad use case for an up and down the stack in pretty much.
• 39:39
  It just feels like there there's lots and lots of software that would benefit from this pattern and and this approach and kind of the ability to have a, a language in an environment that could do that has been a big, big win for us, I dare say.
• 39:53
  Speaker 3: Yeah. I mean, in the past life, I've used Tcl for Python to do something quick and dirty like this. And I I wrote, like, our own IDL, like, protobuf IDL from my past job, you know, in c plus plus. Yeah. I I
• 40:06
  Speaker 1: did one of these I did one of these in Pearl. That was that was that was a mistake.
• 40:13
  Speaker 2: That that was probably, like, 2,004 or something. I don't know. I feel like you can based on the language based on the choice of Pearl, you can really pin it down.
• 40:22
  Speaker 3: Yeah.
• 40:23
  Speaker 1: Well, no. That would no. No. Well, no. Sadly, this would have been 2004.
• 40:26
  It it tells you something that I was doing this in actually 2,008. So, I probably it'd have been 2004 for me if it was more defensible decision. But, and that was actually where I ended up redoing that the the that CLI in JavaScript, actually. I know that's kind of what the or JavaScript that was in the note. My first kind of realized that you could use JavaScript in and it was, like, it was useful, but, boy, nothing compared to, what we can go to with with Rust and being able to to apply this just ridiculous power tool, throughout the stack has been has been great.
• 41:02
  And then so, Andrew, in terms of, like, applying this kind of pattern, because this kind of feels like the all of the fun of distributed systems without any of the, like, real pain in the ass of distributed systems. Am I wrong about that?
• 41:16
  Speaker 3: No. You're exactly right. Because there are no communicating processes here. Like, I'm just reporting events from a single threaded thing anyways. Right?
• 41:22
  Like, yeah, there are downstream services, but, like, those are just rest services, and they're just, like, we're just pulling them and grabbing, like, inventory updates. It's not there's no complexity on the distributed systems front here. But when you I I guess, like, what Brian said to me early on is, like, as, the semi, you know, elderly engineers, like, we have this synthesis this synthesis of knowledge. Right? And you you learn that you can combine things and take your experience from one place and use it in another.
• 41:51
  It's not always the best idea, but in this case, it worked out nicely.
• 41:56
  Speaker 1: Yeah. I the the where I I do think that that, and I think there's also a lot of value in when you've got these parts of the system that everyone is kinda looking at someone else to go implement, that is really important. Right? This is ultimately this is the first actual software that a customer runs on the rack is is is the software. And instead of viewing it as regrettable, you're viewing it as a real opportunity to, like, okay.
• 42:21
  Wait a minute. How I I don't have to implement this in the same way that it's been implemented. I I can actually apply all these new techniques. And how can I go do this in a way that's gonna make it faster to implement and it's gonna make it more robust and and so on? So I think that there's a lot of value to that kind of versatility where you're going into a new domain that you that is and you said, like, this is not you you don't do a lot of UIs and I lot of terminal UIs, but this is a real opportunity to apply that distributed systems wisdom to a new domain.
• 42:50
  Speaker 3: Yeah. It's I mean, really, you should take any opportunity. I mean, I just like, me personally, I do not mind digging into new domains, like, as long as I have the the time and the right support system. Right? And Oxide's Oxide's great for that.
• 43:04
  Maybe not so much on the time front lately, but, like, but having, like, just the ability to dig into something new is great. And, like, if it's needed, like, you you feel that there's value there, you can go dig in, but you learn. I mean, there's just an ability to learn something new, and really, like, you can really dig deep and, like, essentially, this was this was totally demo driven development. Like, I did this to show off my peers. I wasn't like this was not like, I don't know.
• 43:30
  Like like, this definitely adds value, and I think the customers are gonna like it. I think it looks great. I think it's gonna look better for real. Like, it'll it'll it'll continue to get better with more people working on it, right, and more designs, like, skills and and coordination across everything. But, like, the impetus for it really was to, like, make it look cool for the demo at first, and see and see how well it worked if I get it done.
• 43:55
  Right? And so, like, the first iteration of it was about 2 weeks of the work and and had a bunch of, like, very weird, like, UI rendering and it had, like, mouse over events and, like, the bulk of that was thrown away. But, like, the core of it being a text user interface and showing a rendering of the rack and other things were were kept. And so it if you have time to experiment, it's always fun to step outside the your your own comfort zone and see what you can do. Honestly, that's been my whole experience at Oxide.
• 44:22
  I have a, an interesting position. I came in thinking I was gonna be doing one thing and kinda did a bunch of other stuff. It's been interesting. Just like in VMware. Yeah.
• 44:34
  That's true. It's true. In a different
• 44:37
  Speaker 2: way. Yeah. In a very different way.
• 44:39
  Speaker 3: Yeah.
• 44:39
  Speaker 1: You know, I I I wanna make a joke about canceling projects, but I can't even do that. It's just too cruel. I'd also I I I so hate it when the it just brings up the the this is in in many ways, like, my, there are many aspects of Soul of the New Machine that Meredith reread, but, the fact that the birth of that project is when when one project, prevailed over another in the the shootout at Hojo's. It's where they they and I and that that definitely stuck with me reading that on the the second read. Sadly, all the shootouts at Hojo's.
• 45:13
  But no shootouts at Hojo's at Oxide. So this is all, all very, very good stuff.
• 45:18
  Speaker 3: Yeah. I mean, we have so much to do. It would be hard to have a shootout. Like, I think we can all find our own domains to be dominant in. Yeah.
• 45:26
  Speaker 1: Exactly. That's
• 45:27
  Speaker 2: right. No no no reason to, like, try to squeeze someone out. Brian, you, you broke the seal on JavaScript. I did wanna mention I mentioned this to Andrew. Have you checked out this thing called replay.
• 45:37
  Io? I learned about it because our, our colleague, Justin, runs another podcast, called devtools.fm, which is great. And he had recommended this episode, and it's a replay debugger for JavaScript in the browser, which sounds tantalizing. The internals of it are ludicrous because, like, it has you know, JavaScript and browser has none of the properties that Andrew was able to take advantage of. And so they're basically inter both interposing on system calls to get those raw events.
• 46:12
  Yeah. Exactly. Taking advantage of the fact that JavaScript is, you know, mostly single threaded, but then also getting deep in the underwear of the, the virtual machine in terms of, like, locking primitives and stuff like that. Anyway, a neat system. Reminded me of some of the stuff Andrew had had was talking about, and he's built in in the WICN debugger.
• 46:37
  But, also, like, you need to start recording from the beginning of time, and desynchronization is a huge problem. Anyway, beat system.
• 46:45
  Speaker 1: Yeah. When you also have there there there's this kinda tension about how general purpose versus special purpose your debugging infrastructure is. And that there on the one hand, if you've got general purpose debugging infrastructure that can debug any program or debug any JavaScript, you can obviously, it's got a very broad service area. On the other hand, it can be really nasty to get a lot of that stuff to work. And, I mean, the one thing I love about, Sandra, is this is very special purpose debugging infrastructure.
• 47:11
  This is just for this application effectively. But as a result, it's it's extremely powerful, and there are a bunch of problems that you don't have to solve because of the way you've architected the application. And I think that we don't spend quite enough time on application specific debugging infrastructure. I think it's really, really powerful, and I think that it it so often, will help inform good architecture decisions when you think about this from from the get go.
• 47:36
  Speaker 3: A 100%. Like, this is the same I mean, you can make the same argument for certain types of testing. Like, I I don't buy the whole, like, certainly, I don't wanna get into this rabbit hole, but, like, test driven development, building out the perfect architecture by, you know, writing your tiny unit test functions. But I do think, like, more sophisticated testing strategies can let you know when you can't see whole subsystems, in your test process because they're just inaccessible. So, like, property based testing, I think, is a is a good way to exercise large portions of your system, with a large input space.
• 48:11
  Speaker 1: Can you elaborate on property property based testing? Because I actually had not really seen it prior to oxide. I know it's it's it's it's out there, but could you just elaborate on it a bit?
• 48:20
  Speaker 3: Yeah. So so it's actually kinda similar, to what I built here, which is that you have at least the ways I I would use it for distributed system. So there's 2 separate property based testing. I guess the simplest way to describe it is you would have a function. Let's say you have a function that, reverses, like, what's gonna do is, like, reverse twice.
• 48:47
  Let's say that this is a simple, like, example that most property best testing, like, tutorials go through. It's like you have a function, and it's gonna reverse a list twice. And when you reverse a list twice, the property is that you should have the same list you started with. Right? So what property based testing does is it generates a bunch of inputs automatically, giving, like so you give it some sort of generation function, and it'll generate inputs that correspond to the right type.
• 49:12
  And then the the test infrastructure will do that, And then it will run through your actual function. So you're not writing you're not, like, writing 7 or 8 individual tests giving, like, various tests manually, giving, like, various inputs manually saying, like, here's a list a, b, c. Here's a list a, b, c, d. Like, the property based test will generate those, and it'll it'll start growing them and getting more random, over time as it runs. And so it's just testing that the property that you want, you can name that property, like, reverse twice, gives me the original list.
• 49:43
  That could be the name of your property. And all it's doing is checking that that assertion holds given a set of inputs. And so it has to be deterministic. Like, otherwise, you're gonna be in a world of hurt. Because, like, if your test just started not doing that properly, coding bad.
• 50:00
  But, like, this is this is why, like, the the separating your IO and making things deterministic, this is how I kinda learned to structure my code that way and how I ended up with learning to build these debuggers. Because in order to make distributed systems property test deterministic, you you don't wanna just like they're you don't want them those tests to call into, like, the local clock while they're running. You want them to pass in the time or the duration for an expiry event. Right? And so that way, every time you run through and you get that same order of events, you're gonna get the same result in the test, and so they're not flaky.
• 50:34
  And so, like, you want that staple manipulation. And so, like, that's the the simplest form of that's of a stateless test would be just generating a bunch of inputs and passing it to a function that's like a pure function. Right? But, like, there's also the the stable version is you have, an actual models. You have you have a the state of your system under test, and then you have a model state.
• 50:57
  And so you're actually writing a separate state machine for the model that's perhaps simplified, And you're running the events. You're taking you're generating a list of events again, and then you're running it through the model, and then you're running it through your your system under test, and you're comparing that the model your assertions your properties now are that the invariance for the model actually hold and match, like, the the system under test. So for instance, you can say, the sequence numbers will never have duplicate sequence numbers in this, duplicated log. So, like, you'll never you'll never see, or you'll never see diverging agreement among all the replicas. And so all you need to do to keep track of that is to essentially maintain in your model, that the replicas alright.
• 51:48
  I'm confusing myself right now. But, in essence, like, you're maintaining, like, you're maintaining a model and you're maintaining the variance, for that model and then comparing them to the system system under test and making sure that they match. And the property based testing part of that is just generating the inputs and running through those assertions and invariance and allowing you to see if they match. What makes it really interesting is that most property based testing tools, when they fail, they give you a history. And that history can be really long and complex.
• 52:20
  But they run through and they, through a process called shrinking. And shrinking actually reduces the state space. So it it steps through. It kinda does a binary search to see where the test would fail. So it starts running tests with similarly patterned inputs that are smaller to try to see if it can get the minimum sequence of events that cause the test to fail.
• 52:43
  And that is a really nice thing because that allows you to more easily debug it. So, like, if you have if you have 2 things, if you have the property based testing infrastructure that can run through your system tests and show you the failing history, and you have a debugger that knows how to play those same events, you can now take that recording of the failing test and run it through your debugger. Right? And so these would be all application building. But if you structure your code that supports property based testing, it's most likely gonna support this type of event replay debugger, and you can use the actual failing test, to run through the debugger and inspect the state and see what's going
• 53:17
  Speaker 1: wrong. Yeah. That's really cool. And so Adam dropped in a link to the PropTest crate, and I think that this and I I this started my understanding is with quick check. So, Nick in the chat dropped in a a link to to QuickCheck from Haskell.
• 53:35
  And is it is this kinda coming from the the Haskell side of it? I mean, have you used what have you used
• 53:42
  Speaker 3: for the property? So I think I believe it was introduced, with Haskell for Quick Check, and that was like a a John Hughes jam. No. That was Simon Peyton Jones jam. But John Hughes was the, so I first started with property based testing in Erlang, when I was working at Basho.
• 54:00
  And so wrote a lot of property based tests and stateful property based tests. And we used a proprietary tool from Cubic, called I noticed. It was like early in quick check. It wasn't a very creative name. But it was an awesome tool.
• 54:14
  And it had, like it actually did much more sophisticated things than either the original QuickCheck as far as I know, and, like, any of the open source tools for Rust. Like, it would generate symbolic inputs, and so it would, like, symbolically shrink the state space, and it had all sorts of funky stuff. It would also do randomization, of the, like, run time. So it would not only, like, manipulate the inputs, but it would also manipulate the Erlang scheduler under the hood through something called Pulse. And there's a Tokyo project, I think it's called Loom, that does something similar, to test, like, Tokyo.
• 54:48
  And so it manipulates, like, the the state space, or the underlying, like, schedules of the of the tasks that are running. But then when I wrote when I was working on Harrow, PropTest did not exist, so I used, QuickCheck, the QuickCheck implementation at first. And I kinda built up on top of it, to build a a debugger there, and and write some tests. But here, I've used, prop test exclusively. So I haven't written as many property based tests as I would have liked before.
• 55:18
  But, yeah, I I use PropTest at Docside.
• 55:22
  Speaker 1: And would and would that be your recommendation for for folks who know the rest and looking to
• 55:26
  Speaker 3: get Yeah. So QuickCheck, they're different. Right? Like, QuickCheck is a cool tool. It I think it was influential.
• 55:34
  A QuickCheck for a while, I'm talking about, and that it introduced the arbitrary trait. With quick check, like, there's a there's a why, like, PropTest versus QuickCheck, I think, in the PropTest book. And so, like, there's a, you know, a doc, e an ebook that that comes with the PropTest, open source stuff. And it explains, like, quick check, does it shrinking and generation based on types? Like, you divine the the shrinking and expansion, like, generation and shrinking, from prop tests are much more coherent.
• 56:03
  So, like, you can derive, you can take a type and then compose them and, like, generate new outputs that aren't strictly mapped to, like, a rust type. And it will shrink in the same order that those outputs are, like, derived. And so, like, it always kinda does the right thing. You can do a few more complex things, I think, with PropTest. And it's got a a few more tools, at its disposal.
• 56:26
  And it's got a really nice, like, introductory book. I think they say it's mostly feature complete. I don't think they're accepting, like, new features to prop test. It just works. Like, you can build things on top of it.
• 56:39
  I, like, I would recommend using PropTest. Yeah.
• 56:42
  Speaker 1: Yeah. This is something I really need to personally add to my arsenal. I really have not. Adam, have you have you used prop test?
• 56:47
  Speaker 2: No. You you and me both. That that link to prop test is, like, one of my, forgotten tabs of, like, must read material that seem to pile up at an alarming rated oxide.
• 56:59
  Speaker 3: Yeah. It's I will say, like, don't be intimidated. It's it's much easier than probably most things you guys have done. It's probably slightly easier to use than implementing DTrace.
• 57:11
  Speaker 2: But is it easier to use than using DTrace?
• 57:14
  Speaker 1: Yeah. But I'm I'm I'm semi elderly now according to
• 57:16
  Speaker 2: you, Eric. Semi elderly. Thank you for that, Andrew.
• 57:21
  Speaker 3: Yeah. Well The
• 57:24
  Speaker 1: so and actually this is kinda like and we we should actually find the do an upside of this that the, the UASIS, I'm probably pronouncing mispronouncing that in the chat is pointing out that, a lot of the stuff that that I would say we like about Rust, existed first in Haskell. So build RS and quote, drive macros, traits, etcetera, which is kind of interesting. I mean, that the it's actually honestly, it's part of what I I love about Rust is that it is drawing on a bunch of different folkways. And I look forward I know I've I've joked with Steve Gladnick that I I personally look forward to the HoppleTalk on Rust about pulling in all these different things. Because, you know, if you had told me a decade ago that there were things in Haskell that would be really meaningful to me, I I would have been curious for details because that would seem plausible.
• 58:13
  Speaker 2: Uh-huh. Curious and incredulous. Yes.
• 58:15
  Speaker 1: And curious and maybe incredulous. I well, I I I yes. Maybe incredulous. In in part because I think that the, you know, what Russ has done is kind of taken a bunch of, I mean, Haskell is, gonna get myself into huge trouble here. But, I mean, Haskell has neophilia.
• 58:33
  Right? I mean, it's like it likes there's it's a great laboratory for exploring different ideas. And when you combine all of those ideas Adam, do you remember do you remember being an AAD bug in 2003? I mean, obviously How
• 58:45
  Speaker 2: could I forget? I I one of my favorite conference citizen I've ever attended, I say somewhat ironically.
• 58:50
  Speaker 1: I'd say that completely seriously. That was so much fun. This is the so, Andrew, this is the automated and algorithmic debugging conference, and, which we were I I mean, I was so I I thought it was
• 59:02
  Speaker 2: so much fun. So much fun, but maybe not my favorite conference, but I did have a great time.
• 59:07
  Speaker 1: That so much happened to that conference. There there
• 59:10
  Speaker 2: was Yes.
• 59:11
  Speaker 1: I felt like I was going to the Olympics. I was so excited. I was Oh, because because they didn't
• 59:16
  Speaker 2: have it every year, and Brian had gotten a paper in, a terrific paper in. And we thought this was going to be truly the Olympiad of people who loved debugging as much as us.
• 59:27
  Speaker 1: And it was a ragtag band that, there were, like, 60 people there, half of whom were I definitely learned about, boy, the French prologue mafia. Do not do not mess with those people. Like, you end up in a dumpster if you I've do you remember the they had some sharp opinions about about prologue? But then do you remember that the the, person presenting on Haskell and, needing to solve an NP complete problem that generate an error message. Do you remember this?
• 59:55
  Speaker 2: I do remember this. Yes. Keep thinking
• 1:00:00
  Speaker 3: wow. What
• 1:00:01
  Speaker 1: a what a language.
• 1:00:02
  Speaker 2: Language. Oh my goodness. I gotta
• 1:00:04
  Speaker 1: do a traveling salesman problem. Actually get to a TSP to actually, Hamiltonian circuits in order to be able to generate an error message to understand why your why your type inference is failing, which I that was kind of, like, the only thing I knew about the language for a long time. So it it definitely did not feel like, it it it felt implausible that it was gonna have immediate impact, but it definitely is.
• 1:00:25
  Speaker 2: Also, dear listener for you thinking, oh, like, I can't wait for the next a a debug so I can attend. That was actually the last a a debug. That's
• 1:00:33
  Speaker 1: that was the last AA debug because they can't, like, they can't get a space. They got kicked out of the Denny's or whatever, and now they can't get a space. I mean, it's it is it's really pretty sad. I and we thought it was only happening every few years, every Olympiad because it was so so extraordinary that you just couldn't have it every year, but it's like, no. No.
• 1:00:51
  They can't get their shit together every year. This is also the the conference in which, well, I mean, a couple of things. This is the, Adam, you had had the idea when we were leaving, you told me that you were very upset that California had taken away our right to eat horses. Yeah. And in particular, the as California is willing to do, we had just passed a ballot initiative banning the the human consumption of horse meat.
• 1:01:19
  And, you take that as personal offense. I think it's fair to say. And you and you were like, you know, I'd never wanted to eat horse meat, but now obviously I do, and we're going to Belgium for this conference, Ghent, Belgium, amazing city. And we are gonna and I just remember you hatching this, like, when we were leaving, like, I wanna eat horse in Belgium. And I just felt like I I mean, I it was my calling to help you on your quest.
• 1:01:44
  Speaker 3: I mean, I'm sure
• 1:01:45
  Speaker 2: I'm in. Let's go.
• 1:01:46
  Speaker 3: I always think of horse meat when I think of Belgium.
• 1:01:49
  Speaker 1: Well, okay. There you go. So we are so well, so now it's important to say that this is like definitely pre, internet on the phone era. This is 2003. So it's like post internet, but kind of like barely.
• 1:02:03
  And in particular, we've got no way of and we don't speak the language. I mean, it's Flemish. I speak French.
• 1:02:09
  Speaker 2: They they speak the language, you know, our language. But yeah.
• 1:02:13
  Speaker 1: Yeah. But we're looking for but but we're looking for horse. We need to go we're looking for horse meat. We need
• 1:02:17
  Speaker 3: to be we need to be able to go into this call. Yeah.
• 1:02:20
  Speaker 1: Yeah. What we learned is that, like, horsemeat is not actually you can't just, like, show up in Belgium and ask for horsemeat. It doesn't work that way.
• 1:02:26
  Speaker 2: Actually, we we learned something else too. We learned that horse is slang for heroin, and everyone at the conference thought we were really trying to party extremely hard even for AA debug. Well,
• 1:02:38
  Speaker 1: because we said we wanted to score some horse, and it was illegal in California. As long as we're in Belgium, we wanted to score some horse. And they're like, you guys look like you have your okay. What? It's like, no.
• 1:02:47
  Don't We were
• 1:02:49
  Speaker 2: just gonna have some beers after it, but I I mean, you guys do
• 1:02:52
  Speaker 1: you. Like, come on. When are we gonna get back to Belgium? Let's get some horse. And then yeah.
• 1:02:56
  Okay. No. No. It's not. And then we had the whole Water Zooey fiasco where it turns out we thought we we were convinced that Water Zooey was horse, but it turns out that was hippopotamus.
• 1:03:03
  Were they stirring hippopotamus to that restaurant? I feel we never got to the bottom of that. It feels like it would have been more exotic. Anyway, we finally found and then we finally passed, and, Andrew, just to your point that you think about from everything of horse meat, we passed a butcher shop that had a giant horse's ass. Right, Adam?
• 1:03:19
  Am I imagining this?
• 1:03:20
  Speaker 2: Yeah. No. This is all true.
• 1:03:21
  Speaker 1: This feels very dreamlike right now. But there's a giant horse's butt, like, hanging, like, a shingle outside of this thing. We're like, these guys have to have horse.
• 1:03:30
  Speaker 3: But I feel
• 1:03:30
  Speaker 1: like, do they have horse in the end?
• 1:03:33
  Speaker 2: No. No. Sure.
• 1:03:34
  Speaker 3: It wasn't like a centaur and you just didn't see the front?
• 1:03:37
  Speaker 1: I did. Might have been a centaur.
• 1:03:39
  Speaker 2: We went the whole the the the the short story made very long. We got but we got no horse.
• 1:03:46
  Speaker 3: It's a shame. Well Feel good for the horse
• 1:03:49
  Speaker 1: to make it even longer by having no. The we got no horse, but the the store was closing in 10 minutes, and I had no cash. And we and I had to run across town for an ATM.
• 1:03:57
  Speaker 2: Yeah. We ended up we had another inspiration too, which is we wanted illegal cheese. That is to say Oh, that's unpasteurized raw milk cheese. That was our downgrade from
• 1:04:06
  Speaker 3: from the horse. Yeah.
• 1:04:07
  Speaker 1: From horse.
• 1:04:08
  Speaker 2: Yeah. Yeah.
• 1:04:09
  Speaker 1: I'm sorry, everybody. This is way down there. Who did this? This is I I but, you know, when when Andrew said send me Elderly, I just obviously took offense to that, and I wanna show that I'm actually full Elderly by just I
• 1:04:19
  Speaker 3: was paraphrasing the Elderly. I thought experience experience was the word I was looking for. I just
• 1:04:23
  Speaker 1: No. I take it as praise.
• 1:04:24
  Speaker 3: Rambling. I take
• 1:04:25
  Speaker 1: it as praise.
• 1:04:26
  Speaker 3: Look. I exactly. It goes on and on and on, and
• 1:04:31
  Speaker 1: there was a horse's ass there. Yeah. Give give me 2 bees for some horse, you'd say. Well, and this this is fantastic. Great work.
• 1:04:42
  That was really, exciting stuff. And I think also it's it just feels like there's a lot that a lot that is practical that you can take from here into whatever software you might be writing. This is all open source. I dropped a link to your PR, Andrew. Hopefully, that's the right thing to drop.
• 1:04:58
  But, this is all part of Omicron, which is our control plane. So if you wanna go explicitly check it out, definitely go for it, and, look for the intro view. I think you've even integrated the debugger now to avenue. I I know it's,
• 1:05:13
  Speaker 3: that It's
• 1:05:13
  Speaker 1: all the way anyway.
• 1:05:14
  Speaker 3: That PR is the debugger, I think.
• 1:05:17
  Speaker 1: Oh, nice. Yeah. There we go.
• 1:05:18
  Speaker 3: Yeah. That's what I that's what I opened up today. And it was just because I wanted to add the single stepping and, like, fast forward functionality, essentially, before I opened it up.
• 1:05:27
  Speaker 1: Well, a lot of great optical lessons. And then there was one other question in the chat. Like, what you call this model? I mean, to me, it's kind of an actor model. I mean, I'm I'm not sure
• 1:05:35
  Speaker 3: I've Yeah. I'd say, like, just is there Like, event driven state machines. Right? Like Yeah. Right.
• 1:05:41
  So there's, like, there's another model. I guess it's a it was popularized, I don't know, maybe 10 years ago or something. And it's not always useful for a variety of reasons, but, like, you have this queue, like CQRS. Right? Command, query, replay.
• 1:05:55
  I forget what it stands for. Essentially, you have this queue of objects and that allows you to do the same thing. Or this this queue of, like, HTTP request or whatever, you know, and and you can replay them through your downstream system. And so, like, people were building similar systems like this so it's, like, they could debug their business logic, essentially, by recording all the information that came in from the outside world through this queue. I mean, the key problem is, like, now you're recording an indefinite amount of information, and you can't take, like, a global snapshot of all your microservices, like, consistently.
• 1:06:28
  So, like, it doesn't it doesn't really work for very large systems. You can think about modeling many systems like this. And yeah. I mean, that's that's one of the takeaways. Another one is just, like, if you structure things correctly, like, this is it's a few days of work to to really build a debugger, especially in a language where, you know, crates are, like, tools, like, libraries or open source that you can just build upon.
• 1:06:52
  Like, I didn't build my own, like, read line library, and I didn't have to wrap anything in our l rep. And, yeah, I didn't build my own command line parser. Like, it all those libraries exist in Rust, and, it's the ecosystem as much as anything that makes stuff like this possible to do quickly.
• 1:07:11
  Speaker 1: Awesome. Well, great work, and, should you have an oxide rack, this will be the first thing you see is is, Andrew's software. You'll see a really amazing terminal experience on the headset. Had this had, much of humanity's scarce resources poured into it because it is it's gorgeous. So great stuff, Andrew.
• 1:07:30
  Thanks for joining us, and, I I'm just gonna tease next week a little bit, Adam, if you don't mind.
• 1:07:37
  Speaker 2: Sure.
• 1:07:38
  Speaker 1: We so next week, we are gonna be talking about the cable backplane, and in particular, a bit of a mishap of the cable backplane, and how we debugged it. We're gonna have, some terrific guests. We're gonna have a mechanical engineer from our, our manufacturing partner, and we are gonna have, an engineer from Semtech, which is our cabling partner. So we've got, 2 terrific engineers along with, with Robert Keith here at Oxide, and that one is gonna be a really fun discussion. So join us next week for our special guests.
• 1:08:13
  Speaker 2: And and is that on Monday, Brian?
• 1:08:15
  Speaker 1: That'd be on Monday. Yep. Great. Excellent. Alright, Andrew.
• 1:08:19
  Thanks again. Yeah.
• 1:08:19
  Speaker 3: Thank you.
• 1:08:20
  Speaker 1: Take care, everyone. Bye.