Latest / Oxide and Friends / Emulating Oxide: How Faux Can You Go?

Transcript

Emulating Oxide: How Faux Can You Go? · Oxide and Friends

• 0:00
  Bryan Cantrill: Hello, Adam.
• 0:01
  Adam Leventhal: Hi. How are you?
• 0:02
  Bryan Cantrill: I'm doing well. How are you?
• 0:04
  Adam Leventhal: I'm doing well as well. I don't have any thrilling sports endeavors to report from the weekend, but, You don't
• 0:12
  Bryan Cantrill: you don't
• 0:13
  Adam Leventhal: I do not. I do not. Unlike last week.
• 0:15
  Bryan Cantrill: Yeah. You know, first, I thought you were apologizing advance for people who had baseball in their bingo card. I thought I I I know. I I was raised impact.
• 0:23
  Adam Leventhal: No. I said no. No. I meant I apologized because I thought we weren't gonna mention baseball and here we are.
• 0:28
  Bryan Cantrill: Oh, I thought it was gonna be like, I was getting ready for like a a baseball brain teaser. Like, you know, like the the you know, like the fourth out? You you you see one of that this
• 0:37
  Adam Leventhal: Oh, Yeah.
• 0:39
  Bryan Cantrill: This was somewhat recent. Right? It was at Red Sox
• 0:41
  Adam Leventhal: Yeah.
• 0:41
  Bryan Cantrill: Jays or something? It was Yeah. That's right. Somewhat recent. Yeah.
• 0:44
  Yeah. A couple weeks ago. Yeah. So I thought this was gonna be a fourth out. And you know what?
• 0:48
  I I guess I guess it did turn out to be a fourth out brain teaser. So here we go.
• 0:51
  Adam Leventhal: There you go. Enjoy everybody.
• 0:53
  Bryan Cantrill: We'll leave it to to John Boy to go tell you about the fourth out. Well, am Oh, I do. Okay. The other thing we should update folks on is we got the recording up from last week. And one of the first comments online is like, I for one would look forward to LLM based property disputes, which I thought it was.
• 1:13
  You promptly told Kevin at the blue sky not to encourage me, but it was too late. I was encouraged. So I think we're clearly gonna have to do that as an episode.
• 1:22
  Adam Leventhal: You think just one episode, not like a three part series?
• 1:25
  Bryan Cantrill: I'm I'm really thinking more like a mini series, more like a a roots kind of a thing. A kind of a 14 parter.
• 1:31
  Adam Leventhal: Yeah. That's right. See if Ken Burns is available.
• 1:34
  Bryan Cantrill: Yeah. Right. See if Ken exactly.
• 1:36
  Adam Leventhal: That's
• 1:36
  Bryan Cantrill: right. The the no. This is yeah. This is like a neighbor civil war. It is I'm I'm sure this is the HOA civil war.
• 1:44
  I I just feel that this
• 1:45
  Adam Leventhal: That's right.
• 1:46
  Bryan Cantrill: This is a very rich vein.
• 1:47
  Adam Leventhal: Right. No. He would he would bring he only he could bring the gravitas needed for this delicate subject.
• 1:52
  Bryan Cantrill: My dearest Margaret, they sent me another response today riddled with errors that have been found with fable. It feels like feels like it kinda writes itself. I just I don't know. The no. A lot a lot of a lot of exciting stuff happening with LLMs and and property disputes.
• 2:10
  But that's that's not why we're here. We've got, we've got Steve Cameron here with us, which is very exciting. Steve, you have only you have not yet been at Oxide for a year. Does it feel like five years to you?
• 2:24
  Steve Karam: It feels like it feels like five years,
• 2:26
  Bryan Cantrill: maybe more. That is such a relief. Okay. That is such a relief because it really it it feels I mean, the last year has been very eventful all around. But, I was looking at him like, my god, Steve hasn't even been here for a year yet.
• 2:41
  Steve Karam: Yeah. Be to be fair, Bryan, I'm, you know, I'm used to that that normal world of, you know, corporate IT where after q four is over, everyone feels like they can take a break for about three quarters, and we don't do that.
• 2:53
  Bryan Cantrill: Right? We don't. We have also it just feels like we have lived many lives. It's it it just feels like we've had a lot going on, in the last year, especially. And so Steve, you were, you joined us for a role that we kind of invented, which is always perilous, which we call product assurance.
• 3:16
  It's always a little dangerous to make up a title for something because on the one hand, do run the risk that literally no one knows what you're talking about. Steve and I felt like we needed something that was going to What is the role that assures that the product you're making is the one that customers want from a quality perspective, features perspective? And so we call that product assurance. Think are we the only ones calling it that? I think we probably are.
• 3:44
  Steve Karam: Maybe, you know, I kind of like it. You know, I enjoy roles that everyone thinks they know what it is and they don't. You know, that's, you know, I enjoyed my stint working in enablement because it's such a loaded word, so is assurance. So if you think about, like, you know, quality assurance, which is what most folks assume
• 4:02
  Bryan Cantrill: Yes.
• 4:02
  Steve Karam: But then you see the product in there, and you're like, hey. What, you know, what kind of magic is this working up? And, you know, I have come to really enjoy the role and kinda how it sits between engineering and every other part of the business, and I get to touch things and break them. And it's, you know, it's a it's great fun.
• 4:19
  Bryan Cantrill: Well, I do think that, like, part of the reason it feels like you've been here for five years is because it ends is an extremely central role. I mean, you end up interacting with lots of different people doing lots of different things because it is I mean, you are you and Paulina and others that are working with you are the ones that are are making sure that we're delivering the right thing. And that's a lot there's a lot of work involved in making those. Let's just say it doesn't come out of the box the right thing. Let's just say that there are some things that need to be done to make
• 4:48
  it the right thing.
• 4:49
  Steve Karam: Absolutely.
• 4:52
  Bryan Cantrill: So but I wanna like segue to this this kind of extraordinary thing that that you built. And I mean, the or what's the origin story of this? Because I actually think I the you know, I think I was not in the room when you initially demoed this. Like, you demoed this maybe the controller meet up initially, but could you walk us through the the origin story for for what became Voxel?
• 5:16
  Steve Karam: Yeah. First of all, I I I am gonna do the the customary, and this isn't just lip service. I didn't invent this whatsoever. Right? You know, I know there's always the building on the shoulders of giant story, but, you know, we one, Voxel has a very strong spiritual successor.
• 5:33
  Anyone who has followed along with Omicron, you know, there is a simulated version of Omicron that you can run. Omicron is extraordinary, and so are all of the components like propolis and crucible. It's extraordinarily easy to set up on, you know, even commodity architecture. We had internally this ability to pull all of that together through one of our open source repos called Falcon, which basically builds lab infrastructure on, Propolis.
• 6:06
  Bryan Cantrill: Huge shout out to Rye Goodfellow on on Falcon. Yeah. It's just extraordinary work building.
• 6:13
  Steve Karam: Yeah. Well, and on top of that, right, there is a truly, truly unemmulatable I never wanna say unemmulatable. The whole the whole title here is how folk can you go. But a very difficult to emulate part of our architecture, which is the, you know, our Tofino switches. Yes.
• 6:29
  And so the introduction of soft NPU and being able not just to simulate, but actually fairly well emulate the behavior of those switches is extraordinary. And it it's really the only reason we can have virtual architecture at all is because we have that switching fabric, between the, you know, the rack fabric of Falcon and the switching fabric in Tofino. We have the ability to, you know, bring up gimlets and sidecars and all of that.
• 6:57
  Bryan Cantrill: And this is soft NPU was very clutch for us. So this is a allowed us to get a an analog that allowed us to to program the switch as if we were programming the switch, but of course, it's all virtual. And that led to our own part of SoftMPU was Yeah. Work that Rye did on our own p four compiler, which is really important. So away from the Intel Tofino architecture, we kind of built all of our own apparatus, Rye had anyway.
• 7:26
  Steve Karam: And so, you know, one of the problems, product assurance, like you said, it was kind of a newer concept of a role. And one of the issues with any kind of testing is, okay, where's our test environments? And anyone who's been in any form of software development or IT knows that test environments, usually they play, you know, not second fiddle, but like twenty seventh fiddle. Right. You know, just just just ahead of education and enablement environments and documentation.
• 7:53
  Right? But, you know, it's it's especially difficult when your product is a freaking rack. You know? You've got this this whole thing with all of these pieces. We've got the rack itself, the fabric, the sleds, the switch zones, the SPs, the route of trial, you know you know the drill.
• 8:10
  And, we have different levels of fidelity, which is a word, you know, I don't want to, overdo the word, but it is a very important word when it comes to emulation and simulation is fidelity. And we have various levels we can test on. We've got, you know, we're we're big on dog food, and you know that. Right? You know, we got our we actually got the rack called dog food.
• 8:32
  We've got these racklets, which are mini racks, that are are fairly faithful. And then beyond that, we have single instance Omicron, the simulated one that you can run, or physical Omicron running on top of a Helios box. But what we needed was a very stable, very modifiable level of of emulation that sits under a racklet. It's easier than a racklet, but higher fidelity than one box.
• 9:03
  Bryan Cantrill: That's right. And so just to give people because people may not be aware of I don't know we I don't Have we talked about racklets here? I'm not sure we have.
• 9:10
  Adam Leventhal: I don't think we have.
• 9:11
  Bryan Cantrill: Yeah. So a racklet is and for those of you wanting Oxide to build a smaller rack form factor, if you were to see a racklet, you'd be like, well, goddamn it. This is it. Just ship it. Like, what do you what do you you already have built it.
• 9:25
  Why are you and I a racklet and Eric Austin on on our double e team, hardware team built a what is effectively a a small rack form factor that has, two switches. So you still have the two Tofinos. You've got a total of four sleds, and then sitting around a power shelf, that has, exactly generally one PSU in the not exactly configured for redundant power. So there's the the the power shelf usually has one PSU in it. But that is a racket, which is kind of the minimum true physical rack.
• 10:02
  And they've been great, but the problem is there's just not that many of them. There can't be that many of them in part because we we don't wanna mean, the the actual hardware involved, especially the sidecars, the switches, It's a lot of hardware. So we would I we just don't have a ton to go around. Sorry, Steve. Just wanna give
• 10:22
  Adam Leventhal: Just to just to be a buzz kill for anyone who says, you know, oh, two switches and four slides is exactly what I need. It makes no economic sense. Right?
• 10:30
  Bryan Cantrill: Have Makes no economic sense.
• 10:31
  Adam Leventhal: We have these two switches which are ready to power 32 sleds, and you're gonna pay for those. And so it it it really is economically not a product, but it's very useful for us for testing. But everyone stops salivating if it feels like that's what you want.
• 10:46
  Bryan Cantrill: That's right. That's right. You don't actually don't want this. Yeah. So I know I know everyone says, know, like, hey, the home labs.
• 10:52
  You don't want this one in the home lab. This is this is a little this is aggressive for the home lab.
• 10:57
  Adam Leventhal: Yeah. A little bit.
• 10:59
  Steve Karam: As our fridge can testify.
• 11:01
  Bryan Cantrill: That's right.
• 11:01
  Steve Karam: So, yeah, just Bryan, you know, kinda to wrap up the the history behind this is, so we have this way of doing it. We we it was called a four by two. Really an internal only just because of all of the pieces it had to touch. And a four by two allowed us to, through a series of lots of shell scripts, which is how all great projects start. I'm I'm I'm convinced of that.
• 11:26
  I'm sorry. But and look. I'm a recovering Oracle DBA, and I admit that, and I'm sorry. But but, you know, shell scripts are the beginning of every great project. So but A four by two allowed us to define, what the rack what the rack was supposed to look like, and then at launch time, used a combination of cargo bays and shared drives and network and and simulated networking to bring up that virtual rack.
• 11:56
  And it honestly works great. It allowed us to bring up racks that were the right version and, you know, test a number of things. When we introduced IPv6 and unnumbered BGP, you know, Ryan Goodfellow, again, was working on those A four by twos and testing it there first because you can simulate that switch hardware, right? One of the problems with it, though, is as we've probably all seen in some way or another, trying to do everything at launch time is a recipe for disaster. It makes launches take a long time.
• 12:37
  It's hard to self heal. If something breaks, you know know damn well it's gonna break in the last, like, two minutes of the forty minute process. And so, you know, we really Ryan actually said, you know, we we need to look at something new. We need to look at something that we can build the right way from the ground up. He came up with the name Voxel for it, which we'll talk a little more about, but it's, you know, Virtual Oxide Emulation Labs.
• 13:05
  Very cool name. I like it. And, yeah, that's that's that's kinda what led to now.
• 13:13
  Bryan Cantrill: And then so what is the difference between Voxel as envisioned and kinda a four by two?
• 13:19
  Steve Karam: Well, I may have played a small part in making the vision a little more than it was was gonna be. Yeah. So Voxel, one of the biggest benefits of how it works is it's two separate steps for build and deploy. Right? So, we have you can plug in an Omicron commit, and it will clone the repo.
• 13:41
  It'll compile it. It'll build one sled image that's perfectly formed and ready to go. And it will snapshot that image so that it can be reused to, be be each of the sleds and the gimlets. Basically, it lets you build the perfect box that will then be used to play all the parts. Right.
• 14:06
  The other big big difference there, though, is all of this is in Rust. A big first step was getting away from the shell scripts as much as possible, bringing it all into Rust. And, in addition to that, you know, if you have one snapshot that needs to play six different roles, you need something to tell it what to do. And so rather than a bunch of over the wire instruction, we we have Voxel in it, which is just a crate that lives inside a Voxel, and it's a simple rust agent that gets put onto each of those sleds as they're launched and tells them you're a switch. You are sled one, sled two, etcetera.
• 14:50
  Those two things were, like, a huge a huge part of just getting it to work. Right? Just getting it to come up really required the control plane to get deployed, Falcon to do its thing thing, and an an initialization agent that says, this is the role you're playing. Don't forget to be a scrimlet. Right?
• 15:11
  Or don't forget to be a a sled three.
• 15:14
  Bryan Cantrill: And then when you when it these things that are effectively emulated sleds Yeah. And then they so they are running there are containers that are running in those emulated sleds as part of the control plane. I how did how do are are those affected in the actual emulation?
• 15:31
  Steve Karam: Well and and this it it gets real interesting here. First of all, Falcon is using Propolis, and Propolis is just I mean, not to toot our own horn or, you know, Ixie's horn or whoever you know, everyone who who contributes, but Propolis is a fantastic, you know, VM. Sorry. I'm losing my words here. It's a fantastic VM software.
• 15:53
  Right?
• 15:55
  Bryan Cantrill: Hyper hypervisor is I think the word you're looking for, but you know what?
• 15:57
  Steve Karam: Maybe maybe
• 15:58
  Bryan Cantrill: invent terms right here all the time. It's like product assurance. So it's a I
• 16:01
  Steve Karam: don't know.
• 16:01
  Bryan Cantrill: It's a it's a VM assurance program. You know? That's a that
• 16:06
  Steve Karam: There you go. So so, yeah, it that's already fantastic. There there's one other the really, tenet of Voxel that we're trying as much as possible to keep, which is don't modify the main software. It was actually really funny because when we were first talking about, multi rack, which is something that we'll talk about more, I'm sure. But when we were first talking about it, I put in all kinds of hacks into Voxel to make Voxel and Omicron act like it was already capable.
• 16:39
  And in our internal you know, basically, just here. Here's a here's a port that doesn't actually exist in Omicron, but it magically exists in Voxel. Right? Stuff like that because it's all software coded. And it was in one of our channels, you know, one of our company channels where I think it was Robert, actually, who said, you know, I don't feel like we should be creating things that aren't actually in Omicron.
• 17:05
  I was like, that's probably a good point. You know?
• 17:08
  Bryan Cantrill: That's a good rule. Yeah. Yeah.
• 17:09
  Steve Karam: Yeah. That's cart before a horse, and that's a that's something you can a trap you can fall into with emulation. You're you're rewriting the the foundation. So it's like, yeah. Why not?
• 17:19
  You know?
• 17:20
  Bryan Cantrill: Well, this does get into that fidelity point. And maybe it's worth calling out a little bit the the difference between emulation and simulation. Yes. I mean, because I think it it I mean, well, first of all, these are all lies. Right?
• 17:34
  Where the this is all, like, scope of lying. And where does the lying end and the truth begin? But how would you delineate the difference between simulation and emulation, Steve?
• 17:45
  Steve Karam: So that is a wonderful way to phrase it, first of all. I like that. We so I'll give you a great example, and this is something that I'm really excited about with Voxel is, we have this thing called SP SIM. SP SIM is a very lightweight set of stubs. It it it responds as a as one of our SPs should, one of our service processors.
• 18:09
  Right? It is it responds as it should to a sled. And so in our virtual environments, we can use SP SIM to play that part so that we can do things like initialization of the virtual rack and bringing it online. But it's obviously not a real SP. It's it you know, it's not a it's not a microcontroller.
• 18:32
  And so that is a to me, is a great example of simulation. It is a a program that can run and holds an input open to take requests and return mostly faithful responses. Generally speaking, one of the downfalls of simulation is when you use it, you return expected responses. Simulations are a lot of times happy path, and that can be a problem from a testing perspective because it's literally an input and output that returns what what it it's meant to return what's expected. Right?
• 19:09
  Yeah. In Voxel, we have that, obviously. It was available to us, and it was responsible for some of the first racks. And I apologize to any AI dislikers on the call, but this is gonna be the first mention of LFMs you hear on the call. Well, no.
• 19:25
  It's not the first. Bryan started it in a way. So Yeah.
• 19:27
  Bryan Cantrill: Yeah. I started with property disputes, so we're we're fine.
• 19:30
  Steve Karam: There you go. So I basically gave I I gave a weekend task. I was working very heavily on a part of Voxel Bring Up, and I was like, I just wonder what's out there. I had heard of, like, Unicorn, I think it's called, for, microcontroller emulation. And I, you know, I know it.
• 19:48
  I think it runs on dot net if I'm not mistaken. I'd I'd have to relook. But, you know, I was wondering what is out there for emulating these microcontrollers, and especially what's out there in, like, something like Rust where we can trust it and we can use it anywhere. And the answer was a big fat from what I could see, nothing. And so just for funsies, I gave a weekend task to Claude, and I said, hey, can you create an STM32 H753 microcontroller emulator?
• 20:18
  You know? There were some things that, you know, I gave it, for example, the way that our SPs work, where we have the a and b areas for flash, and we've got all these different components. We've got a root of trust, which I wasn't, you know, an LPC 55 board, which I wasn't even ready to even think about yet. But if you think of all the pieces that make up our service processors, it's a tiny little chip on the sled. But what it does, you know, it has a lot of different protocols and memory areas and everything else that are very important.
• 20:50
  And, I was shocked that Claude actually did a decent job.
• 20:54
  Bryan Cantrill: So mean, I would just assume that if you were to tell me, like, hey. I'm gonna do this. I'm like, no chance. It just it's Waste of time. Right?
• 21:01
  Yeah. Waste of time, waste of tokens, you know, whatever. It's the I just feels like or I I mean, in all honesty, it'd be like, interesting experiment. I don't think it'll yield much because it there's it's just so there's there's so much to go do Yeah. And so much to have to go emulate.
• 21:17
  Steve Karam: So so following that experiment, it didn't work right off the bat. I mean, it started up. The process ran, which was cool, but it couldn't actually really do much of anything. The good news is is after just honestly, it only took probably two or three hours of tweaking after that, and I was able to flash a bare minimum hubris archive to it and actually get you yeah. It was like, woah, the machine.
• 21:44
  Right? And to be clear, this is not silicon. So it's like many times slower, and not nearly as faithful. But the fact of the matter was using hubris, using, you know, humility, I was able to flash this firmware to this emulation, thread, which was really freaking cool. I'm like, well, cool.
• 22:04
  Maybe there's a chance. Yeah.
• 22:07
  Bryan Cantrill: And did and you just put a bare bones hubris on image on there that effectively did did did But but you were able then so you were able to run humility against that and see, like, a a colonel had started and there was a task?
• 22:20
  Steve Karam: Not right away because of some changes that had recently taken place in hubris. Okay. Fair. Fairly soon after. So you know?
• 22:27
  And I had played around with Hubris a bit before. I'm not, you know, deeply involved with what the team is doing, but I this is where the home lab part comes in. I actually have a Nucleo STM 32 h seven fifty three z I board running hubris that serves as the service processor for my home NAS. So I can do console, I p v six connect, take, you know, all of that through it. Just you know, it's wired into I've got IPCC communications going.
• 22:58
  It's wired in just like it should be. And I use that for my home net my home system, actually.
• 23:05
  Bryan Cantrill: So says you can't home lab with Oxide. You could you can home lab with Oxide. You just, yeah, exactly. They just gotta get get a little nuclear eval board, like some DuPont wires, and you're, you're in business.
• 23:14
  Steve Karam: Oh, and you should see it. I mean, the du the the DuPont wires are everywhere because I've got these, like, really janky, you know, sensors from Amazon, and, like, all of them are solder soldered on, you know, like, crooked and stuff like that. You know, some my fault, some their fault, just bad pins. You know?
• 23:31
  Bryan Cantrill: So Steve, right now, Adam is writing down in his notebook image for episode. Right. At minute 24 or whatever. This is the I got you covered. Exactly.
• 23:42
  We got you covered on the jank.
• 23:44
  Steve Karam: So further on emulation. So, you know, that was cool, and that was a good start, but it takes a lot. Right? So basically pushed it forward. And the the key really is if you think about how Oxide initializes, Oxide is very, very pull heavy, meaning it's constant you've got services that they they can afford to that are polling for updates on a near constant basis.
• 24:12
  And when those updates come in, they know what to do, and they move forward. I'm sure, you know, someone like Adam could talk way more about that than I could. Right? But from an emulated service processor perspective, that sucks. Because to be in constant state of waiting, you need to keep a thread constantly spinning, and you are going to absolutely kill any fake performance your emulation does.
• 24:38
  Bryan Cantrill: Mhmm. Yeah.
• 24:39
  Steve Karam: On top of that, if you're gonna emulate a real SP, it needs to have the QSPI host boot flash. It needs to have SP rot to talk to the root of trust. It needs to have, you know, ignition regs. It I mean, there's so much that an SP needs to be able to have instructions instructions for. For.
• 25:01
  And so, basically, what I was doing was spin up a Voxel lab, find out why RSS would not work, you know, initialization would not work. When it failed, find out that, you know, it was hanging on being able to record a temperature measurement or whatever else. And so, you know, not just initialization, but other little parts. And it was just going basically one by one and modeling the bare minimum set of instructions and the bare minimum set of sensors and things like that that it required. But the point is they're actually there.
• 25:39
  They're running hubris. And so and and they've gotten even better. Ben Stoltz, who's listening in right now, he's on our hubris team. He came in and added, you know, Boodleby, which is, you know, our our our root of trust secure boot. Right?
• 25:55
  And other amazing features like, you know, Glasgow Bridge to be able to use humility to to trace this thing. And it's gotten so cool because I really do think it's very cool. It's gotten so good that it actually has, like, the the the root of trust caboose. It's got dice identity. It's got digest and attestation built in.
• 26:19
  It works with sprockets in the way we bring up our racks in a secure manner. You can flash new firmware to it. In fact, one of the things I'm testing right now is being able to run Oxide's native system update, our self-service update, and watch it flash the new firmware to these emulated boards.
• 26:41
  Bryan Cantrill: That is wild. It's wild.
• 26:43
  Steve Karam: Right? And seeing this little process and, like, so many cool little things you can do when you have it like that. Like, Eliza on the team gave me an awesome idea. Hey. Can you expose, you know, certain addresses as sockets so we could just write whatever we want to it?
• 27:00
  And that allowed me to, for example, just write to it, you know, just touch a file and boom, I can make the temperature 700 degrees, you know, or whatever else because I can expose any of those of those sensors or whatever else's socket. So that that level of flexibility and, god, I I hate it because I've used fidelity, flexibility, and and faithfulness. So it sounds like I'm I'm coming up with corporate corporate bylaws
• 27:24
  or something.
• 27:25
  Bryan Cantrill: Or or you you sound like a marriage counselor. This is actually good. You know, this is all of us get get the benefit here by yes. These are all very important.
• 27:33
  Steve Karam: So but, all of these things come together. So I'm gonna I'm gonna because I I have a tendency to, blab. So I'm gonna go back to the original question is when is it enough? To me, it's enough that right now, Voxel with SP EMU, which SP emulator. So Voxel with SP EMU, they run, full multinode or multirac labs inside of a single Helios box, and they run them using stock Omicron with no modifications, either from source or from a tough repo.
• 28:11
  They run stock hubris, and they run stock hubris images. So the sidecar and Gimlet images are ones that actually ship with the product, and no modifications are made. And to me, that is kind of the barrier of lies when they when the emulation can work with your release product. Right?
• 28:34
  Bryan Cantrill: Yeah. Well, and and it because that allows it to really be used also as a development vehicle. And maybe, Andrew, this is maybe a good opportunity to kinda get you in here because Yes. You've been leading the charge on our multi rack effort. And I mean, one of the challenges of multi rack is, I mean, it kind of says it in the tin, you kind of need multiple racks to go develop it.
• 28:58
  And again, it's not that that hardware mean, obviously, we have hardware. On one hand, we have plenty of hardware. The other hand, like, it is ultimately a scarce resource. And you wanna be able to do lots of things that require an arbitrary amount of hardware that we actually don't have.
• 29:13
  Steve Karam: One of the problems with my a multirack, and I'll just, you know, kinda to speak up to this part. Every single problem I talked about with testing is even worse with multirack because you need two of them. Right? So the whole premise behind multirack, which, you know, I won't go into explicit detail. I'll save that for Andrew or someone else.
• 29:32
  But the whole premise there is that we need multiple racks to be able to talk to each other in a manner that allows them to work together. Right? Whether that's spreading, you know, silos across them or our our different processes like crucible across them. Right? Not promising anything.
• 29:50
  That's that's my disclaimer on that. I'm just saying that's the that's the kind of concept you need for multi rack. Being able to take up two of those racklets even would be really hard. It'd be really heavy. If we're having trouble getting one racklet for testing, think of getting two.
• 30:06
  Right? Right. And now we need two that need to be wired in such a way that they can be turned into multi rack but not be committed to multi rack. And that's, again, a big part of, like, what, Ryan Goodfellow and the and the, networking team have worked on is this ability to mix and match and make modular how we do testing labs in the physical space. So we needed to be able to do that in the virtual space as well.
• 30:33
  And so Andrew basically parachuted into the Voxel project, told me my code was crap, which is awesome because it is. Look. I my only ever, like, true blue, I was a developer as the job experience was doing Python. So you can imagine what that's done to my coding practices. I learned on q base GW basic.
• 30:57
  I still
• 30:58
  Bryan Cantrill: like base oh, nice.
• 30:59
  Steve Karam: I still like Gochu. Nice. You know? And, anyway
• 31:03
  Bryan Cantrill: Oh, you know, I I I feel Adam, this is an important point. Can we bring the chime for our Dykstra tweet storm episode? Oh, heck yeah. One that with Dykstra worried about basic corrupting an entire generation. It's like and it did.
• 31:16
  And fast forward, and here we are. You're the that corrupted generation is now directing Claude to write an emulator. So suck it. Dykes, right? So it's all working.
• 31:24
  Steve Karam: Just go to line 40. And and yeah. So, you know, he came in and and it was actually really great. So not only did he come in and bring in the pieces of SLED agent and RSS that were and initialization that were needed to make Voxel work even better, he had exactly what was needed to make the multi rack part work. And that's everything from the cabling, which is all, again, virtual.
• 31:51
  You know, we've got a single you know, you know, the the code is now open source, so you can see it. We have a single rack dot r s file that that basically is all the cabling. Right? And that's where
• 32:03
  Adam Leventhal: That we
• 32:05
  Bryan Cantrill: is wild. So mean, you've got virtual cabling and so you've got, obviously we talked about the soft MPU work, and then we have built effectively a virtual scrimlet, is our adjacent, the switch adjacent compute sled. Yep. And then we so I mean, you've got enough fidelity there. Again, to be able to have like the actual run the actual software is just extraordinary.
• 32:32
  Steve Karam: I mean,
• 32:32
  Bryan Cantrill: I I
• 32:34
  Steve Karam: gets even cooler, honestly, when you're running it, because you can also do it in different ways. You can run Voxel as a three node lab. You can run it as a six node. Oh, okay.
• 32:45
  Andrew Stone: Hey, all right. That worked.
• 32:47
  Steve Karam: Andrew's here.
• 32:48
  Bryan Cantrill: Hey, we have Andrew's here.
• 32:50
  Andrew Stone: Sorry, keep going. You want, I can, you can
• 32:53
  Bryan Cantrill: probably I'll just get it spin
• 32:54
  Steve Karam: it with thought and then you can go because I've been going on. But yeah. So it's very flexible. You can run three nodes. You can run six.
• 33:02
  You can have three sleds up and initialized as your bring up a voxel, but two waiting to be added as future sleds to test plug and play. Right? You can now do, I want two sleds of five or two racks of five sleds each. All of that is customizable through a single TOML file and built, you know, defined through software, which, you know, that's really nothing new when it comes to, like, how VMs work, but it's being done for what represents an entire rack, thanks to all the work it built on, which is what makes it really cool.
• 33:40
  Bryan Cantrill: Yeah, that is amazing. So yeah, Andrew, do you wanna talk about kind of your intro to Voxel and kind of realization that like, wow, this is actually gonna be useful for me.
• 33:48
  Andrew Stone: Yeah, so I used a four by two quite a bit, submitted quite a few patches to it, mainly for work on trust quorum. Cause like the level I work on, I don't really care about VMs, the things that our customers actually want to run. But it's all about like control point software that runs on a Lumos, in the zones and kind of leads up to that and early boot path and stuff like that. And so having like, because Bracklets are sparse, having some sort of emulated or simulated, system where I can run like four individual instances of a SLED, specifically like four different SLED agents and four different global zones that requires like things, the global zone can only, you can only run one instance on a machine. So if you have like a single Helios box, you can deploy a bunch of other zones.
• 34:38
  Zones are like jails, or containers, on Linux. And, and so you can only run one global zone. So like, okay, that's, that was my intro to A four by two. I needed it So I could like manually test things that I didn't have property based tests for, or write, you know, kind of automated client driven tests, spinning up a four node cluster required for trust quorum. But then, somehow I got tagged to work on the multi rack project and I realized that, well, having a single rack was not going to work.
• 35:12
  What I really needed was like, multiple individual racks that were isolated somehow through some magic, and which we could start to build out multi rack software. And I knew nothing about how the networking bits worked, and to be clear, this is still very early on in the project, And so, I don't know, a few weeks ago, Steve demoed Voxel or whenever he demoed it, and then I actually got started building out the multi rack software and got to the point where I could deploy something similar to our Rack setup service, which was our MultiRack joint service, and that's kind of analog to that, and what that needs to do is essentially spawn Trust Quorum on a new sled. So we have one sled that's been initialized and it's running all our Oxide software, then we have another sled, or another, rack that is sitting there idle, and we don't want to run RSS because that'll set up like two split brain disjoint systems. What we want is to start just enough software on the new sled and configure it so that the two sleds can participate on the same underlying network. And so I needed a mechanism to be able to do that because, A, we don't even have two racklets connected to each other in a lab right now that's being worked on, And so we needed some software mechanism for me to make progress on the beginning of this on the beginning of this multi rack project.
• 36:42
  And so Steve had And a so I started using it.
• 36:45
  Bryan Cantrill: Yeah, and then Adam, you wanna provide some, because I think it's some helpful context in terms of where we are and aren't. So we've got a rack scale machine, which is great, but then if one wants to deploy multiple racks, it's kind of reasonable to want those to act coherently as if they are a single larger entity. And that's that's what this multi rack work is all about. Did you want to provide any additional? I mean,
• 37:10
  Adam Leventhal: No. No. No. I I could just imagine people thinking that when we talk about a cloud that implies that that already existed.
• 37:18
  Bryan Cantrill: Yes. Well, you know, a part a a large part of our endeavor right now is building the product that people think we already have. So that that is and so, Andrew, the the the networking components in particular are I mean, because this is a a a good example of how this is like a gnarly problem without using any emulation. Because there is this kind of like a cabling issue, like in order to have multiple I mean, when you've got a rack scale machine, it's got you obviously have the two the the two switches there. That's fine.
• 37:50
  But you actually wanna have this thing coherent as, like, as multiple racks. They actually need to be connected to one another somehow. And that networking is extremely important. Do you wanna describe what our vision is for that and how we emulate that in Voxel?
• 38:08
  Andrew Stone: Yeah, I'll do my best. And I think I'll lead by saying that there's really two parts of that. And so the thing we're trying to get now, get working now is kind of the minimum viable connectivity so that the control point team can go and finish in parallel the multi rack work while the rest of the higher level networking stack is built out that's required for multi rack. And so we really have two routing protocols that we deal with at Oxide. One is DDM, that's the Domain Driven Multipath, I think is what the acronym stands for.
• 38:41
  That is our internal, routing protocol. So, that runs across the backplane, so the rear ports of the Oxide switch. So we have each of our switches as a transit router on it, each scrimlet rather attached to each switch has a transit router on it, and then each global zone in each sled has its own router, and they exchange routes, for, both overlay and bootstrap traffic, And so, the bootstrap traffic, essentially we have this internal routing protocol and prefixes for our bootstrap network, which is not leave the rack allows us to get sleds communicating amongst themselves so that they can participate in trust quorum and decrypt things such as the rest of the networks, such as the database that hosts the control point data and allows us to bring up the reveal enough of the underlying configuration so we can start the underlying network. And then those prefixes, each SLED and other thing, like, have their own slash 64 on an IPv6 network, and so those get broadcast around, and so now we have the problem of, like, we have these individual sleds in a rack have, underlay addresses, and we need to be able to get, sleds on another rack to have underlay addresses that are reachable, and so there's two ways we could do that.
• 40:12
  We can use DDM, and that implies some sort of layer two switching where like the racks are on the same subnet, via switch, or they're like literally cables, a single like QSFP cable connected between two front ports of two switches, like a switch on each rack. And so that's what we built out in Voxel. On top of that, we'll be building BGP with some tunneling stuff. It's kind of beyond my full comprehension, like how the internals of that work. And so I don't want to butcher it.
• 40:46
  And I think it's more important to talk about what we're doing like right now with Voxel. And so that is DDM.
• 40:53
  Bryan Cantrill: And so just to kind of restate what you said. So what this is doing virtually is something that we've always kind of had the idea about. I mean, this goes back, Adam, to when kind of we were very first envisioning sidecar. And because we have the, we've got all the rear ports for the rack and then we've got the front ports for the uplink, but we've got like way more front ports than we practically need. And one of the early observations was like, well, hey, when we wanna go do multi rack work, you could actually envision these front ports being directly connected where you can connect one rack to another via the front ports, which sounds like, okay, that's a great idea.
• 41:34
  But now Andrew, actually requires us to like, there's a bunch of software that needs to change to allow the disposition of one of those front ports to not be an uplink. That doesn't just happen automatically. Surprisingly,
• 41:47
  Andrew Stone: it does not happen automatically. And surprisingly, was actually less software than I thought it was going to be, which was really nice. So, Trey, one of our, network engineers, networking team engineers, who works a lot on BGP code and is now in kind of in everything. I've been working closely with him on multi rack and he basically told me like what needs to happen is we need, so there was actually essentially, like there's a hard coding for DDM that allows it to only run on the rear ports. And so we needed to enable like each port has its own DDM state machine that manages the traffic, and so we needed to start a state machine for whichever front ports we needed.
• 42:30
  And then we also need to say, hey, these front ports no longer serve uplink traffic, so they're not going to be like participating in BGP and there won't be any conflicting, any confusion among other software. So, we basically
• 42:43
  Steve Karam: just Yes, have
• 42:44
  Bryan Cantrill: go ahead. You're basically telling the switch software, hey, this front port, I want you to treat this front port like it's another rear port effectively. Is that Yeah,
• 42:55
  Andrew Stone: that's a fair way to put it. But first, what did need to change and what was kind of tedious was the getting the fact that we want specific front ports to turn on down to the switch port. That turned out to be much more code than just the code to tell the switch, to do the right thing.
• 43:11
  Bryan Cantrill: Oh, interesting.
• 43:12
  Andrew Stone: So there's a lot of layers to the control plane. So we had to change, Wicket, like add a message to the commissioning API, which then changed the RAC network config, and then we had to build the RAC join service, so we could actually, like, have a service that handles the request with the RAC network config, and then we had to add a field to that, to each specific port config for the front fields that say allow DDM traffic, and then we had to send that down, from the new service to the Bootstrap agent over the Bootstrap lockstep service, and then that had to be, inserted Well, actually it gets sent there by the joint service, inserting it in the Boot Store, and then it's reconciled by the scrimulet reconcilers that only run on the scrimulets. And then the scrimulets go ahead and the reconcilers go ahead and talk to the DDM client and say, Hey, I want you to run this front port. And then that does the magic of essentially like toggling it and saying, Hey, you're not on Upwork anymore and you need to enable IP six. And then it also talks to the PDM admin client and says, now start a state machine for this port.
• 44:18
  And so you have this whole one from user over the tech port to get down to the switch. That's like, and then yeah, go ahead, Bryan.
• 44:27
  Bryan Cantrill: That is and and none of that, correct, is in Voxel. That is all in the software that needs to run on top of the the racks.
• 44:36
  Andrew Stone: Correct. That is all production software. And so where Voxel comes in is at the top and at the bottom. And so, we ended up demoing some of those on Friday and, and initially my idea was like, Hey, like some of the software just hadn't merged through yet. And Voxel, like, like Steve had created, as he mentioned earlier, like this magical pathway where like QSFP two and QSFP three were automatically like connected in what he called interconnect ports in Voxel.
• 45:05
  And so it's essentially like shelling out to to Illumos to say, like, create these ports, right? And then doing the manual things with SWAT on with the command line switches to like touch the switch ports, but like Dendrite can do all that. The problem is like, was contacting Dendrite to do it. And so like Trey and I wrote software to do that. And then, so what I ended up having to do was basically delete that code in Voxel.
• 45:29
  Voxel still generates the config for those QSFP2 and QSFP3 front ports, and now it puts allowed DDM traffic, that one boolean that I added, in the right spot, and then it also now has access to a new client API that allows us to send a multi rack join request, which contains the information about which ports to turn on. And so now Voxel is actually using the real progenitor clients to send requests down through Wicket, the new Wicket commissioning API to to turn on those ports, and then that makes its way all through the real software that I just talked about. And then once it lands down in Dendrite, Dendrite is actually using SoftMPU, to configure these ports in the emulated, switch software. So the real the dendrites doing the real code and just at this bottom layer, there's this emulated, switch hardware going on. And then even the megamite stuff to start the start the, state machines, that's all real code as well.
• 46:30
  So, like, we really are just filling in at the top and bottom with Voxel and, like, in Dendrite. And then yeah. Like, you have this, like, you have this, orchestration engine for deploying these racks with, a very simple Voxel config, and everything comes up, and then you kinda use what Voxel provides in configuration and run it through the RealOxet software. And then what Steve is doing and starts with the SP SIM, or SP EMU is to do a similar thing that we have with SoftMPU before the SPs. And so we plug in all the simulation at the bottom layer, but like all the control point software is supposed to be, like, it's gotta be real.
• 47:07
  Like, without testing, without like actually using the real dendrite software and like passing through the right stuff, like, you don't have a demo, right? You're just kinda lying.
• 47:19
  Bryan Cantrill: And it is mind blowing to me that we have developed all that we, that you've been able to do all of that development, which is like right at this hardware software interface, a lot of this stuff. And the fact that you've been able to do it on Voxel, which is Steve, how many weeks old is Voxel? I mean, it's like No, it's like Like 12?
• 47:40
  Steve Karam: No, yeah, well, maybe 12. Yeah. No. We, you know, a four by two was around for a very long time. And then Voxel, you know, started as, hey, collect these shell scripts in a different way, and and run them differently, and then it turned into Rust and all of that.
• 47:57
  But, yeah, probably around, what, late May, June is when it got started. And it was just it was really funny because it was just one branch of an empty Voxel repo for a while called Voxel Proto. And everyone's like, I I think this should just be main at this point, you know? Because it was that was the only real thing that was there with, I think, like, 70 something commits on it. So
• 48:20
  Bryan Cantrill: yeah. But I mean, Steve, I mean, you you've gotta be I mean, to me, this is just mind blowing that we've gotten all of this to work with enough fidelity. I mean, Andrew, it feels very like all of the work that you had to do, and I got Rain up on stage to talk about a little bit about the commissioning API because I think there's an interesting story there. But all of that work, it's like, we got like a lot of confidence that none of that work feels like, okay, this is work that we've had to do to accommodate Voxel or that this is work that, I mean, it feels like we've got, we were very true to what is needed on actual hardware there.
• 48:56
  Steve Karam: I'm gonna say it this way. My my PR reviewers, Andrew included, are very strict on me about that point, Bryan. Is, you know, let's be true to you know, as true as possible anyways.
• 49:09
  Andrew Stone: Yeah, it is, it's super important. Like, personally have limited time on earth and I don't know about the rest of you, but I do not like wasting your time writing software that's just immediately going to be thrown away or that's like counterproductive to what we're trying to get done. Sometimes like that happens, sometimes you have to do it, but like for the most part, we want to proceed in semi linear manner as much as we can, I think? And so, it's like having Voxel there and trusting that it's going to do the right thing, has been great. And so I, can't say enough good things.
• 49:45
  Like I did, so this was also my first introduction to like working in dendrite and it was actually pretty straightforward and that I had a lot of fear there that there was going to be a lot of complexity around managing the switch, but it turns out it's just like another progenitor API. And then like, there's some underlying low level software that turns things on. Right. And like, yeah, had to pump through some configuration and look through some of the code, but it was all pretty straightforward. And at the bottom layer, it's all abstracted out with SoftMPU or the real switch.
• 50:16
  So, yeah, having that layer means that you're writing the real code and you're insulated from a lot of complexity by the abstraction and like doing things twice like that, having the real thing and the emulated thing means that hopefully you can build a better abstraction so that when you actually have to write that code, when somebody new has to come in and write that code, it's somewhat easy to do.
• 50:40
  Bryan Cantrill: Yeah, totally. And I mean, makes it now anyone can run multi rack on in, I mean, if you've got a Helios machine, you can run Voxel on multi rack, which is just, I mean,
• 50:50
  Andrew Stone: mean, you can run a very limited thing of multi rack, which is that you can get a nice routing loop served up by DDM. Like, let's like, I want to make clear that like multi rack, you can have two racks connected via soft NPU and dendrite ports, and you can have DDM, exchanging slash 64 prefixes, which we don't even want to do. We want to change slash 50 sixes and then query for the 64s, because once we scale out to like many, many racks, we don't want to announce 32 prefixes for each rack unnecessarily. But beyond that, the control point, like we have enough now to get working on the control point software, And we have people that are starting to do that, like right now, like it has enabled us to move forward and start building out the full multi rack, software and nexus and, you know, and sled agent and so forth. But right now Yeah.
• 51:43
  Adam Leventhal: And like how much how much more work can that be? Right? I mean, that's
• 51:45
  Andrew Stone: About a couple weeks.
• 51:46
  Steve Karam: That's the easy part. Right? It was yeah.
• 51:48
  Bryan Cantrill: I mean, there's a degree to which that's actually I mean, I do think that's true. I think that a lot of this I mean, there was a lot of gnarly bits here in terms of a cold start problem. And I I think that I mean, it does feel like there's kind of a tailwind in particular to, like, you now have got something you can paralyze more easily in terms of adding more folks. But, Rain, I wonder if you might elaborate a little bit on this side quest that we kinda went past quickly on the commissioning API because this ended up being a really important discovery in terms of the need for this thing, that that you kinda jumped in and filled the gap on that's been useful in a couple of different domains. Do wanna talk about the commissioning API a little bit?
• 52:24
  Rain Paharia: Yeah. Hi. I was afraid to be here. So this is all completely impromptu. But, so let's see.
• 52:31
  So longtime fans of the podcast, will be aware of Wicket. It's come up today as well and, you know, Ring the Chime, etcetera. So Wicket is the kind of it's it's this really nice UI that you get. Like, our customers, you know, when when we wheel the rack into the network, when we plug it in, customers get to see this nice Wicket UI, and then they get to update, do a recovery based update, which we call an update, and they also get to initialize the rack via this UI. And certainly, we for the first little while, for for several years, that architecture worked really well.
• 53:11
  But we started shipping lots of racks. And so we started building this great automation on top of that. And Wicket was not previously set up for that kind of automation, and the internal APIs were unstable and it was hitting this unstable API. So when Andrew started working on the multirack stuff, which is which is what he was just talking about, we kind of realized that we would have to break this unstable API. And and so that would cause all of this other tooling to break, and this other tooling had become existentially important to the company because this is how we were able to deliver as many racks as we were.
• 53:54
  So we built a proper versioned API. So we we ended up using the same technology that we use all throughout the stack. It's just another another drop shot API, just another progenitor client. All all the stuff that we built everywhere else in the stack also gets to apply here, which I think is really, really cool. And I also look at it as kind of completing our story.
• 54:16
  Right? Like, front door, right, which is the customer centric API has three interfaces. Right? It has the GUI, which is the web console, a CLI, which which we provide, and also an API. Right?
• 54:28
  And so you can kind of program against any of these three things. Now the side door, which is what Wicket is, also has the full complement of all three APIs. Now these are not customer accessible. These are for internal use for now. But they're set up in a way that, you know, we kind of get to leverage all of this stuff.
• 54:48
  And so, so the commission API is is is the result of that work. It was kind of two weeks of really compressed work that we needed to do. It got done. It unblocked like, it kind of our racks kept getting delivered, and it also unblocked all of the work that Andrew and and Steve and all are doing. So, yeah, it's it's super cool that I I honestly, like, imagine having an API to do firmware updates throughout, like, your rack.
• 55:20
  Right? Like, this is just like an API. Right?
• 55:22
  Steve Karam: Yep.
• 55:23
  Rain Paharia: And that's what we get, right? And I think that our clean sheet of paper has enabled us to do things really in this modern, really tightly scoped manner that I think really shows the power of our product.
• 55:36
  Bryan Cantrill: Yeah, I totally agree. And thank you so much for, I mean, really hard work on that. And Adam, I mean, it must be very gratifying for you, Adam, to get the, I mean, in terms of like the work that you and Dave did on drop shot and progenitor, some of the earliest work we did at the company. At a at a time when you were kind of like, you know, we had, you know, nothing and we are kind of like, we need computers first. And you're kind of like, and Dave are off on like, seemingly putting like on a pretty distant island.
• 56:08
  The
• 56:09
  Adam Leventhal: Like, what what is the foundation of a computer if not a web framework and SDK generator?
• 56:14
  Bryan Cantrill: I ask you. That's right.
• 56:16
  Adam Leventhal: That's right. No. It's awesome to hear that, like, that has has borne fruit because because even as you alluded to it, like it was not clear that it was the right decision at the time as often these kinds of infrastructure things are not.
• 56:30
  Bryan Cantrill: Well, was kind of like, I mean, and maybe the better metaphor is that we were you know, trying to dig a tunnel through a mountain range. And you're like, well, we're gonna go to the other side of the mountain range with these spoons and, you know, we're gonna we'll get to work.
• 56:43
  Adam Leventhal: We're setting up a picnic for when when the boys finished digging through. But yeah.
• 56:48
  Bryan Cantrill: Yeah. Totally. And it was just it is really, mean, to your point that we are now at that bit where we're able to build on the bits that we built. And it does feel like so often software takes much, much longer to develop than you think it should. And Andrew, correct me where you disagree, but it just feels like somewhat recently have been Some of this stuff is happening just quickly, faster than I would have anticipated.
• 57:24
  And Steve, I think that like the Voxel's playing a really load bearing role there. And it should be said that like we part of the reason we're able to build Voxel so quickly is because this is something you've been able to really engage LLMs on. And what has been your workflow for actually developing and especially debugging Voxel?
• 57:45
  Steve Karam: So, surprisingly not, you know, Claude, go do this. I'm gonna take a nap. But, there's been so it's it's actually been very helpful. I will say that LLMs were very, very, very good at enumerating just like, you know, different registers and finding the little bits that make up a full system. Because, you know, look.
• 58:11
  Unless I guess unless you're, you know, Adam Leventhal or Bryan Cantrill, you can't hold all the pieces that make up the rack in your head at one time. Right?
• 58:20
  Bryan Cantrill: There's What I I sir, I think you are confusing me with Robert Moustache, first of all. No.
• 58:26
  Adam Leventhal: Was like, fact check,
• 58:27
  Bryan Cantrill: I definitely cannot keep all the pieces in my head.
• 58:29
  Adam Leventhal: It's been a very long time.
• 58:31
  Bryan Cantrill: Well, wanted to hear Adam say it. Exactly. I think, Steve, you're right that I mean, none of us can. The good news is that we've written a lot of this stuff down. Yeah.
• 58:42
  And this gets to the, I mean, and I feel I've said this a lot when it's not resolving property disputes. The LMs are amazing at document comprehension. If you've got stuff that's written down, whether that's in code or in RFPs, you can really leverage that to do things that would be just a really, really tough pull otherwise.
• 59:03
  Steve Karam: Well, yes. And first of all, I you know, you've said load bearing and leverage, which is making me wonder if you are in fact an LLM, but that's besides the point. Genuinely. So so that's a great point. So like your framing here.
• 59:21
  That is one area where things like RFDs and all of that really, really help because that backup context helps not fill in the blanks, which, as we all know, is one of the biggest problems. If I if I tell, you know, Claude Fable five, and I say, hey. Go scan this drop shot API. Go scan this and find all of the different areas that have to be modeled to make RSS work, just as a as a just off the off the cuff example. It's gonna do a pretty good job.
• 59:52
  If you pass in context in the form of an RFD around our vision for it, and in the form of a design doc on why we chose to do it a certain way and etcetera, etcetera, you're going to get a much crisper result. And that's that's really the area that I enjoy is assembling that, you know I mean, hell, that's that's what all of these frontier model companies are doing anyways. They're assembling way more context than they know what to do with. From my perspective, it's always been a thing about assembling the right bits of context, in a token chomper friendly way, if you will.
• 1:00:35
  Bryan Cantrill: And what so one, it rewards document intensive engineering cultures. And I mean, think this is one of the kind of the ironies is that the the the more rigorous your engineering culture is, the more rigorously you can use LLMs that the
• 1:00:52
  Steve Karam: and
• 1:00:53
  Bryan Cantrill: I I I think what it is done and you kind of get into like all of the bits that it has managed to emulate or that we managed to emulate, should say. I mean, it's exhaustive, right? I mean, would just be, and the other thing I love about Voxel is that it is both extremely important and it's also like the stakes are somewhat low at some level in that, we know that the fidelity is not gonna be completely true for everything, obviously. And there'll be things and, and I know that you've been, Andrew, your iteration with Steve has been kind of discovering those things that we need to improve in Voxel.
• 1:01:30
  Andrew Stone: It's staged for me yelling at Steve on a Saturday night, those are the stakes.
• 1:01:36
  Bryan Cantrill: Right. I mean, is kind of importantly, like this is not code that Voxel for all of how important it is itself does not run on the rack. And especially if, know, kind of going by that kind of prime directive, Andrew, that you had about not modifying the software that runs on top of it, making sure that the software that runs atop of Voxel has no awareness at some level that it's running in an emulated environment. It just presents opportunity to make our own software better when you use it that way. And so as a result, you're able to kind of do this thing that allows us to deliver software much more quickly, but in a way that where we aren't as worried about some of the pathologies that you'd be worried about, which I'm sure, I mean, and again, there will be issues where we, and I'm sure Steve, you've already found many of them where we have Yeah.
• 1:02:29
  Needed to shore
• 1:02:30
  Steve Karam: Well, I mean, think about, like, SP EMU. We're already under the problem that we're running as a thread rather than running inside of you know, on a silicon chip. So we're gonna be way slower, but the timing matters. And the amount of wedge issues I had to deal with because of overengineering on on that communication were insane. Right?
• 1:02:54
  And then there's always the, you know, the fun little things where you you spin up a a chat to try to diagnose something and and, you know, an LLM will confidently tell you that, no. This absolutely is not a problem. You can have the root of trust and SP sharing a thread and communicating that way, But you break it out and get a as I did last late last week, broke it out and got a 500 x performance improvement to where the rock racks can actually launch with both of them. Right?
• 1:03:24
  Bryan Cantrill: Oh, wow. Yeah.
• 1:03:26
  Steve Karam: Yeah. So, I mean, I had multiple iterations of back and forth and debugging. And finally, it was funny because it was Ryan Goodfella that was like, I feel like if we're letting a couple threads get in our way, we're missing the point here. You know? I like, was you know what?
• 1:03:41
  That's a that's a really good point. I'm gonna try that. And I I am gonna just go ahead and not be as frugal with my threads. It's been
• 1:03:49
  Adam Leventhal: a lot
• 1:03:49
  Bryan Cantrill: of those threads. And is and so that and that allowed you to get so initially, you had a single thread that was being used for both the SP and the ROT.
• 1:03:58
  Steve Karam: Yeah. Exactly. So you're basically emulating, you know, the the the seven fifty three. You're emulating the LPC 55. You're trying to have them communicate.
• 1:04:07
  And they let's be real. They're not very talkative all the time. Right? I mean, they they can be, but you wouldn't expect it. But during RSS, during rack bring up, it is very important that one not wait on the other or that they not deadlock.
• 1:04:24
  And so the I mean, technically speaking, you could reduce it after the fact, but that wouldn't be realistic to the product. That wouldn't keep the fidelity the real thing. So in the end, okay. Fine. You know, four node rack, that's four extra threads.
• 1:04:37
  I can deal with that. So
• 1:04:39
  Bryan Cantrill: And I think I mean, we also have the luxury of having software I mean, kind of having done this at a time when the software you're running on it is relatively robust in terms of, like, you you know, the the image that you're running is the RT. The the image you're running is the SP. Like, this is relatively robust. It's been the field of and that I mean, that allows you to to know that when you've got an issue, you can kind of look to Voxel first. We Harder have a yeah.
• 1:05:06
  I mean, an emulation issue, which which is helpful. I mean, it would have been harder to do this when the software was less mature, honestly.
• 1:05:13
  Steve Karam: Absolutely. And and, you know, even if all you can do is, hey. I just I just did this new firmware version. Let me flash it to an SP EMU process first. You know?
• 1:05:23
  If it breaks there, it's probably gonna break on the real thing too because, you know, it's even in a way, you know, more fragile. Right? But it seems to be working well with every iteration for the past few, you know, few releases that have been tested on it. You know? And if if we go back to what's real and what's not, you know, the control plane itself, the things Andrew was talking about, Omicron, Nexus, SLED agent, cockroach, etcetera, etcetera.
• 1:05:49
  Those are all unmodified real zones, real work. The host OS, Helios, is a stock image and, you know, just booted inside of a Propolis VM. The only part about Helios that's not real, in this case, it's not Helios itself, but the boot chain, is that on our real sleds, we have Picohost bootloader. You know, we have our phase one and phase two boot, whereas, you know, with the the RAM disk and all of that. So that is the one area where I'd love to see some growth, like, the future is us be able to do the two phase boot and, you know, we can get around that now.
• 1:06:26
  But, you know, as we've talked about, getting around that is is not a trap we wanna fall into too much.
• 1:06:32
  Bryan Cantrill: Yeah. So that's where you get to our lowest level of platform initialization and where, like, Propolis just is not offering so to to get into the weeds, Propolis, for example, does not have the SMN is the actual network inside of the AMD part that where you can talk to the SMU, you can talk to the service management unit. And, you know, we send messages to do things like bring out PCIe engines and so on. And we don't have that fidelity in Propolis. So you've got to kind of fast forward through all that.
• 1:07:07
  I would also say like, yeah. Yeah. Go ahead. I think you're gonna say what I was about to say. Yeah.
• 1:07:11
  Andrew Stone: Go ahead.
• 1:07:12
  Steve Karam: Was gonna say, to be fair, would we want to have that kind of fidelity? Other than for Voxel, I can't imagine, you know, really a use for us doing that anyways.
• 1:07:21
  Bryan Cantrill: Yeah. I I that's it. I think that, like, it's not really that helpful. And that's also, the kind of stuff where, like, when that when the software at that level is broken, that early platform initialization, and, again, this is our software that eliminates the need for the bias, not the the basic input output system. Yep.
• 1:07:40
  Like, the sleds don't boot. Like, we're not looking to Voxel to help us No. Be able to to to actually debug those issues. And there's not much value to being true there. I mean, you you can just kinda, like, fast forward to, let's pretend this thing boots Yeah.
• 1:07:56
  Because it does. And then and take it from there.
• 1:07:59
  Steve Karam: And what it can do is the Emulated SP can absolutely, fill in the phase one,
• 1:08:07
  Bryan Cantrill: It the
• 1:08:08
  Steve Karam: can absolutely switch from slot A to slot B. The only fake out part is going to be saying, hey, this is already on disk. The kernel, you know, the user land part is all real. It's the kernel part, boot process that's, you know, just let's just smooth through that, right? Which, you know, what's most important from a testing perspective of self-service.
• 1:08:30
  Obviously, we need to test the phase one, and we can do that on racklets. Voxel can be used for everything that comes after, right?
• 1:08:39
  Bryan Cantrill: For everything. Yeah, exactly. And then for the software, I mean, I think especially as you get to higher levels of software, you care, I mean, bluntly, you care less about that lowest level of platform initialization. I think what's one of the things that's kind of interesting about our design is that you still actually do care. You actually care more about the presence of, certainly much more about this presence of the service processor and the root of trust than you do about that lowest level of how that system was actually initialized.
• 1:09:07
  Yeah. Because that is very relevant runtime. We still need the S inmate and obviously update is in charge of flashing all these things and so on. So I think that's a very good choice to make.
• 1:09:17
  Steve Karam: A simple example that goes back to, you know, modeling what comes up at what time was, one of the troubleshooting things I ran into, like, later in the game when I started introducing the Root of Trust into it as well is, you know, SP EMU wasn't, didn't model the ICE 40s status registers for A0 and A1. Just to put it plainly, the FPGA, I didn't model those. Claude didn't think to model those or tell me about them. Right. It was just one of those things that popped up.
• 1:09:47
  And I had no idea why the gimlets, the virtual gimlets power sequencer was churning forever. So I had a pegged core for no reason and a power sequencer, a virtual power sequence. It was sir, that was just churning, and it turned out it was all because SPE mode didn't model the FPGA's a zero and a one status registers. Registers. The moment it modeled those, which is fairly simple to model something like that.
• 1:10:18
  Right? Yeah. Once it modeled those, I added that, and boom. Pegged gore goes to idle, and the power sequencer knows it's complete, and it's done.
• 1:10:26
  Bryan Cantrill: So the so in terms yes. I haven't got into how you model the power of sequencing. So you basically just it realizes like, oh, I'm in I'm in a zero. How okay. I don't know how I got here.
• 1:10:36
  Or or did it I guess you have effective because what you are modeling then so do we actually attempt to load a virtual bit stream on that and then you just chuck that bit stream away? I haven't actually ventured into this part of the emulation at all. Yeah. There is a I know Adam, you've it's been ringing a lot of chimes here, but you can go back and ring a chime. I think for our certainly for our humorous episode, if not for our Tales in the Bring Up Lab episode.
• 1:11:04
  But we've got an FPGA image that's in that hubris image that we load. We load that bit stream onto the ICE 40. And we actually do on the Gimlet anyway. And but how do you you must just chuck all that out or we just don't do that?
• 1:11:25
  Steve Karam: That's really what it no. It's no. It it is it is as faithful as possible by just being there are products or a part of it saying, am the ICE 40. It is somewhat simulated, I will say. For that point, it's not going to be forever, if I'm going to, if I'm gonna get the actual secure boot working.
• 1:11:45
  All of these pieces will have to be modeled, you know, to better fidelity. But for right now, that bit stream, it it gets used and chucked, as you mentioned.
• 1:11:55
  Bryan Cantrill: Right. Yeah. Yeah. Yeah. Interesting.
• 1:11:57
  We and we actually don't again, this is kinda like the early boot thing. Like, we actually don't need the fidelity at that level. I mean, you just kinda need to, like, just lie to the service processor and be like, yeah.
• 1:12:06
  Steve Karam: Yeah. Well, it gets
• 1:12:06
  Bryan Cantrill: a little fun midstream.
• 1:12:07
  Steve Karam: Because, like, Stoltz, who who could probably speak to that better than I could. But Ben Stoltz, he is going to be using SP EMU for a talk that he's doing on embedded systems and basically saying, alright. Everyone clode s clone SP EMU and use this to bring up, you know, an emulated board, and here's what we're gonna do with it. And so in that regard, it is, you know, I used it purely for Voxel. Ben is the one that's taking it beyond just Voxel and really adding to it.
• 1:12:40
  Bryan Cantrill: That's very cool. And I assume in doing this, I mean, you obviously have learned a lot about the components in the system.
• 1:12:47
  Steve Karam: It's been
• 1:12:48
  Bryan Cantrill: It's like wild.
• 1:12:49
  Steve Karam: Like, I'd I almost didn't know or I didn't know there were a lot of these little bits and pieces, and I gained more and more respect, like, with every single iteration and every lab I launch, which is awesome. It's also gotten me, you know, talking to a lot you know, like Andrew and I, for example, working on multi rack has been awesome. Learning a ton about proper Rust programming from Andrew. And, you know, it's just been yeah. Every single piece of Oxide has to be touched for this to work.
• 1:13:20
  Bryan Cantrill: Well, and this is why it feels like you've been here for five years because you've been this is a lot of these conversations. Well, was looking that
• 1:13:30
  Andrew Stone: thought Steve was here like much longer. Like, I was like, wait, how long have you been here? You've been here for a couple of years. It's like, no, no, no. Eight months or whenever it's been remarkable.
• 1:13:38
  Bryan Cantrill: You're like, that's not right. That can't be right. I know. It's just, it's amazing. And I mean, when you were initially tasking, you know, kind of Claude over the weekend to take a swing at this.
• 1:13:50
  I mean, Steve, you could not have envisioned that just again, twelve weeks later, this would be such an accelerant for the multirac work.
• 1:14:00
  Steve Karam: I I couldn't, and I also couldn't have envisioned that I would read so much flagrant bullshit, to be blunt. Is is, you know, all of the responses from every single new at the very beginning, it's like you completely have misrepresented what, you know, these these racks do and how they work, etcetera, etcetera. And it really took a ton of back and forth on a real racklet and showing actual output and input and logs and and then it bringing up an a four by two and saying, see, look. You know? And then as, you know, anyone who's worked with it knows, and then it compacts, and you gotta teach its, you know, dumber younger brother the same things.
• 1:14:41
  Right? And, you know, it's yeah. It was it was very interesting. I definitely didn't expect it to get to this level, to where we could be, like, over the week or, yes, last week, got it to work with our official, continuous integration, you know, our official CI tough repos that we produce with every commit in the Omicron repo, those can now be downloaded and used to build a Voxel lab as is. No modification required.
• 1:15:15
  Bryan Cantrill: That is really cool.
• 1:15:16
  Steve Karam: And that's with the Emulated SP, with the commission API, all those pieces pulled in. And then from that, over the weekend was playing with system update and basically got system update to the point our our our self-service update, flashed the firmware onto the the emulated SP and root of trust. It replaced the host OS as it saw fit, as it needed to, all of the zones, zone files, everything just like a real update would. The only thing it failed on was a lack of boot disks because this system is modeled without boot disks, which is, why I've got a couple new PRs trying to figure out the best way to do that in Omicron right now.
• 1:16:00
  Bryan Cantrill: Well, is amazing. And I think kind of a requisite plug, Steve, that we're adding to the product assurance team. You and Paulina are expanding the If this is exciting to folks, we're is a team that I think is really I mean, it is at the epicenter of everything that we do. And I I think if if the stuff that's that's exciting to you, go check out that that job description. Go listen to our our episode with Garrett Gayoros on our approach to hiring, but consider it because this is, I think that sometimes people are overly dismissive of these things.
• 1:16:38
  And at Oxide, we take this role really, really seriously. And it's very important to us.
• 1:16:44
  Steve Karam: I will say it has been a have spanned God, I've been I think I got started in IT when I was 16 as a database developer. Spanned decades of computing. I will say this role has been one of the most engaging and exciting I have ever been in. And that goes from system administration to development, to database administration, to engineering of various types. Product assurance, it really is like this perfect mix of actual, some quality assurance.
• 1:17:22
  We have our automated tests and things. We want to make them better. But we are testing hardware and software on a fully distributed system, figuring out what customers want from the rack. Not from their database. What do they need to be able to do with a table?
• 1:17:40
  It's what do they want to be able to do with the, you know, the cloud you own, as we say. I don't know if that's a bell ring or not, but whatever. Right? So we are building out incredible edge cases. And, also, just for fun, for anyone that likes this kind of stuff, one time, I took down the entire office network doing a SYN ACT flood test, and it was celebrated a little, kinda.
• 1:18:09
  So, you know I it was it was yeah.
• 1:18:11
  Bryan Cantrill: I think it was it was celebrated asterisk.
• 1:18:14
  Steve Karam: I you know, it was principle of the thing. You know? It's so but yeah. I mean, so, you know, if if you're if you're really enjoying if you're really if you really enjoy, you know, like, just digging into every piece of a piece of software and saying, why does the customer care about this, and why does this part matter? Product assurance has been amazing.
• 1:18:36
  So, yeah, join
• 1:18:37
  Bryan Cantrill: us, please. Great.
• 1:18:38
  Andrew Stone: Well, another plug.
• 1:18:41
  Bryan Cantrill: It's not
• 1:18:41
  Andrew Stone: just like, it is very much like, I don't know, like, I think we were talking earlier about like QA, like being kind of very down the list of things that people care about in their software. This is very much an engineering position. Like Steve is working with engineers all day, asking them what they need. And like, we all jumped on this when we saw how important it was. There's, there's like multiple people that hold this question about not having a main branch, but it came up because like five different people had already contributed to it in one week after Steve did the demo.
• 1:19:13
  And then I was like, well, like, why are we all opening PRs against Steve's like hidden branch? And so, and like, it's like, that's rapidly, like people are gonna, if your stuff is good and it's gonna help us make more robust, rigorous software, we're going to jump on it. And it's fun to work across team like that. And so it's just great. And another thing that I want to give a plug for BoxL in particular is that like I've talked about, like I kind of led to say like, yeah, like it adds a little more over A four by two, but one thing that it does really well, which I think a big part is getting rid of the scripting languages and like, besides whole, create an image and deploy that image, which makes it usable for people that don't want to do the builds themselves, but what it really gets is it's very ergonomic to use from an engineering standpoint, right?
• 1:20:05
  We have internal manufacturing software and Steve like took the initiative to make the commands mimic that. And so like your fingers already know how to talk Voxel if you've been using Pilot to log into our Racklets. And so it's very easy to transition. Like, a four by two had somewhat more bespoke things, and it was just more limited. Right?
• 1:20:24
  Like, it didn't have the resources behind it. Like, as you can tell, Steve is putting in a ton of time on this, and that means that, like, it's very developer friendly, and it's very easy to add for us because it's just in Rust. And so, like, it is, you are building like a production quality product used by Oxide to build more production quality software. It's not a step down from everything else we're doing, right? Think people have talked about support engineers, right?
• 1:20:52
  Like you're going to pay support the same amount as engineering. And yeah, I mean, the same goes for product assurance and everything else. And it's a good team to be on.
• 1:21:04
  Bryan Cantrill: Yeah, absolutely. And it's been, I think also Steve, the kind of value of this is, I mean, obviously our software has been open kind of from the beginning, but it can be hard as an outsider to contribute to Omicron. And this makes it a lot easier. This makes it easier for folks to actually go get their head around the entire system by kind of running it in front of them in a in a Helios VM.
• 1:21:27
  Steve Karam: So And on that note, you know, I so I was lucky enough to, when I joined Oxide, it was before I mean, we were still in the midst of the whole DRAM crisis, but it hadn't, you know, gone to critical levels yet. And I was able to get one of those. It's a minis forum a two mini PC. Right? It's got 32 threads.
• 1:21:48
  It's got 96 gigs of RAM, and I threw a faster NVMe in it. And so 96 gigs of RAM, 32 threads, I can easily run six to eight node racks inside of that or multi rack. Right? So I can spin up three or four nodes racks and, you know, two of those and run that all inside of this little Helios setup, you know, I've got on a minis forum sitting in my closet. It's it's pretty wild.
• 1:22:22
  It's cool being able to do that. My one of my next goals for Voxel is something that Ryan Goodfellow and I have talked about, which is being able to run it on things like our bench gimlets. Right? Run it on a the full might and majesty of a gimlet that's sitting in the office, for our internal use, it would be fantastic. And, yes, roly poly, I have I have three minis forms of various types.
• 1:22:48
  Bryan Cantrill: That is awesome. Could you actually, maybe in the chat, you can drop the the the kind of the specifics of the of the minis forum that you're using. Only
• 1:22:57
  Steve Karam: side of it is it has no, it has no, serial. So it's not good for running as is, but it's great for running Voxel because you don't need the serial for that as much. Right? It's got serial built in thanks to Falcon.
• 1:23:10
  Bryan Cantrill: Yeah. Right. That's awesome. That is great. Steve, this is really exciting stuff.
• 1:23:19
  Andrew, it was really great to watch your demo on Friday, kind of remarkable about how just kind of quickly all this has come together. And now we're able to really get into kind of the meat of that multi rack work. And rain really great to see the commissioning API be a load bearing in that really kind of accelerating that. And then Adam progenitor playing the progenitor playing it's and drop shot playing me
• 1:23:44
  Adam Leventhal: the On the scene, yeah.
• 1:23:45
  Bryan Cantrill: It takes a village. And so that spoon that you started with on the far side of the mountain range so long ago has yielded a glorious tunnel. Is amazing.
• 1:23:57
  Steve Karam: Yeah. It was actually pretty fun when there was one point where I Andrew was reviewing one of my PRs, and he's like, dude, why are you doing all this? Just use the API that's out there. And I was like, you know, there's an API for that? And he's like, there's an API for everything.
• 1:24:10
  You know? So that that really, really makes developing anything that layers over this super easy.
• 1:24:18
  Bryan Cantrill: Yes. It makes it much, much easier. That is awesome. Well, again, great stuff, and really exciting. Steve, it's been a great first five years working with you.
• 1:24:29
  I'm looking forward to the next five here coming up in the next couple of months.
• 1:24:33
  Steve Karam: All for it. All for it. Yeah. I mean, know, LLMs only came out what, like a month or two ago, I think.
• 1:24:39
  Bryan Cantrill: Exactly. Right. No. It's just it it is it is really amazing work.
• 1:24:43
  Steve Karam: Awesome.
• 1:24:44
  Bryan Cantrill: Awesome. Alright. Thanks everyone. See you next time.