Latest / Oxide and Friends / Holistic Engineering with Robert Mustacchi

Transcript

Holistic Engineering with Robert Mustacchi · Oxide and Friends

• 0:00
  Bryan Cantrill: Think I've invited Robert to speak.
• 0:02
  Adam Leventhal: And Craig and Giarc have dinged. Now that I've given them I've I've succumb to their their demands.
• 0:10
  Bryan Cantrill: Listen to podcast parts when Craig and GRX says it starts. I do I also do love this is where you get a little bit of a Tomac and Zayman thing and that they, like you would think you might have some redundancy with the twins here, but they actually apparently think with with a like mind.
• 0:23
  Adam Leventhal: They have a yeah. So they shield at least a like set of permissions in Discord.
• 0:28
  Bryan Cantrill: Listen. Tomac and Zaymont had a had shared the pain. These guys share permissions. Makes more sense.
• 0:36
  Adam Leventhal: It's a little less It makes more sense. Fantastical.
• 0:38
  Bryan Cantrill: It's a little more pedestrian, a little more a little little more quotidian, but come on. I mean, this is just like, you know, we're just Yeah. Trying to be realistic over here.
• 0:45
  Adam Leventhal: There you go.
• 0:47
  Bryan Cantrill: Robert, you're here.
• 0:49
  Robert Mustacchi: Yeah. Good evening.
• 0:50
  Bryan Cantrill: Excellent. How are you? You know, the our former colleague, Alex Wilson, said, you know, it is really great to see you kicking off Robert Masstaki appreciation week this way.
• 1:01
  Robert Mustacchi: That's funny. I got a very different, sounding message from him.
• 1:06
  Bryan Cantrill: Well, I I I don't know if you quite ate the hook the way I did, but I was just like, oh, we is that a is that a thing? We do is that like an annual thing? Because we have a our colleague CJ has, kicks off and there there there is an Eric Anderson appreciation day here at Oxide for one of our employees. So, Mike, seems like a reasonable thing that we would do to have a Robert Mastocchio appreciation week. I don't know, Mike, have we have we so after like I can't quite tell if he's joking or not.
• 1:34
  So I did ask him like, are we is that is that like a thing? Have we done that? He's like, oh, yeah. It's definitely a thing. Like, we've got it's documented in rm.c, and there are it's actually an extraordinary block comment complete with diagrams.
• 1:48
  And I'm like, Okay. What? Where where's this? Like, I've clearly missed something. I'm like, have I missed a, like, annual celebration that we do about and then my mind's going to super weird places, like, have I always been, like, out for our mystaki appreciation week?
• 2:05
  Is that a source of resentment that Robert I mean, I just don't know. You know, I'm just like, and then like, where is rm.c? And so I did I looked for I went to some pretty obscure places looking for this thing and I finally, I I can't find it. And I'm looking at some by the way, I do not recommend looking some of these r m r m dot c, I mean, Robert, I guess this is the curse of having a a name that is a an old Unix command, but, hey, you know, I've got BMC over here, Basement Management Controller. I'm not it's not it's not good over here.
• 2:34
  Yeah. How have you ever been in r m dot c?
• 2:38
  Robert Mustacchi: I have a little bit. It is not a track information.
• 2:41
  Bryan Cantrill: Yeah. It is it is really not not a good looking file, but I can't find this thing. And so finally, after like a half an hour, I'm like, here. Okay. Look, I'm I'm gonna have to ask the embarrassing question.
• 2:50
  Like, where is rmdot c? He's like, oh, no, I just made all that up. I'm like he's like, but there but he said, but there should be a Robert Maslachian Week. I'm like, there okay. Alright.
• 3:02
  Alright. There we go. There we are. That way that there definitely should be. So
• 3:05
  Robert Mustacchi: Yes. I think better off about it.
• 3:09
  Bryan Cantrill: Well, Robert, we are very grateful for you joining us. And so, here's what I and actually it was funny because, you know, just earlier today, we had someone by the office who was asking about how have you been at Oxide? How have you done hardware software co design? Like, how do you how are you able to do that? How they're just like, it's organizationally challenging.
• 3:32
  It's obviously technically challenging. And I was like, you need to you have to tune in to the podcast tonight when we have Robert on for Holistic Engineering because so Robert, you and I have had the blessing of working together for a really long time, for I mean, I for much of your professional career, and much my professional career at this point too, but, I guess a greater percentage of yours, just given our relative differences in age. But you came out to Sun as a well, originally to microelectronics. Right? As an intern in 2008, maybe?
• 4:06
  When was that?
• 4:07
  Robert Mustacchi: Yeah. Yeah. I did a brief stint on the, KT verification,
• 4:14
  Bryan Cantrill: Burlington.
• 4:15
  Robert Mustacchi: And, you know, you got to see, the glory of a microprocessor that would ship really soon, not ship. So you I can you kind of got to learn lessons that we're seeing today, from, you know, when you see Spark roadmaps from, some of the other CPU manufacturers, you know? You gotta Right. Exactly. Okay.
• 4:34
  Bryan Cantrill: I I I lived that right. Okay. So then and then the next summer, you interned that would it was the next summer that you interned with us at Fishworks in San Francisco. Right? Is that
• 4:44
  Robert Mustacchi: right? Yeah.
• 4:44
  Bryan Cantrill: Right. And that was great, obviously. And then we had a, If I may, I'm sorry.
• 4:49
  Adam Leventhal: So first of all, his internship was great. I I and I can't take anything away from that. I would say that the thing I remember most from that internship is that Robert then and continues to be an amazing baker and, and maker of confections of all kinds. And I think it was that internship. I I think it was that internship.
• 5:11
  Bryan Cantrill: It was
• 5:11
  Robert Mustacchi: for Monday Friday?
• 5:12
  Adam Leventhal: Yes. Which spawned, I mean, the the It it got it is true. The social food. That that launched the 1,000 ships, but
• 5:20
  Bryan Cantrill: It it really did launch a 1,000 ships. God, you know, Adam, I'm really glad you mentioned this because, you know, people thought they were tuning in for holistic engineering, and they're actually tuning in for the origin story of what we call food for money Friday. So do you, Adam, would you wanna or maybe Robert, may Adam do the honors to tell the origin story of food for money Friday? Because I do think it's important.
• 5:41
  Robert Mustacchi: Sure. Yeah. I can add my commentary, you know, the background thoughts I have, all different things were going on.
• 5:46
  Adam Leventhal: There I I mean, I
• 5:47
  Bryan Cantrill: I just got
• 5:48
  Adam Leventhal: at at I do I do wanna sort of get to the actual point of, like, what we're talking about, but I would just say that first of all
• 5:54
  Bryan Cantrill: The point of what we're talking about is food for Monday, Friday, sir.
• 5:57
  Adam Leventhal: Okay. Good. Then allow me to digress. So Robert brought in these cheesecake brownies, which I think, Robert, you said took, like, all the butter in every Safeway. You'd, like, Smurf butter from all the different Safeways in San Francisco.
• 6:13
  Robert Mustacchi: You know, there's no more. I mean, we we used it. That's why there's an egg shortage now. So
• 6:16
  Bryan Cantrill: That's right. So that's why butter is in these pa this packaging, it's really hard to open because Robert was murfing butter back in back in 2009. And so they put a lot of legislation plates under lock and key. You gotta ring the bell. Someone comes in and locks it for you anyway.
• 6:31
  Adam Leventhal: It's California. So so Robert brings in this delicious cheesecake brownie with a 1000000000 calories. And I can't remember. I think we had already gotten to some like light food bedding here and there. And I think I said something like
• 6:46
  Robert Mustacchi: I, I,
• 6:46
  Bryan Cantrill: so I don't I'm okay. I'm I don't think that's part of her. It's correct, but okay. Yeah. So
• 6:50
  Adam Leventhal: I I think I was like, Brian, I'll I'll pay you up $5 or whatever to to eat the rest of it. And there was a lot a lot left. And I think you, like, in a rare act of self preservation.
• 7:03
  Bryan Cantrill: So it is my recollection Barbara, go ahead. What is your recollection of mine?
• 7:05
  Robert Mustacchi: I was gonna say that basically, there's I think I was gonna say there's about 1 inch of a 9 by 13 on the 9 of the, you know, on the 13 inches side that's been gone.
• 7:13
  Bryan Cantrill: Yeah. It's been gone. And my recollection, and actually it's interesting that we've got like a the difference of dollar figure, I recall it being $10.
• 7:21
  Adam Leventhal: Okay. Okay. Well Or
• 7:22
  Bryan Cantrill: maybe it was 5.
• 7:23
  Adam Leventhal: Inflation adjustment.
• 7:24
  Bryan Cantrill: You say, I'll give you $10 to eat the rest of it. Yeah. And it is like 10:30 in the morning. Yeah. Which is gonna be part
• 7:33
  Adam Leventhal: later. Yes.
• 7:34
  Bryan Cantrill: It well, so I need to I am, like, I am extremely intrigued by this offer. This offer is extremely interesting to me, and I feel like this is not something that had happened in the past because I just felt like this this was like a bolt of lightning from my maybe a way to call back to my youth of like, we can actually we can earn I can earn money while doing what I'm best at, eating. So this is like, Okay, this I am extremely interested, but I have to leave for a meeting. I have got like a- Jay Jacobs (zero fifty six:fifty
• 8:11
  Adam Leventhal: six): So what's the self preservation? So the other thing I remember is- Jay Jacobs (zero fifty
• 8:13
  Bryan Cantrill: six:fifty six): Oh, self preservation. I absolutely wanted to to I, like, I this you have made an intriguing offer, one that it didn't even occur to me to to counter offer.
• 8:24
  Adam Leventhal: What I what I also remember is that even I think even before you had declined the number like, as soon as I said, you know, 5 or $10, everyone else on the team was kicking in. It was up to $75 almost immediately.
• 8:37
  Bryan Cantrill: That is my recollection too, and like an idea whose time had come. This just exploded. I mean, everybody I I mean, no, I know that's like, I had to run for the meeting, and it felt like within seconds, the pot was at $75.
• 8:51
  Adam Leventhal: It was like, I too would like to see this person commit ritual suicide.
• 8:56
  Bryan Cantrill: Right. And then our colleague, Julie, gets wind of this. And she's like, wait a minute. Is someone, like, is someone offering $75 for someone to eat all of that cheesecake? Like, I am I'm definitely in.
• 9:06
  I'm gonna eat it all. Yeah. It's like, great. I leave.
• 9:11
  Adam Leventhal: That's right. And we had we had some time based constraints that I won't go into the specifics of.
• 9:18
  Bryan Cantrill: I don't wanna go into I I don't wanna go to the entire IOC rule book here in terms of the, you know, but with the, you know, was it a sanctioned event? You know, there's like, listen, there's a lot of, but and I came back to Julie looking in front Robert, what is your recollection? I recall her having, like, one 1 and a half inch by 1 and a half inch square remaining.
• 9:41
  Robert Mustacchi: Something like that. I mean, I'll admit I was kind of in shock the whole time. You know, you bring this in, you're kind of trying to sort of be like, Hey, here's something nice for everyone. And then it's like, that becomes food for money.
• 9:50
  Adam Leventhal: Yeah. This thing that I thought would be a delicious treat for everyone, I guess has become your prop bet.
• 9:56
  Bryan Cantrill: I guess, am I addicted to a crime or merely to very poor judgment?
• 10:01
  Robert Mustacchi: I'm definitely ruined cheesecake, I think, for some time.
• 10:06
  Adam Leventhal: Oh, god. But, yeah. Brian, it was like an inch and a half by an inch and a half square.
• 10:11
  Robert Mustacchi: Yes.
• 10:11
  Adam Leventhal: And you're just like, Julie, well, just, like, come on.
• 10:13
  Bryan Cantrill: I know. I was like, just put it in your mouth and force yourself to eat it. She's like, that's what I've been doing that's what I did for the last 4 of these squares. Like, I I've already like, I literally can't look at it without wanting to throw up.
• 10:25
  Adam Leventhal: So now this is the part of the story when usually I tell it to folks, and I'm like, and then Julie couldn't do it, so she didn't get the money. And everyone is aghast. Wait. You didn't give her the money?
• 10:38
  Bryan Cantrill: And I think no.
• 10:39
  Adam Leventhal: I don't know. What kind of lesson are we teaching the children? And this is, you know Yeah. Exactly. And we had an arguable child with it.
• 10:45
  We'd like we had Robert as a
• 10:47
  Bryan Cantrill: young adult. Arguable, You know? Robert as a in college, very, for example, to the intern who was shocked and aghast that we had taken his very kind gesture and turned it into something that was just depraved.
• 11:04
  Adam Leventhal: That's right.
• 11:05
  Bryan Cantrill: But, no, she didn't get the money. She got the she got the cheesecake. So it's like, like Uh-huh. Oh, Robert, I can't believe you worked with us anyway. We're so grateful that you that that despite it all So, we we we will absolutely need to do a podcast episode only on Food for Money Friday, but let's just say we've had other A
• 11:25
  Adam Leventhal: teaser episode.
• 11:27
  Bryan Cantrill: There's just a teaser episode. We have had other food for money adventures over the years and Robert was not the not not not for the last time with the intern involved. So, we have, which has become increasingly ambiguous for me. That's the yeah. I someone in the chat is asking.
• 11:47
  This is hazing. It's like, don't know that we actually volunteered to what we're doing to some legal.
• 11:53
  Adam Leventhal: Yeah. No. I mean and no. We weren't, like, you know, a ritual of, I don't know, whatever position it was to like eat 90% of a cheesecake. No, this was all voluntary.
• 12:03
  These were all consulting consenting adults.
• 12:06
  Bryan Cantrill: That's right. Hazing is there are many attributes of hazing that are absent here. There is nothing. Yeah, exactly. So there
• 12:11
  Adam Leventhal: actually is another, so, there's a kind of bait and switch, follow-up to this internship, which is Robert, Robert interned with us. Great. Not cheesecake included, but not with but not the only reason. It's a great internship. Yeah.
• 12:28
  Bryan Cantrill: Yeah. Yeah. Great.
• 12:28
  Adam Leventhal: Goes back to Brown University to finish out, get gets his degree. And we made Robert an offer to join us full time, and he accepted. But in the meantime, we were off getting acquired by Oracle. Yes. And and then, you know, you and I were off figuring out how not to work for Oracle.
• 12:50
  Bryan Cantrill: Yeah. And I felt very so we had I and and Robert, I left what was in Oracle before you joined. And my recollection is that I reached out to you as I was leaving. I think I did anyway. I certainly felt very bad that you're gonna be joining a bit of a ghost ship.
• 13:08
  Adam Leventhal: Yeah. And, Robert joined, and I don't I don't know if it was clear that I was leaving or, but I was like, Robert, you know, it's a great learning opportunity, is me teaching you about all the stuff that I've done and everything that I've ever owned in case it comes up. And then I think I left 2 weeks later.
• 13:29
  Robert Mustacchi: Something like that. Yeah. That's a good good learning exercise.
• 13:32
  Adam Leventhal: There we go.
• 13:34
  Bryan Cantrill: Yeah. It was nice. So but fortunately, Robert, you, joined me at Joyant. And, you and I did a ton of things together at Joyant. But, something that you and I worked closely on early on was the the port of KVM to SmartOS, to our Lumos derivative at the time.
• 13:56
  And, was that, I mean, what is your memory of those years? I mean, is that is, that was a, it was a bit of a terrifying project in that, we really needed to succeed and it was really hard. But so, you and I worked very closely on KVM, and, which is great. And then what kinda in that so what my also kinda memory of those years is all of the the the kind of our early interactions with Intel, and you really owning that kind of the the the interaction with Intel. That seemed to happen pretty early on at Joynt anyway.
• 14:37
  Is that is that an accurate recollection?
• 14:39
  Robert Mustacchi: Yeah. You know, I think we had some very early bits at 365 California. And then it was actually I think when Keith came on Right. He kinda was really driving more of those programs, and I was working with him there. And then I took over more and more of that as, Keith retired to the farm for the first but not last time.
• 15:01
  Bryan Cantrill: For the first but not last time. And so the and I mean, you'd always I mean, I obviously always have had an interest in low level software and sharing, and I've always kind of been at that hardware software interface, but definitely found yourself there very much at at Joyant. And the the, I mean, I I well, do you have anything formative that you wanna talk about? Because there's definitely something formative at Joynt, obviously. And I don't know Alex is here in the chat, but, we definitely had a, there were many episodes over over the the what, how many years were we together at Joynt?
• 15:40
  Almost 10 years, 9 years at Joynt.
• 15:43
  Robert Mustacchi: Yeah.
• 15:44
  Bryan Cantrill: But at that kind of that that low level system software interface.
• 15:48
  Robert Mustacchi: I don't know. I think a lot of just learning how to learn and learning how to debug, especially in those early years. Because there's a lot of times where, you know, I would might know the kind of question I wanted to ask, but not how to ask it. Or you or a lot of the rest of the team would kinda come in with some of the detrace or other other bits there. And I feel like also where one of the great adages, that I've kept in mind when being faced with gnarly problems, which is basically write debugging tools when you're stuck.
• 16:19
  Bryan Cantrill: Yeah. Interesting.
• 16:20
  Robert Mustacchi: Right. I don't know. I feel like that's a lot of so a lot of it is, during the times of really just figuring out how do you go learn a new subsystem that no one's been in really before and there's no one to go ask. Right. It's like, go ask like, how does this work?
• 16:37
  How does the NetStack work? Or how does the USB stack work? It's like, oh, just go use some DTrace, go use some MDB and start figuring it out.
• 16:46
  Bryan Cantrill: Okay. So this is a really interesting point about kinda learning how to learn and and help not being on the way. Like, there's no one you're in a subsystem where there is actually, like, you are gonna be, if you're not already, you're gonna be the local domain expert. So do you have a particular methodology when you're when you're going into, because I mean, one of the the, I mean, one of the things that you, among your many, special attributes, I mean, you you're a terrific code reviewer. And it mean, how much do you go into a kind of a new subsystem just by looking at like, just the actual code?
• 17:25
  Robert Mustacchi: I'd say, often there's some kind of group of questions I'm trying to ask or answer, and it'll be some combination of looking at codes. Basically, I have almost always have c scope, C scope with Vim integration that I've inherited from Dave and others over the years. Enough that I can set up Dave's keyboard and we can have the same key bindings, which is shockingly convenient. Wow.
• 17:49
  Bryan Cantrill: Yeah. God. That's that's amazing.
• 17:52
  Robert Mustacchi: But that and using that with a combination of DTrace, and this is where, like oh, I'm trying to get this. Some of the the classic one liners, like, instrumenting a, module like all entry probes in a module and aggregating on probefunk, is 1, you know, aggregating on certain stacks and just, like, seeing what happens, trying to trace control flow or data flow.
• 18:26
  Bryan Cantrill: And then, are you do you when you're kind of in a new subsystem like that trying to ramp up on it, are you, like, writing down questions that you're trying to answer? I mean, how do you because I
• 18:35
  Robert Mustacchi: mean, you you That's actually a good point. Because I think one thing that I I've noticed that is having questions in the notebook and writing stuff down in there. I think that's one of the other things that I found really valuable, just trying to figure out what are you trying to do, or, trying to go to the trying to diagram out on a whiteboard what the how the subsystem works and flows. I think, I remember we were debugging one of the what was it? There was something with the X2 APIC for the APIC's PSM driver.
• 19:13
  And that that block diagram is now in, like, osenter.c, but it filled up, like, 2 joint whiteboards. And a lot of it was just trying to understand, like, how can I understand this control flow well enough to know, like, what's going on? Where's everything flowing, etcetera? And that that's been a useful that's definitely a useful just kind of how can you understand it well enough to explain it to someone else? And I think that that's the other thing is, I was often sometimes sitting there talking with other folks in the office, or on chat, and using that as a way to kind of, like, have them ask sometimes they would ask me questions.
• 19:51
  Like, I know sometimes Josh and I or Dave and I or Patrick and I would just go there and, like or Alex, we probably did this a bunch at, 645 Montgomery. I remember there's that little bench behind your desks. And I feel like there would be a lot of kind of questions and back and forth there. I'm just like, how does this, you know, kind of the old, like, add, you know, one of the useful things about being a TA is by the time you can finally explain to someone else, you might start to have an idea of what's going on.
• 20:23
  Bryan Cantrill: Yeah. That's really interesting because I mean, all 3 of us were TAs as undergraduates. It's very formative for all of us. Mhmm. And, yeah, I mean, this is not a deep thought, and anyone who's taught knows this, but, boy, when you when you teach something, you have to learn it with a whole different level of mastery.
• 20:40
  It's not, like, you you can even do well in the course, and then you go to TA and you're like, oh, my God. How did I even, like, pass this thing? I barely, like, I I really I'm learning so much more. And because when people ask you sometimes when people will ask you a question and you'll give an answer where they'll be like, oh, yeah, that's like a convincing answer, and you're like, That is not a commit, like, you are convinced by this answer because you don't actually, you're asking me the question because you don't actually know, but I actually know the limitation of my own knowledge and I have not given you a very good answer actually, and I actually need to go research that because I'm not, so that's interesting. And then Robert, I also think it's interesting that you're you mentioned the blog comment that actually gets because I mean, your blog comments are, I I mean, Napoose Ultra, the the the ASCII art in your blog comments is the stuff of legend.
• 21:27
  The, and I and I say this as someone who prides himself on his own ASCII art in my own block comments, but,
• 21:35
  Robert Mustacchi: Honestly, it's mostly just because I don't I just if I don't do it, I'm gonna forget it all. And having it written down, like, I've gone back to some of these and been like, oh, good job, past me. Like, I'm glad this is here.
• 21:46
  Bryan Cantrill: I know. I was just gonna say I forgot this entirely. Well, and the the number of times we often talk about past Robert around here, and you will be like, we'll have a question you're like, God, I don't know. I'm like, actually, past Robert knows the answer to that question. I actually know that.
• 22:01
  And then like, fortunately, past me wrote this down. I have heard you say that so many times over our shared career and, but it's so because you mentioned that, like, when you're stuck right at debugging to right at debugging tooling. I definitely agree with that. But when you're stuck, you also write the write a block comment when and you can go make an artifact better by just explaining how it works.
• 22:25
  Robert Mustacchi: Yeah. I think as I've been looking back for my notebook here for things that are, things that aren't meeting notes, a lot of it really is, you know, just a whole bunch of things around, like, you know, if there are problems, it's like starting with questions I'm trying to answer and go figure out what those are. You know, even on Thursdays or Fridays, LRDIM hunt is the same thing. Just like just like what are some of the things what are some of the observations? What what's different?
• 22:56
  You know, and I find that writing stuff down is a helpful focusing thing and that those for me.
• 23:01
  Bryan Cantrill: I learn
• 23:01
  Robert Mustacchi: a lot better. I mean, everyone everyone learns in different ways, so it's not gonna be the same for everyone. But for me, physically writing in mostly electrical curve that looks good from a distance is,
• 23:13
  Bryan Cantrill: Robert's cursive is, remarkable. It is, I think, we we it's it's fair to say. It it gives me great Robert, there has been there have been times when I have asked you to go back and read your own cursive and even you can struggle. It is beautiful, but it's just like I but it it's got this very I mean, obviously, you've seen plenty of of Robert's curse of it. It definitely I feel like I'm reading a letter that Alexander Hamilton wrote to me.
• 23:40
  You know what I mean? It's got like it's got that kind of like 18th century vibe to it. Which is like I this is, it's gorgeous, but I just can't make it out. So, Robert, so this was all, I mean, kind of as the years go by at Joanne, I mean, you're going into to like kind of rappelling down into deeper and deeper caverns. And then I I mean personally, I feel the Naples Ultra of this was on on January 1, 2018, when, we saw a rather disturbing story on the internet.
• 24:20
  I assume. Do you want to describe what we saw on the internet and where that kind of led you?
• 24:27
  Robert Mustacchi: Yeah, actually I think I think my mind is that it was like you or Alex probably saw it first, probably with, with Spectre and Meltdown, kind of this is being like the was it put back in Linux in a way that starts leaking and everyone's kind of denying it until they stop denying it? I think is the
• 24:47
  Bryan Cantrill: Yes. And it has become a hacker news story on January 1, 2018, if memory serves. And, I think I am I mean, certainly, I Alex, I'm sure saw it first. I definitely DM'd you being like, are we do you think this affects us? You're like, I don't know.
• 25:06
  We're learning about this for the first time, and this is us learning about Spectre and Meltdown, and then discovering that, we are vulnerable and we are running in production in a public cloud. And, Meltdown in particular, was really, really acute. John Masters had done terrific work having a real, like vivid proof of concept of Meltdown.
• 25:27
  Adam Leventhal: And how many, how many Intel CPUs did you have deployed at this point? It can't have been that many. Right?
• 25:34
  Bryan Cantrill: Yeah. You know, it's funny, like, that is definitely like not a question we were gonna go answer because it was just like, it's not actionable. It's, you're just like, how far is it down to the bottom? And it's like, it is enough and you're at a height where you're gonna die and nothing else actually matters. So it's like
• 25:50
  Adam Leventhal: it Beyond that, right?
• 25:51
  Robert Mustacchi: Beyond that, it like More than 10. Yeah.
• 25:56
  Bryan Cantrill: It was a lot. And, Alex and Robert, needed to go implement kernel page table isolation. And, I mean, Robert, I don't maybe that doesn't stand out for you as much as it does for me, but I that is one of the singular engineering efforts I feel I've ever been kind of in the proximity of. I mean, it was really extraordinary, under like the, just, on the one hand, maybe the pressure was clarifying. Because you're just like, well, I didn't create this problem and, we'll definitely like it's very clear what to go do.
• 26:29
  This is not there's not a question about like, what is the highest priority thing to go do, which I feel is something as engineers we always grapple with. It's like, the highest priority thing is to go do this neurosurgery on the VM system.
• 26:44
  Robert Mustacchi: Yeah. There's a lot of different parts to it and so it was a combination of Alex and also John Levin.
• 26:50
  Bryan Cantrill: Right. Yeah. Right.
• 26:51
  Robert Mustacchi: Very helpful to have, while working on that and obviously lots of conversations with others. But, yeah, there there was a we were able to kinda split up the that work into a bunch of different pieces. I think I dealt with per CPU page tables, which was an exciting thing in its own right. I think Alex dealt with a lot of the trampoline assembly, but we also kind of settled on a somewhat unique solution. I feel like that hadn't really been done by others, with the per CPU page tables.
• 27:28
  Bryan Cantrill: But So do you could you talk a little bit about the problem we were solving? Like, what was the problem that we needed to go solve? What actually is Yeah. This is and this is just to be clear, this is for Meltdown, not for Spectre, but Meltdown was a much more acute of 2.
• 27:42
  Robert Mustacchi: Yeah. That's that's a great question. So, on x86 ARM, and a bunch of other and common RISC 5, RISC V, CPUs. When you use the MMU, you have page tables that describe, virtual to physical mappings. So every process has its own address space, and maps to generally disjoint physical memory and those page tables describe where they exist and different attributes.
• 28:10
  So some of those attributes say, you know, this page can be read, this page can be written, this page can be executed, One of the attributes are basically permissions in terms of which, what the privileges are required, to read, write, or execute that page. So you can really kind of think of this as that, there's is a whole bunch of memory that people sometimes call kernel memory and then memory for processes but effectively, if you kind of take let's just use the 4 gigabyte 32 bit address space as a simple example for a second Every process has a 4 gigabyte address space or, you know, 64 bit process has 64 bits, with a bunch of holes. But the top gig in that 4 gig address space is always the kernel, and it's the same in every process. But, and that's there so that when you make a system call, you can start executing kernel text and you don't have to go try and, you know, basically change the image context, change the page tables because that's generally expensive and potentially causes cache invalidations and a lot of, you know, it's the root of a lot of, CPU performance challenges.
• 29:23
  Bryan Cantrill: So And this has always been in x86. I mean, other architectures have done like Spark did things differently Yeah. Spark did with address based identifiers.
• 29:31
  Robert Mustacchi: Yeah. Exactly. But and ARM through various incarnations these days, especially in the 64 bit, ARM V 8a profile, It looks very much like an X86 does, in that regard. But eventually there's a bit in there, or, you know, a few bits that say, is this should this page be a kernel page or a user page? And effectively, you know, what that's meant to say is that if you're a user process, even though those kernel pages and those kernel VAs exist, if you try to read them, you'll get a page fault and, you know, then the kernel will come and, you know, drop a signal on you to basically, you know, say you've been reading something that you can't.
• 30:11
  Unfortunately, through the power of speculation, what basically happened is that, that check happened, but after all the side effects of doing the read, we're pretty much done. So everything other than, you know, it doesn't show up in your register, but it's impacted, but it was loaded into all the caches and everything else such that, you could still see it. And basically, so basically you could read any arbitrary piece of kernel memory, you want. So whether that was, you know, someone's packets, security keys,
• 30:51
  Bryan Cantrill: you
• 30:51
  Robert Mustacchi: know, someone else's file system cache data. It was, yeah.
• 30:56
  Bryan Cantrill: And to be clear, the way you would do that is because you can, what you are controlling, that you should not have been able to control, but you are able to get this thing to do a load for you, but you can't see the results. So, the trick is, how do I exploit the side effects of that load, namely allocating in the cash? And then what can I go do to exploit those? And the things you can go support, as it turns out, you can have a conditional branch that then also gets executed speculatively, and then that can do something else in the cache that you can then go observe the side effects of. So, you can kind of chain together just to People may wanna know, because I think one question people have is like, Well, wait a minute, why didn't someone discover this a lot earlier?
• 31:39
  And the answer is like, it's sophisticated in that you are
• 31:45
  Robert Mustacchi: you do have to kind of
• 31:45
  Bryan Cantrill: chain these things together and I think that this had not this had been kind of in the abstract, we knew that kind of speculative attacks could be could happen. I don't think anyone thought that it would that they were gonna be this brazen. And in particular, it is really bad, that it would speculatively execute on these addresses that you don't have the ability to read. It should the the chip should never have done that. And Yeah.
• 32:11
  Robert Mustacchi: There there have been a bunch of a buildup in literature of kind of different l 3 cache attacks of kind of, like prime and like prime and pump and other things where you kind of start using L3 cache as shared resources, but people didn't expect you could kinda go through the page tables. Or, my favorite one really is eager fpu, which is kinda just a fun one of just like, oh, you really can speculate through everything.
• 32:36
  Bryan Cantrill: You really can speculate. I mean, with eager fpu in particular, and this is like fast forward now, I don't know, maybe 9 months or 10 months, and we're having kind of what these constant calls with Intel where they're reviewing yet more. Because it is also true that, like, once the that that the group discovered Spectre meltdown and I mean, we knew this was gonna be just like, oh my god. It's gonna be an absolute bonanza. Because now people know this is a target rich environment, and I can now go explore every unit in the part.
• 33:05
  And Robert, I think it was the eager FPU when I had missed the call with Intel because of another conflict. And you said, well, we've got another disclosure. What is the one unit we have not yet heard about from them? And I'm like, I we've not heard anything about the FPU. You're like, yes.
• 33:21
  It's the FPU. It's like, oh my god. Literally every unit. Yeah. But yeah.
• 33:28
  I mean,
• 33:29
  Robert Mustacchi: I'd say that's that's been one of the the one of the small advantages of working for smaller companies that you get to explore a lot more of this stuff, than you just really do at kinda some of the larger places because there's just you know, we're less cemented teams. So often, you know, there's not like, you know, a big kernel org sitting by or Right. You know, even if you look at, like, Apple, you know, what they have, they have a lot of different, you know, groups there is that, you know, a bug comes up, you're gonna pass it off to that group, and, you know, you're not gonna chase it down or look at it or have to figure it out, which is sometimes with a blessing and a and a curse. It can be great to have, you know, it's great to have a lot of different colleagues, but sometimes, it means there's less opportunities for you to learn or kind of move around in that regard.
• 34:11
  Adam Leventhal: So I'd also say, Robert, if there's fewer opportunities for you to I I think one of the things that you do well, I'm sure we're gonna get into this, is cast that that I that is so well informed about disparate parts of the system. And I think that you're right that, like, a strength of a large organization is that you can have experts narrowly focused and, you know, it's it's sometimes just a lot easier to be productive along those lines. But I think what they miss is the kind of system wide insights that you've been able to make, you know, at Oxide, for example, because of your aware because of diving deep into all these different areas.
• 34:49
  Bryan Cantrill: Yeah. I I think that the because I I agree, Adam. I think that, like I don't know, Robert, sorry to speak about you in the 3rd person, but I feel like that is something that I feel I've I've kind of have seen more at, like, at Oxide, where it's like, at Joint, like, you dove deep into so many different domains. And then coming to Oxide, maybe it's a good opportunity to kind of fast forward Oxide a little bit, but, you know, when we, we started the company and, able to raise money, and were able to convince you to join us, And you'd been thinking about a lot of these you'd gone deep in a lot of these disparate areas to the point that you were now really beginning to synthesize them. And maybe it was just that Oxide gave us the opportunity to do that synthesis.
• 35:37
  But, is that I mean, as you kinda think about your own story, is that is that kinda when that synthesis really begins to come into focus?
• 35:45
  Robert Mustacchi: I think it's when we're able to execute it most most directly. I mean, so I think there's a you remember Keith and I had like Keith was driving together. We had kind of this dog patch pitch. Must have been 2015, 2014? Yes.
• 36:07
  Maybe 2016 at the latest? Can't be that much.
• 36:10
  Bryan Cantrill: Oh, I did. Not 2016. It would be like the 2014, maybe even 2013.
• 36:14
  Robert Mustacchi: Yeah. Of kind of where we want to go, but with the constraints but with the constraints that, you know, effectively, you know, we're not doing our own boards. We're not really getting to that low access. You know, how do we how do we work with folks there? And at the same time, there's broader business constraints around, you know, who you have to work with, what's available on the market, you know, ultimately, to be economically minded in different ways.
• 36:39
  So But,
• 36:40
  Bryan Cantrill: you know
• 36:40
  Adam Leventhal: Robert, can you talk about more, like, what what problems you were trying to solve with dog Dogpatch and what were the solutions? But before you get there, I just wanna say, Brian, you mentioned hiring Robert. You failed to mention that he was actually your 1st hire. So, he is employee number 1.
• 36:53
  Bryan Cantrill: Joshua Sharfstein (zero zero six:fifty six): first hire and also, I feel we definitely are so desperate for Robert that as soon as Robert is Robert kind of like tentatively said like, I guess I work here now. And we were both like, yes. Yes. You work here now. Yes.
• 37:15
  Yes. You work here. It was there was a moment of where it's like, okay. Like, Robert they don't so, like, sorry, Robert. You can't take it back.
• 37:20
  Now you work here. It's the, but, yeah, we were, I mean and and it it takes a special like, I and Robert really, obviously, appreciate the I mean, just like the because the the company's nothing at that point, you know? And, like, and, again, I know vividly where I was anyway, Robert. I don't know if you remember where we were, but on kinda Jess's little back deck there as, you know, you'd obviously had some questions. I felt we kinda knocked down all the questions, and you're like, alright.
• 37:46
  I guess I'm I guess I'm in. I don't know.
• 37:48
  Adam Leventhal: I know. But it's like raising money, certainly, it gives you that feel of tangibility, but it's when other people are now saying you know, people you've worked with, in Robert's case for, you know, 5, 10 years or whatever, saying, yeah, okay, I'm in. Let's do it. It it's a it's a new kind of reality setting in about the the the venture you're heading down.
• 38:11
  Bryan Cantrill: Yes. Yeah. And not too long thereafter. And I I wanna get Robert back to dog patch in a second. But, and I'll I'll I'll definitely drop this photo into the chat because I've got, I mean, Robert Ruppert.
• 38:24
  Robert Mustacchi: Is it the whiteboard? Yeah. Because I was I was just saying, if you have that, you should you should drop that in. So I feel like that's that's emblematic of the of the discussion we have. It's like, ah, like, let's drop these different pieces on the whiteboard.
• 38:34
  And then we go back and forth. Like, oh, let's talk about what's connected. You're like, do you just me fully connected? And
• 38:39
  Bryan Cantrill: of course,
• 38:39
  Robert Mustacchi: you are right. But
• 38:41
  Bryan Cantrill: Yeah. Yeah. I mean, it was it was really great because they it it, we had all of these different thought so Robert is we've got all these kind of different elements of the system. And Robert is like trying to figure out like, okay, what are the kind of the connections between, you know, the root of trust and, you know, what's the the we've got the switch, we've got all these different elements. And I'm like, you're this is gonna be a fully connected graph.
• 39:05
  And, and sure enough, it it after not too long, it it was. So yeah. I'll I'll I'll drop in that photo, Robert, but it's, it's definitely a great one. But do you do you mind just to go back to Adam's question, like, what was Dogpatch? What were the problems with the cable with respect to Dogpatch?
• 39:23
  Robert Mustacchi: Yeah. So I think, to really start to get at Dogpatch, I actually had to rewind to actually back to Phishworks and AK, which was the appliance kit. So there was a lot of hardware software integration we did there around manageability and serviceability in particular, and if you're ever fortunate enough to use one of those, you know, there was a lot of things it did around just like, hey, detecting drive failure, indicating that, blinking an LED, and even being able to blink the LED, tell you where it was, already started resilvering, you know, swapped in spares, sent out emails, did, you know, contacted support potentially get to get, replacement parts shipped, etcetera. And, you know, that was kind of at a single box level. And really and that was really 2,007, 2008.
• 40:17
  Yeah. Then you kind of fast forward and, you know, it's, you know, even 2013, 2014, and the ability to kind of deal with that data center management at scale, you know, kind of getting to that warehouse style computing, is really limited unless you're one of the really big players. You know, you're making an LED blink reliably. Surprisingly challenging. Actually, even I mean, honestly, in half times, even after you figured it out, actually, you've learned that the server was built incorrectly and a bunch of things, to drive, you know, to the basically SaaS expander or maybe the drive, it's like the drive backplane cables were swapped.
• 41:00
  So what you thought was drive 0 was drive 8.
• 41:05
  Bryan Cantrill: And so you were literally lighting the wrong LED. No, I'm not.
• 41:08
  Robert Mustacchi: The wrong LED. So,
• 41:11
  Adam Leventhal: Hey. AE drive failed. Come on. I don't know.
• 41:13
  Bryan Cantrill: Exactly. Actually, like, a drive failed. So I want you to remove one of the good ones. Oh my god. Did did that help?
• 41:21
  That helps. Exactly.
• 41:23
  Robert Mustacchi: But also all this had to be done very manually and, you know, our the joint operations team put a lot of work into dealing with and managing that and dealing with the kind of lack of features. But, you know, the serviceability and manageability story which, you know, worked good at, you know, the 1 to 2 system or if you had multiple systems, you know, at FishWorks we really wanted to bring, together, and you actually go see if I actually have this deck still somewhere.
• 41:55
  Bryan Cantrill: Yeah. The dog patch deck. We've got the dot. Yeah. Yeah.
• 41:57
  Yeah. Yeah. It's a it's
• 41:58
  Robert Mustacchi: remind myself what else is what else was actually in there. But, yeah, I think this is actually where you know, I think the big thing is, you know, there's a lot of fights between what's being done by the OS and what's being done by the BMC in those systems. Because basically the BMC is basically where there's a whole bunch of value add from the vendor, for varying degrees of value.
• 42:24
  Bryan Cantrill: Oh my god. Daikou just almost came out my nose.
• 42:27
  Adam Leventhal: I know. I could those air quotes are just like screaming.
• 42:31
  Bryan Cantrill: Oh, God. Yeah. I needed a little bit of a warning on that one.
• 42:39
  Robert Mustacchi: Yeah. And just the features of that, the added had were were challenging.
• 42:46
  Bryan Cantrill: Robert's being very forgiving to not mention this, but this is when we first heard about Redfish. For whatever reason, I somehow got I I'm like, I think this can really help us. I'm like, Redfish, this seems really interesting. This is gonna solve exactly this kind of problem that you mentioned. And I remember I remember you being like, I don't think you understand what this is.
• 43:07
  Like, that's not gonna you are no. No. Sorry. This is taking an HTTP layer and smearing it on on top of the same garbage. This is No.
• 43:16
  This is not gonna help us at all. This is And that, of course, sort of, I was like, no way better. I'm sorry.
• 43:21
  Adam Leventhal: But, like, that is the marketing of it. Right? Like, that that is the Yes. Ostensible value add of Redfish. Just It's
• 43:28
  Robert Mustacchi: it's it's totally divorced from reality. And it's gotten slightly better than when it first came out as a empty schema. But, yeah, as I'm trying to look through this to see some of the other things, like, obviously, you know, some of the classics, dealing with firmware, you know, the different kind of architectural challenges we had there. We actually were thinking about how do we eliminate, the BIOS in UEFI and basically just do a, you know, a small, you know, basically let the bootloader take care of all the a whole bunch of stuff, you know, that for us was Ipixie at the time. So it's just like how do we basically just get out of these different layers that are differently broken?
• 44:15
  I think one of this is during that as we were writing this, this is one of those times where we had, I think it was like a, oh, like a Dell, maybe like Haswell Aero Server, and after we typed reboot it would just hang in the BIOS. Like after we, you know, like like we just reboot like Oh my gosh. And it would only happen on a warm reboot. And, there's a lot of back and forth of being like, well, it's your fault. You know, don't tell
• 44:44
  Bryan Cantrill: me about this.
• 44:45
  Robert Mustacchi: And it's like and and you're trying to, like, you know, less some of us would say it less politely, you know. I think, yeah, this is where we had good good cop, bad cop, and psycho cop. But, you know, it's like, hey, the BIOS has just, like, erased all of our program text, and it's taken over. How is it, you know, and it's job is to restore it from an arbitrary state, so how exactly is it our fault? But actually that that in of itself is a lesson as to one of the things that we actually did in oxide, which is getting rid of 2 different reboot paths, to actually simplify the system and streamline, streamline things.
• 45:23
  So that is, you know, in a standard system, if you type reboot, it's not gonna do a full post. It's not gonna go reset everything. UEFI is gonna kinda be clever and or the the real reality is that the CPU actually isn't gonna erase everything. So even if you reassert the reset line, if you're in this ACPI s 5 state, there's a whole bunch of state that stays across that. So that also means there's 2 different path 2 different initialization paths.
• 45:52
  Some of this data you'll see is in some of these data you'll see is in some of these data you'll see is described as being in certain power wells or as sticky across resets. And so we're just like, let's have none of that.
• 46:01
  Bryan Cantrill: Let's let's
• 46:01
  Robert Mustacchi: kill all power. And that actually made it easier for us to build a more reliable system with actually less, you know, with less codepads to actually think about because we can actually say there is no warm reset, there is no way there. Then at the end of the day, what that means for customers is that, hey, it works more reliably more of the time. Because a lot of the challenges here is that you have all these different code paths and it's hard to actually test reset in a bajillion different ways. Or what does it mean to do a warm reset when, you know, you've been up for 2 years versus you've been up for 30 seconds versus, you know, I've hot swapped all these drives upteen times versus, yeah, I've done nothing, so I've just been up for, you know, 2 minutes.
• 46:41
  So it really got us out of, that problem once we ironed out some of the bugs.
• 46:47
  Bryan Cantrill: Yeah. What's really so and this is again, this is ad hoc side. This is not something I I dropped the the dog patch deck, from Joyant, into the chat, Robert. But the, circa 2014 is when I've got it. So Yeah.
• 47:00
  Yeah. The, and actually, I have actually 2014 on there. And you, I mean, you will see a lot of the oxide vision there, in terms of illuminating both Bias and UEFI. Much easier said than done. And Well,
• 47:18
  Robert Mustacchi: no one took us
• 47:19
  Bryan Cantrill: up on
• 47:19
  Robert Mustacchi: it, so
• 47:20
  Bryan Cantrill: No one took us up on it. And so, I mean, I mean, a bunch of things. I mean, at 1, I mean, I think this is the kind of thing that nobody disagreed with us, that this is the right thing to go do, but it just seemed like, God, it seems really, really, really, really hard. AMD thought it was impossible. And it was in order for us to I mean, it's this is the part of the part that is not well documented.
• 47:47
  Oh, and I mean, do you wanna describe some of your trials and tribulations about, like, how did we do this? How did we pull off this? Because, I mean, I think this is I mean, unfortunately, like, we had AMD's cooperation at the level of AMD was not gonna, like, get in our way. Right? I mean, AMD is, like, what you're doing is so hard, we don't know how to help you, but we're also, like, we're supportive of this effort in the abstract, which is extremely important.
• 48:14
  But it's also like not hugely hugely helpful. How did, how do we go about on that particular problem?
• 48:21
  Adam Leventhal: They're help they're helping by not actively sabotaging us is what it sounds like.
• 48:26
  Bryan Cantrill: Listen. That's I sorry. Like, what they get What's that? They're good. That's ahead of the class for you.
• 48:34
  That is that's that's terrific. They actually, you know, they did actually one better in a very important way. And this is why people may have seen the OpenSil effort from AMD, around OpenSilicon initialization. We are very, very supportive of that effort even though we're not using it at all because it was tacking in AMD. AMD was going in a lot of the same directions that we were going.
• 48:58
  So it actually but but that didn't like, what we were doing was far earlier than open before open cell was in really just kinda still in in its kinda earliest phases. So we really we're we're on our own. And, Robert, talk about I mean, this surely, the the work that you did there to understand Giza has gotta rank as one of the your your most challenging projects in terms of understanding a foreign code base?
• 49:25
  Robert Mustacchi: Yeah. And I'd say it's it's even more challenging as one that we I mean, at least at the time, there was really no good way for me to build or run or instrument. So it was really that was really all about code inspection.
• 49:40
  Adam Leventhal: Robert, can you talk about what a Giza is and, like, where Yeah. Or, you know, what we needed to do, in in that domain?
• 49:47
  Robert Mustacchi: Yeah. Yeah, Adam. That's a good question. So, Akisha is effectively one part of AMD's boot software. So it contains both the, you know, both the all the kind of binary blobs at the PSP or other kind of hidden cores run, but also really contains a whole lot of all of the x86 initialization things.
• 50:12
  So, you know, how does how do memory mappings get set up? You know, how do various pieces of the data fabric of, you know, CPU initialization, you know, turning on and, you know, when you start there's only 1 CPU turned on and even though operating systems have this, you know, traditional IPI dance, there's a whole bunch of other stuff you have to do in advance to start this. You know, the AMD SoCs, like others, you know, has a 128 PCIe lanes, but those PCIe lanes can be carved up into arbitrary different slots depending on what board you have. So how do you communicate that? So, AGISA, which looks like what does it stand for?
• 50:53
  AMD Generic Encapsulated Software Architecture.
• 50:57
  Bryan Cantrill: Only what are the letters and acronyms?
• 51:02
  Robert Mustacchi: So that that whole bit is designed and ties into a separate, these days, UEFI code base. So it and of itself is not a complete is not the complete picture. You also need it's built up as a series of UEFI modules that runs in the, PEI and DX and DXY, phases, which are different phases of boot and still means that you need, like, a tano core or other UEFI implementation to kind of fit alongside it. You know, it will do anything and everything from setting up I squared C devices to, you know, that's where SMBIOS tables are created, to just, you know It's presuming where
• 51:56
  Bryan Cantrill: ACPI tables are created as well. Right? I mean ACPI tables
• 51:58
  Robert Mustacchi: are created there too. And just a whole bunch of a whole bunch of a whole bunch of stuff. So, yeah, so that that was an effort where
• 52:07
  Bryan Cantrill: And the so it also should be added, Robert, that it so you're like, okay. So you got this, like, early platform initialization thing. So you have to, like, follow all this the code flow through that. It's like, yeah, about the code flow, can you describe what makes following the code flow through here absolutely brutal?
• 52:29
  Robert Mustacchi: Oh, yeah. So as I said, these are all UEFI's designed in a series of different modules, and they all basically rely on different callbacks firing. So because these modules are kinda coming up and loading in probably somewhat defined but arbitrary orders, they'll often wait, you know, so before I begin, like, the PCIe module or the mbio module might start loading but it's gonna register a callback that fires when all these different, PPIs, which is a UEFI term, are provided and, lots of them, and there's not like, or at least the things we had access to. There was no clear map of, like, this is the expected ordering of these different, these different sets of services. So it's like, when does the logical SoC service begin?
• 53:27
  When does the memory map service start? When is it blocked on? So a lot of it is basically trying to come up with this, effectively callback driven control flow, and trying to understand what is that just by purely reading, which is not straightforward and definitely not always correct.
• 53:47
  Bryan Cantrill: Well, and and so, like, unlike with the kind of other things, when, like, when you went into the page tables to understand that, to understand the VM system, You're armed with the source code and you also have some tools that you can use to actually observe the running dynamic system. You don't have any of that here. You have Or very, very little. I mean, I guess the thing that you could I mean, you you you can go dork with some of these attributes, these APCV tokens, and you can, I guess, watch how this and I guess you did do that because I know at least once you're like, could you go over to this machine and see what's happening? Cause I was in the office and you were not.
• 54:27
  And I'm like, this machine yeah. I was gonna say that.
• 54:31
  Robert Mustacchi: So that was that was a very different, yeah. That that was, dealing with the hitting one of the hidden cores.
• 54:37
  Bryan Cantrill: Oh, right, right, right. That was actually sending messages to Right, right. This is on our own software. We're actually you were not actually trying to understand their software. You're actually running your own software.
• 54:44
  Robert Mustacchi: Yeah. Yeah. That wasn't our software. We're trying to send a message to the hidden core, which is responsible for PCIe initialization. We're also trying to send it a data structure that has, you know, all the mapping of all of our lanes and all of this fun stuff.
• 54:56
  And then we send a message and, as you know, then wait a while and surprise, it came back to the bootloader prompt, which is never a good sign.
• 55:09
  Bryan Cantrill: Never a good sign.
• 55:10
  Robert Mustacchi: The kernel's up, so like, in theory, if I had, you know, in order to take a page fault or a double fault, you know, you'd trap into KMDB and, you know, you could debug. And that's when I started asking Brian, like, going off the system and, you know,
• 55:24
  Bryan Cantrill: that's the thing. Powering off. Yes. Yeah. It was it it was like you were putting a message in a there were a lot of, like, messages in the bottle.
• 55:31
  Where you see, you're putting a message in a bottle, you're sending it out to one of these other hidden cores, on the dye. And in this case, you're putting the message in the bottle, chucking it towards the island, and then the island was somehow, like, bursting into flames and sinking into the ocean. You're, like, I I don't send that one, I guess. Like, okay.
• 55:49
  Robert Mustacchi: Like, okay. That's a that's
• 55:50
  Bryan Cantrill: a no thank you on that message. Let me go meet you.
• 55:53
  Robert Mustacchi: The island launched global thermore nuclear war. So we
• 55:56
  Bryan Cantrill: don't know. He actually like, woah. Okay. And it's brutal.
• 56:03
  Robert Mustacchi: Yeah. And that one came down to a missing right to one of the registers in the effectively the mbio, which is Northbridge IO, which has parts of what the memory map are. So we were basically trying to do DMA to an address, and it didn't have a it was missing information that told it whether that was DRAM or memory mapped, IO. And so it probably hit some internal error. And, you know, especially since then we didn't have good observer, but we were still working on a reference platform.
• 56:45
  So we didn't have our service processor, our other stuff there. So we couldn't see if there were, you know, some of the low level, certs were being fired that would trigger something on a pin. Right.
• 56:56
  Bryan Cantrill: We had nothing.
• 56:57
  Adam Leventhal: So how on earth did you do this? Like, what was it just back to the code and reading more? Like it just
• 57:04
  Robert Mustacchi: Yeah. Well, and I think this also gets to, some of the tooling stuff. So we had actually built up a whole bunch of random, demods in KMDB, because there was no user LAN, so it's all Right. All in the kernel debugger to basically be able to read and write some of these, different register spaces. So there's the system management network is one of them, and then the data fabric is another one.
• 57:28
  And having that tooling be able to do that just let us kind of do some inspection. You know, we use this in other problems because we could that is something that we could use on our without the oxide architecture. So we we actually sometimes would compare and contrast that to what we saw on i86 PC, on a standard PC. But for this one, it was really code inspection, double and triple checking, rereading, getting it wrong a lot, and I don't know. There was a bit of a yeah, a lot of the time it was kind of a blur.
• 58:03
  I'll be honest. But,
• 58:04
  Adam Leventhal: were the, were the iteration loops on this just like, you know, 5, 10, 20 minutes? I mean, it sounds like
• 58:13
  Bryan Cantrill: Depends. It's a lot. It, I think you I I yeah. Go ahead. It has.
• 58:18
  Robert Mustacchi: The actual boot and, like, load time isn't that bad. It's really more of the mental
• 58:24
  Bryan Cantrill: Yeah. The
• 58:24
  Robert Mustacchi: mental effort there, I'd say. Just, like, knowing what to do next. I mean,
• 58:27
  Bryan Cantrill: you you even love to have so much to iterate on that you're blocked on the iteration loop, which yes, is like 20 minutes, but like that's not even the problem. Like, you have the despair loop, which is actually much longer, which is
• 58:40
  Adam Leventhal: Oh, yeah. I mean, you're I'm factoring in the trip to the therapist and then the drive back home and right on.
• 58:45
  Bryan Cantrill: That's right. Well, and I I do wonder on some of that stuff because in Robert, you so frequently, you tackle these problems where the stakes are high. Like if we don't resolve this problem, we've got a serious issue. And but you you you know, you you're you've always got like a very cool head when you go to debunk these. Do you have like is it I mean, do you just have like a an innate sense of confidence?
• 59:10
  Are you not as scared as I am? I'm terrified. Are you are you not terrified?
• 59:16
  Robert Mustacchi: No. Don't worry. I'm plenty terrified. And I I think part of makes you feel better. You know, I I think part of it is also that I actually am never doing this alone no matter where I've been at.
• 59:26
  Even if Yeah. Other folks don't necessarily have all the background, You know, that time, you know, Keith and I were working together a lot. But, you know, other folks would sit there, help listen to things. It's been true when I've been bugging up and down across the stack, whether it's software, whether it's hardware. You know, there's it it really is a team effort.
• 59:52
  And even though some of the debugging is there, you know, is sometimes a solitary activity at first, you know, a lot of the other pieces like writing it up, having you know, especially Keith and I when we were doing Bring Up, I think the fact that we were working together, even though we're tackling different parts, means we would write up different things in some of our bug reports, review each other's notes, ask questions of one another, and the act of asking questions and being forced to answer them, is there. You know, that's that's been the same thing that's true when we've been bugging, you know, some of the t six stuff which, Nathaniel and I were working on recently or, you know, other things that, you know, we're we're doing that are up and down across the stack is, you know, I think the first thing is that, you know, you're not alone and you have the expertise or just even just the different perspectives of all of your colleagues. And I think that's that's really invaluable in and of itself Because they'll ask you questions that might make you think in a different way or prompt different thinking.
• 1:00:55
  And, you know, it it can never be just, you know, yes, sometimes you need some time where you kind of you you get off the world and just kind of stare at things. But really it is working with others, I think, that ends up being necessary to kind of get through some of these harder bits because that's, I think, partly what helps helps you get through it. Because, you know, if I'm stuck and I see Keith making progress, then that helps there. Or if I see us making progress, you know, on the board work or higher up in the, you know, the control point or other parts of the product or, you know, down below, then that that kinda helps, you know, motivate you to kinda keep going and, you know, it's not all bad.
• 1:01:34
  Bryan Cantrill: But Right. Well and and I think that I mean, it's because it's part of what is so kind of remarkable, but the way you are able to approach the system is you do oscillate from these the absolute lowest layers of implementation. Like really, stuff that is often, like, does not have a lot of eyes on it. I mean, like, the the deepest aspects of system implementation, ones that are absolutely required for system correctness and liveness. And then you're able to oscillate back up to this really much broader view.
• 1:02:08
  I mean, I we need to drop a link to rfv63 in here. But in terms of, like, you know, just so at the same time, you're kind of like in the kind of the deepest possible muck in the lowest levels of the system in terms of early boot. You're also, like, at the whiteboard in terms of conceiving of what the kind of the networking interfaces we wanna have for customers and the way we kind of think of that problem. I mean, could you speak to that a little bit? Because I I mean, just that ability to oscillate, I just feel is extraordinary, and it's been such an asset, I think, to your teammates so many times over.
• 1:02:51
  Robert Mustacchi: Sure. I can try. I mean, I think there's a lot of stuff that's just virtue of, you know, being at the company early. There wasn't really a lot of other folks to do stuff.
• 1:02:59
  Bryan Cantrill: Yes.
• 1:03:01
  Robert Mustacchi: So, you know, there there there is a lot of stuff where it's just like, hey, how do we start thinking about this? And I think for me one of the things that's there is that, you know, you'll see in that actually R5063 was like the last of a was like the last of a set of networking docs, It really started from, you know, higher level product comparison kind of feature use cases, user networking API that we wanted in another doc. And then with those that in mind, how do we start building up the lower level stuff?
• 1:03:33
  Bryan Cantrill: And I know we go to this metaphor a lot, Adam, but I'm sorry. These are the Federalist Papers for Oxide. These are these that these early RFTs from Robert.
• 1:03:43
  Robert Mustacchi: But I I think a lot of it comes back to you know, for some of these things are things I've been thinking about for a while so that helps. You know, even though RP 60 3 there, like, there's a lot of retrospective in on past attempts there or goals or, you know, things that worked well, things that didn't work well, you know, what we learned from paper reading and other stuff. And, you know, I think coming back to the dog you know, if you go back to that dog patch deck in 2014, there's a lot of the high level goals and, you know, I'd say experience and kind of usability goals. You know, are still kind of, you know, are things that are kind of at the always in the back of my head. So I think that that that's always helped to kinda anchor that or kind of raise questions about, you know, what does this future look like?
• 1:04:29
  What does it mean? How does it actually tie back to those goals that we have? And then I think sometimes Yeah. Sorry.
• 1:04:36
  Bryan Cantrill: Go for it. No. No. Go ahead. Sorry.
• 1:04:38
  Go ahead.
• 1:04:39
  Robert Mustacchi: I think sometimes, you know, you need to work being able to get you know, well, maybe it's a little bit of thrashing, and sometimes you need to kinda really focus on one or the other. It helps to kinda for me at least, to go to different different things. Because sometimes some of these small bugs are, you know, small bugs are kinda low level details. You can kinda really just focus in on it, but having the broader context of how that fits into the system is helpful or a good distraction, from that. So I don't know.
• 1:05:10
  It's not that It's not really good answer.
• 1:05:12
  Bryan Cantrill: Yeah. I mean, no. No. That's a great answer. And I think it's I mean, it is really I mean, I and I have always believed, I mean, out of you YouTube, this is, like, our a unifying belief, I would say, across the company, is that details really, really matter.
• 1:05:25
  And that you that if you wanna understand things at the highest level of a system and understand how these big pieces go together, you really need to understand low level details in order to be because if you don't, that's where you you get these kind of emergent behavior that that really runs contrary to goals you have as a system. So I think it's really, really important. But, I mean, your that that your ability to do it, Robert, has just been extraordinary. And the thing that I wanna ask you about because as you say, like, okay, it's, you know, it's early, so you're putting together the kind of, you know, as our John Jay, you're putting together the Federalist Papers of Oxide, But we are then adding people to the company that are able to go pick up pieces of that. And, you know, you've always been, I think, terrific about really enabling other people to go pick up these pieces.
• 1:06:18
  I don't think anyone has ever felt like, oh, god. I can't touch networking because, you know, I can't I can't touch this aspect of the system. Because you're always like, no, please. Someone, please come on in. Like, water's warm.
• 1:06:27
  I would love to help you ramp you up and help you understand what I understand so you can go tackle this. Can you speak to that a little bit as well? Because I think that's that's part of what has been has enabled you to to be so effective.
• 1:06:40
  Robert Mustacchi: Yeah. I mean, I I think you're right. I think there's a there's a lot to be said that, you know, I think the important thing is that as you go from, you know, the first couple of us that hire to 10 to 20 to 30, you know, and and you continue to grow, is how do you help, teach other people what's going on? And how do you help ramp them up? Because I think that's equally important because yeah.
• 1:07:00
  And, the amount that one person can do is is bounded. And, sure, you can, you know, work overtime or, you know, push yourself a bit, but that will never be as effective as actually, you know, getting more people together. So I think that that to me is kind of a is a important question of how do you help teach people? How do you help them learn? How do you try to be helpful and help, you know, let folks be productive, and share knowledge?
• 1:07:33
  Because otherwise, yeah, I don't know. So I think that's the thing.
• 1:07:38
  Bryan Cantrill: We got into this we got into this in the RFP episode that you joined us for. I mean but, obviously, I mean, you are our most prolific RFP author, and, you know, I think it's been great for people to kind of people to join the company and are actually also I've had I'm sure you've had this too, Adam, where, like, people who have not yet joined the company, like, Robert, you're famous to people who have not yet joined the company, because there's, like, I've read this guy's work. I wanna meet this person. Like, this is I I feel like I know their voice. Yeah.
• 1:08:08
  And but it's
• 1:08:10
  Robert Mustacchi: Yeah. But I I think that's you know, having the docs is good, but it's not it's not it's necessary, but not sufficient for that because I think,
• 1:08:17
  Bryan Cantrill: right now,
• 1:08:18
  Robert Mustacchi: I don't wanna just show up and just be like, hey, here you go. Please read this, you know, a 100000 word dissertation, you know, 200000 words and, you know, come up with a summary and, you know, go from there. Because I think that I think that's, that can be very overwhelming. The same way, you know, this wasn't all written, you know, Rome wasn't built in a day. This wasn't all written in a day.
• 1:08:39
  So, I I think it's important also to figure out, you know, how do you kind of introduce these topics, kind of, you know, get to more and more detail. And then I think a lot of it is also just not being being really willing to if people ask for, hey, how does this work? You know, being willing to put together an ad hoc presentation, whether it's formally with slides or not on how how does this you know, what's going on. You know, this is, another example that is, you know, colleague was asking about, you know, hey. I'm trying to get more into some of the PCIe hot plug stuff.
• 1:09:09
  Like, can you help me understand how this actually all fits together with this? Because, you know, the PCI spec is long and, involved in not, you know, the easiest thing to read. So, you know, the answer to that is, you know, yeah, let's find a time that does this and, you know, get into it. I think that's that's partly how you do that and then helping make sure folks feel that it's okay to ask those questions too is equally important, which, you know, is always an ongoing effort to kind of build those reports because, you can't you can't necessarily know someone's struggling, so all you can do is try to be open and willing to answer stuff and, try to be helpful.
• 1:09:49
  Bryan Cantrill: Right. Yeah. We well, and I think I mean, just getting into to PCIe because, I mean, it's another total someone in the chat. PCIe is a plug. Yes.
• 1:09:59
  Oh, yes. I mean, another and it because it feels like, you know, this is true for all these domains where it is an entire universe of complexity. And, I mean, having, you know, helping people navigate it. And then I've always felt that, like, you know, you're so charitable with your own knowledge. It definitely, like, I think, I feel anyway, that you inspire people to be like, oh, yeah, I can learn this stuff too.
• 1:10:27
  Like, I can actually this stuff is actually learnable. This is, it feels like it's very dense and I don't understand it yet, but I just need to I, I can do it. I can actually learn this stuff, which is very inspiring because there's a lot to go learn. Do you want to talk a little bit about like the just as a very concrete example, I'm not sure if you want to take the the this recent T6 problem and or this recent algorithm problem and or both as I kind of I I just say another actually, no. Before we do that, actually, yes or go ahead?
• 1:11:03
  Robert Mustacchi: No. Thank you. Go ahead.
• 1:11:06
  Bryan Cantrill: Actually, let me back up because I actually I have a different question I wanna ask you. And then then we'll get to those. Because one of the things that I definitely appreciate about, again, the way you think about the system is, and I know other people appreciated it as well, is your ability to seemingly see around corners. And the number of times I will like hit an issue, and then I go into an old RFD of yours where it's like, oh my, like, you know, I like look under my chair and like, oh my God, Robert has already written a dissertation on this exact issue that I've grappled with and that I even read at the time, but didn't really appreciate. How do you, I mean, because I feel it's like a real, I mean, it's a tremendous gift for lack of a better word, to be able to kind of project a system into the future and anticipate some of the issues that we're gonna hit on a system that's not yet built.
• 1:12:01
  Do you have any tricks for doing that? I mean, what is that how do you do it?
• 1:12:06
  Robert Mustacchi: I wish I could claim work. I I wish I could omnipotence or something like that, but that that clearly, something like that.
• 1:12:13
  Bryan Cantrill: You make a lot of sense though if you did. I mean, like, you know, now would be the time.
• 1:12:17
  Robert Mustacchi: Yeah. I mean, I think, that's a good question. I think a chunk of this I picked up from just working with Keith over the years.
• 1:12:25
  Bryan Cantrill: Yeah. Keith's also very, very good at it.
• 1:12:27
  Robert Mustacchi: There's a lot there. But, I think this sometimes gets back to the earlier kind of code review is really trying to ask why. And really trying to understand how does this fit into the system? How is someone going to do this? Or if someone wants to do X, what else does that mean they need to be able to do?
• 1:12:47
  Or how does it work? For some of this stuff that we're seeing around, I think some of it's cheating that, you know, we've been thinking about for over a decade. Dogpatch is 14, that's 2025.
• 1:12:59
  Bryan Cantrill: Yeah.
• 1:13:00
  Robert Mustacchi: So someone's just had a long time to just marinate in the back of the head with just different experiences. And I I think part of that also is just a bit of just, paying attention to how are folks using things, how are operators using things? I don't know. I wish I had a better answer for you there so I could
• 1:13:21
  Bryan Cantrill: No, no, no, no.
• 1:13:22
  Robert Mustacchi: I think a lot of it is just trying to list a lot of it's listening, and then a lot of just digging and really needing to understand the low level details before you can kinda go and make the high level answer and how the 2 inform one another.
• 1:13:40
  Bryan Cantrill: Well, I also think that, like, this is where the the amount of time you spent in the details just really helps informs your engineering wisdom. I mean, just to give you a really concrete example of of your wisdom and how we benefited from it. Recently, we had a a Murata PowerShell issue, that Eric Austin and our team had done a terrific job debugging. And, Eric is like, I I think this is a control loop issue, and I think we're gonna need we're gonna get a firmware drop from Murata that fixes this. And I remember thinking like, wow, that is I mean, in some ways, that'd be great, and but this is a a problem that we definitely needed to resolve.
• 1:14:18
  Murata did a great job, resolved it. And fortunately, we had an a mechanism that we had not built anything on top of. We we had not built any of the software. But the the the the actual PowerShell itself had a mechanism to update its software. And that's something that you really believed in strongly when we were looking at PowerShell selection.
• 1:14:39
  I just think it's like it's an example of what I mean of like where, you know, not just evaluating kind of the artifact you have in front of you, but like what's gonna what could potentially go wrong with this where we would need to be able to update the firmware on a a PSU that is not, you know, it's it's not necessarily something that the first thing that a lot of people would think about. And, you know, you kinda have to have a certain kind of quantity of scar tissue, to really be able to think about it. And for many years, we didn't need it, you know, for many years. It would have felt like, well, I don't know, maybe maybe this firmware always does work. But no, of course, we do need it.
• 1:15:18
  And what we were able to build on top of it, it's all worked and we were able to upgrade that The firmware was terrific. But it was a great example of you you really saw around a very important corner.
• 1:15:29
  Robert Mustacchi: Yeah. And I I think that that often just comes back to, some let's say just scar tissue and experience. Right? Some of it was we've had times where we couldn't upgrade firmware or we couldn't get, you know, we didn't have the tools to or even get firmware updates, and so we've just been burnt by that, over the years. And so, I I do think there's also just, you know, if you really wanna think about how you approach and think about firmware and just, you know, just the same way it's like software and just that it's not this, while it can be very hard to understand just because of the, specifics of the way, you know, you're working with vendors, you know, obviously, if you just get a binary blob, it's hard to really get into there.
• 1:16:23
  But, you know, it's a thing that can fail and needs to be updated just like anything else. So, yeah. And I suspect that, you know, if we probably go back to dog patch, we probably have some commentary on the firmware that we were dealing with, and just the operational problems there and trying to figure out how do you get to a better model, you know, where even is all the firmware? If it's there, assume something's going to go wrong or if there's an EEPROM that has data, assume it, you know, it has to be flash, You know, we're gonna need to flash it and then, you know, once you start thinking about in some some portions, then it turns out that that's actually true in, you know, a lot more stuff. You know, there's nothing that's just specific to I I think a lot of it is also just trying to take what are the things that you learn from one area and don't just how can you apply them to others?
• 1:17:15
  And sometimes that'll be right, and sometimes, you know, that'll misfire, and maybe not be quite the right starting approach. But I think it lets you kind of ask other questions and helps you kind of think about that. So, you know, for some of us, it's like, you know, no one would say, ah, we should never be able to upgrade the software and the product. So Right. You know, I the same would kind of be true.
• 1:17:39
  It's like, well, if the only way I you know, someone comes back to you, you know, what's the service experience? Okay. Let's say we did have to do this. If there was no way to do online, to upgrade the firmware, the rectifier through software then that means I have to send a tech out to every DC of every customer, and that the pull went out one at a time and do this and that's just not you know, that works when your N is small, and it's, you know, a bad day for someone, but it really doesn't scale. So that's that's another big part of just, you know, where are these things that are you know, what would be the remediation if it fails?
• 1:18:17
  Bryan Cantrill: And just asking yourself that question for all for all these because I mean, as a result, like, we did I mean, the the fact that we did our own PowerShelf controller, which is not necessarily the first thing that I think other people would think about. It's like and, I mean, Murata to their to their I would say Murata, some other PowerShelf, manufacturers had different opinions about us doing that. And, it was a real tribute part of the reason that we have a great relationship with Mirada is because they were willing to accommodate the fact that we're like, no, we're chucking out your Power Shell control, we wanna do our own. And that was a very, very, very good decision because it'll it gave us control over and and not just the ability to update the firmware, but also that too now, but the just the level of observability into a part of the system that doesn't generally get that kind of observability.
• 1:19:05
  Robert Mustacchi: Yeah. And I I think what what helped is, again, you know, from the kind of RFTs or docs building up on top of one of another, I think it's something like RFT 82, which is the one by kind of operator design principles and facilities, for operation, you know, has something about firmware upgrade in it. So then as you're going out and kind of doing sending these questions out to different vendors, you can go back and say, okay. What are the things I need to think about from there? You know, what are the kind of the key things?
• 1:19:32
  You know, how does it reply to things? Even just not just firmware, but, you know, when a rectifier, we'll just pick on the PowerShell, fails, identify which slot it is, what serial number it was, where is it, and how does that turn back into just different features that you need, and how do you tie that into basically different operator stories? And then you know, that same thing is true of failing disks. Right? I think it's much easier for us to think about, you know, a disk fails, I need to pull it, I wanna blink the right slot.
• 1:20:07
  Well, the same is gonna be true of a rectifier. The same is gonna be true of a fan. The same is true of, you know, a transceiver. So, you know, there's a lot more similarity in some of these things even though there are differences.
• 1:20:23
  Bryan Cantrill: Yeah. So I'm gonna try to make r f d 82 public while we're in the podcast. So RFT 82 is not currently public, but, I think it's the it's a it's a very good one to, to make public, because that yeah, that that really, it just it it captures, I think, an important aspect of your own approach and your kind of you're you're thinking about the way we ask different questions of different parts of the system. Yeah.
• 1:20:52
  Robert Mustacchi: Check your, Steve. Yeah.
• 1:20:53
  Bryan Cantrill: As I
• 1:20:54
  Robert Mustacchi: and this isn't also, you know, there's a lot of different folks who write feedback on there, and there's a lot of different work there. So it's not again, this is another one of those things that's not just, you know, a single person doing it. You know? There there's you know, we're we're taking these ideas, you know, urging colleagues to kinda get feedback on them, explore it more, get different perspectives, make sure we're communicating it well, that, you know, that makes it better. So it's not, you know I
• 1:21:28
  Bryan Cantrill: think I've just made it too public. So so I Yeah. Yeah.
• 1:21:34
  Robert Mustacchi: You can find this,
• 1:21:35
  Bryan Cantrill: you can find the other Oh, it's a bubble. Exactly. I mean, definitely made something public. So, let's hope it's RFD 82. You say you may or may that is go may take a second for the this is, like, us.
• 1:21:46
  This is a terrific RFD API. So we may that may take a second, but, hopefully, people will be able to see that at some point real soon. Okay. So the and I think, Robert, a a82 is a great one to in terms of your again, your own kind of disposition. So maybe to kind of just dive back down into some of these specific technical details of things that you've been dealing with, like, recently, because I I mean, the the the the t six issue is a really interesting issue.
• 1:22:19
  I'm not sure if you if you're willing to get into the weeds on that one, but I do think of that as a very concrete and I how much do you know about this one, Adam? Have you have you,
• 1:22:29
  Adam Leventhal: only Robert was describing a bit of it to me, maybe a week or 2 ago as I heard a lot of t 6 buzz. But for, t 6 is the the NIC silicon that we're using in the next generation server. Right?
• 1:22:43
  Robert Mustacchi: Current.
• 1:22:44
  Bryan Cantrill: In the current generation.
• 1:22:45
  Adam Leventhal: Current generation. Pardon me.
• 1:22:47
  Bryan Cantrill: Yeah. So this is, in in Genworth.
• 1:22:50
  Robert Mustacchi: Yeah. So we do something a little weird, with the Nick. So there's something
• 1:22:55
  Bryan Cantrill: that should In stark contrast to the rest of the system where we are just, like,
• 1:22:59
  Adam Leventhal: down down the fairway, the normal
• 1:23:01
  Bryan Cantrill: Down the fairway. Exactly. It's like we also are a little weird with the NIC. So welcome to the oxide.
• 1:23:07
  Robert Mustacchi: So We've
• 1:23:08
  Bryan Cantrill: gone our own way.
• 1:23:09
  Robert Mustacchi: One of the things that we are really trying to make sure we could do, because the NIC has its own firmware, and configuration, which changes a whole bunch of different settings and things there, is we wanted to make sure we could validate slash attach that information. And we went through a bunch of different ways as a group to figure out, you know, how could we do that. And what we settled was that the NIC actually has its own manufacturing mode built in, so which is useful because some of this config file are things like, you know, what PCI you know, it's a whole bunch of information for the PCI E certies or, you know, describes things about how Ethernet should work. You know? What do you have I squared c for transceivers?
• 1:23:53
  Do you have different phis? Etcetera. So this information is all very critical for the NIC to work, but,
• 1:24:01
  Bryan Cantrill: you know,
• 1:24:01
  Robert Mustacchi: we didn't wanna have just a one time factory programming process here because what if we need to update it? What if something got wrong? You know, how do we deal with it? So we end up using this feature, the NIC called manufacturing mode, which basically, has the NIC boot out of an internal mask ROM. It doesn't enable Ethernet.
• 1:24:22
  It doesn't load any firmware, that would run on some of the cores internally. You know, it it doesn't do a whole lot of stuff, but it gives us access to the hardware blocks for the Nik's own, EEPROM and Spy Flash. So basically what this means is rather than basically creating a complex mux system to change ownership of these devices, like we have to for the host Spy Flash, here we basically read and validate this through the NIC in its manufacturing mode. So what this means is that every time the server turns on
• 1:24:59
  Bryan Cantrill: booted. Importantly, like, this means we are able to do this. We don't this is not the SP running in some context before the host CPU is up. This is the host CPU is able to do this while it itself is up.
• 1:25:10
  Robert Mustacchi: So the way we've rigged this up is that on the board, we have a GPIO strapped. So the system, the NIC always shows up in manufacturing mode. Then because we have power control over every device, we can basically validate all this and effectively, you know, we don't necessarily turn off all the power here. We kind of reassert the reset of the device and then basically boot it back up into mission what they call mission mode.
• 1:25:35
  Adam Leventhal: And so the the way that normal folks would program their NIC is like having a little, like, a Norflash or whatever kind of strapped to it that had this configuration information and it loaded up autonomously. Is that
• 1:25:49
  Robert Mustacchi: so, yeah, the way mission mode works is exactly that way, Adam. They basically have a little spy nor flash, a little easy prop, and it reads from that. And so, we have those same things. It's just that in manufacturing mode, it's basically bootstrap it. We just do it through the NIC.
• 1:26:05
  And, you know, on a normal on a normal PCI card, this is a little jumper that's there for the factory done once. And then you update this you update the you can update the firmware from the NIC while it's live. But,
• 1:26:19
  Adam Leventhal: right. And as as you were saying, like, doing this every time allows us to put known attested valid, you know, bits there as opposed to having this question of what's what's living persistently. Yes. And did, you know, some nefarious actor manage to wedge some bad bits down there?
• 1:26:41
  Robert Mustacchi: It also solves the factory and the egg problem. Like, how do you get the initial version on there? It's just like Right. There never it doesn't matter if there ever was something there or not. We just always thought we think should be there.
• 1:26:50
  Adam Leventhal: Right.
• 1:26:51
  Bryan Cantrill: Gotcha. And and this is like, oh, having the recovery path be the primary path, Adam. It's like this is so you don't you don't have this kind of path that you, like, rarely execute that when you needed to execute doesn't work. And it it also would because this is a real problem with, like, NICs have gotten really, really complicated. And it's like, oh, by the way, like, there's another computer over there.
• 1:27:11
  Like, SAR I mean, it it, you know, it's it's not really clear, like, who's in charge of the system anyway? And especially with, like, you know, smart NICs, they're being pretty overt about, like, no. No. I get I've got my own CPU. I've got, like, I load my load my own OS image, and it's like, no.
• 1:27:27
  No. No. You we we don't we want any of that. We want to actually be we want to have all this information flow in a way that it's attestable, that we know exactly what it's what we're putting out there, where we so we don't want that level of autonomy out of the system. We don't want this thing for a bunch of reasons, security, reliability, a bunch of things.
• 1:27:47
  So, I and I'm not this felt like a huge, win that we can use this manufacturing mode to do this. I mean, a really a real important aspect. I mean, it it kind of reminds me of the our the the fact that we we don't actually do warm resets. We only boot because we don't wanna have this kind of, like, state accruing in places.
• 1:28:09
  Robert Mustacchi: And it definitely simplified the you know, we talk about hardware software co design. This definitely simplified the electrical design. Definitely the last thing you want are more spy muxes and other things in the way dealing with voltage translation, you know, dealing with questions of even which spy port would we have to connect it to? You know, on what device? What way would data flow?
• 1:28:35
  So it it really simplified a lot of stuff, which was
• 1:28:37
  Bryan Cantrill: Yeah.
• 1:28:37
  Robert Mustacchi: Actually, this is a
• 1:28:38
  Bryan Cantrill: very good point, Robert, because it's like, I do feel that when we, you know, I'd always thought that, like, you know, there's kind of this back and forth over what controls a certain aspect of the system, whether it's the SP or the kind of the host CPU. And there's a kind of Conway's law thing that, you know, you got different orgs that are kind of dueling for power so you don't have an oxide. But, like, even if you are, like, you you have, like, total organizational harmony, just when you have something that is effectively dual ported, that is kinda owned by 2 things, you have all these, like, electrical issues that are really thorny about, as you say, like, level translation. There's just a bunch of stuff, that we wanna sorry. Just to underscore the importance of this.
• 1:29:20
  Robert Mustacchi: Yeah. It's I mean, we have a team that can has gotten it right time and time again, but also, you know, same way people joke, you know, the best software is the code you didn't write. No bugs in the code you didn't write. You know, there's no electrical problems in the if you don't put those things down. So, and sometimes they're necessary, but, you know, if you can get away with it, it makes it a lot smoother.
• 1:29:42
  So anyways, all that said, the way that the this kind of whole problem is that the mask rom starts up in PCIe, gen 2 it's basically it's hard coded in silicon to start as a gen 2 by 8 device, as opposed to a gen 3 by 16. And we had occasionally seen some failures, and I think it was really Josh Josh Glulow who insisted on you we kinda do some of this boot loop testing where we'd occasionally see devices that had a surprisingly occasional failure, to train the device in manufacturing mode. And if we came back to it later and tried to restart things or took another lap it would often work, or even if you tried to reset it after that it would come up just fine, but that first time on a cold boot, it wouldn't turn on.
• 1:30:41
  Bryan Cantrill: And so when you say it does not train, what does training a link mean? Who's doing that training and what does it mean when it fails to train?
• 1:30:48
  Robert Mustacchi: Yeah. So that's a great question. So, what we think of this is, so there's a complex state machine in the PCI docs that basically a PCI device goes through to basically have, a PCIe link come up. So when your operating system wants to go read or write from a register, that ultimately gets transformed into a transaction on the PCIe bus, which is a point to point link between a port on the CPU and the downstream device. So the NIC in our case.
• 1:31:22
  You may have switches or other things, you know, in more complex designs but you know, really you can think of PCIE as a bunch of point to point links generally between something on your CPU, which they may call the root port or an upstream port, and a device, often called the downstream port. And so link training is a process of basically figuring out, you know, shared, effectively what are the shared ways we're going to operate. So, for example, because PCIe is backwards compatible, you can take today's Gen 5 devices and put them in a PCIe Gen 1 board, and the link will train at PCIe gen 1. You know, you might have a root port that supports up to 16 lanes but you might put in a NIC that only has one lane, you know, a small one gig link. So it will do that.
• 1:32:13
  To make these links work at high speeds, is a very complex process because you have to figure out a lot of equalization and tuning, so that they can interact. Effectively link training is this process in this kind of large state machine that basically, hopefully ends with the link training, so basically successfully completing the state machine. And it's really done by the, PCIe device that you're plugging in, and really the PCIe root port, which is generally a whole bunch of hardware, in your CPU that probably itself has a secret core running stuff too that no one tells us about. So
• 1:32:52
  Adam Leventhal: Is this about figuring out analog kind of coefficients and constant values that make things flow with minimal errors?
• 1:33:00
  Robert Mustacchi: Yeah. I think that that's one part of it and, you know, that's definitely a large part of it and other parts of, like, the digital protocol communication. Gotcha. You know, what features can be used. So, you know, the PCI Seq has done a lot of work and, you know, it's a testament that we can, you know, PCIe has been backwards compatible back to its first release.
• 1:33:21
  Bryan Cantrill: It's amazing. Yeah. So that's so yeah. So alright. So that so that is what it means to train.
• 1:33:30
  So when we're not training occasionally in manufacturing mode, we're just giving up on this. Like, I devices, that's not working. I don't know why.
• 1:33:40
  Robert Mustacchi: Yeah. Yeah. Basically, we don't see a device come up. So there's a register in the root port, that says, you know, is there a PCI link established? And if you read it, it says no.
• 1:33:54
  Bryan Cantrill: There sure isn't.
• 1:33:56
  Robert Mustacchi: There sure isn't. And you're like, well, that's very sad. So there's a whole bunch of stuff that we were trying to do to figure this out, because, you know, we had some challenges in the t six initial initially. There's some oratum we found the hard way around,
• 1:34:18
  Bryan Cantrill: you
• 1:34:19
  Robert Mustacchi: know, it eating some double resets and some other conditions. So,
• 1:34:24
  Bryan Cantrill: double burst. There's a
• 1:34:25
  Robert Mustacchi: lot of, there's a lot of investigation that you know, we kind of split up and kind of took this in a couple of different phases, which is, I think, right as this was kind of kicking off, I was disappearing on vacation for a bit. But I worked with Nathaniel and Josh and Nathaniel started to basically go through a bunch of, you know, just different questions we had electrically. You know, is there a chance that this could be happening because because we don't see the device coming out of reset. So the one of the first things we kinda looked at and, Jose, you can correct me where I'm misremembering some of this was trying to bifurcate, you know, did the device assert it coming out of reset and did we ever try to even begin PCIe initialization or not? Because depending on the answer to that, that would take us down 2 very different paths, and the chip itself has a little pin that says that, that says it came out of reset.
• 1:35:22
  So there's a bunch of stuff we looked at there and secondarily, because of a whole bunch of the low level work we had done, to boot, we knew how to read out the state of the PCI state machine diagram, that the root port thought it was in. So what this meant is that we could go look at the root port, it has, you know, basically a ring buffer of the last 30 ish state transitions it's performed. So we can figure out what's it what has it been doing? What has it seen? As kind of a guide and you compare that against the PCIe spec and there's a more or less a one to one correspondence between those states and the state machine.
• 1:36:10
  Bryan Cantrill: And and how are we are we able to query that because of our own holistic boot? Because we who's got access to that? Is that that that that
• 1:36:22
  Robert Mustacchi: You could actually, anyone who has access to the system management network. So it doesn't, strictly speaking, need to have holistic boot, asterisks.
• 1:36:33
  Bryan Cantrill: Right.
• 1:36:34
  Robert Mustacchi: Sorry. So anyone can read this that is, whether you're, you know, in holistic boot or on the oxide architecture or just running a Lumos in general, on Linux or other systems, you can actually read from the system management network. Now knowing what to read and what maps to what, can be a little more challenging because the the the big gotcha, as we said earlier, is that the CPU is flexible. So it has a 128 lanes. Right.
• 1:37:02
  And there's a mapping, when you talk to that firmware between some of those lanes and what underlying hardware, resource they're using. I mean, so is Robert, like you're saying that, like,
• 1:37:16
  Adam Leventhal: anyone can read it, but holistic boot allows us to read it coherently and know the what the fuck is going on. Whereas, like, in in other scenarios, it's possible, but kind of academic. Right? Like, the
• 1:37:30
  Bryan Cantrill: They're very hard to make sense of it. Yeah. Yeah. I mean,
• 1:37:34
  Robert Mustacchi: you can with enough expertise, you can build up a mapping. That's just certainly a lot faster when we have a data structure that tells us, this is what it is for this. Like, this device, which should have the T6, read all these registers. It's certainly a lot simpler and certainly a lot easier because we also integrated, a bunch of register grabbing initially into the actual boot and training path itself. So we could actually just, on a debug build, for example, we'll just automatically collect a whole bunch of different, registers from the PCIe core and the PCIe, port, which corresponds to the root port just by by default.
• 1:38:15
  And so certainly that is where this is a lot easier because, that's not as straightforward to do outside of building it really into the system itself.
• 1:38:27
  Bryan Cantrill: Yeah. Totally. So you were able to go through that to figure out with why is this thing not training?
• 1:38:34
  Robert Mustacchi: Well, yeah. So that that we were able to eventually kind of first do that first kind of bifurcation and say, okay, the device is always coming out of reset. Then, you know, then we can go through and figure out, you know, why. Because basically we no longer have to go look at the we had a lot of logical questions but that ruled out a whole huge class and, I'm guessing Nathaniel's thankful was back in my court a little bit.
• 1:39:06
  Bryan Cantrill: Yes. But
• 1:39:09
  Robert Mustacchi: but yeah. So then when we started looking at this and, you know, we had this boot loop stuff that Josh had put together and we had a modified version that was grabbing out some of this register state at every loop. And actually, Angie, had actually already gone through and analyzed a bunch of it, prior to me coming back, to this problem when I was coming back from being out for a little bit. So once we kinda had the sense of that these were all here, these were all very similar, and they were all ending in a similar state, it got us, it was pretty suspicious because what we actually saw and so to understand how PCIe works, you always train a PCIe device to gen 1 no matter what. Then from there, you're gonna go to different speeds.
• 1:39:59
  And basically, you're part of the gen 1 negotiation, you're saying what else you can support. And then the device will go and, go to these higher speeds. 2 versus 3 is very different, and then 345 are kind of a different path. But we'd see that we basically got to gen we just successfully trained to gen 1, we would go down the upgrade path to gen 2, we would think, you know, we got to gen 2, and then as you're kind of going through the state machine and you're waiting for the other side to acknowledge, all of this in the like recovery dot speed substate or something like that, It'll just be like, Ah, well, we timed out. And we're going back to detect, which is basically the entry state that says, you know, start looking for
• 1:40:50
  Bryan Cantrill: something here. Yeah. Yeah. Woah.
• 1:40:53
  Robert Mustacchi: Woah. Well, actually, what was that's it wasn't quite nothing here. But, you know, the whole point is basically you go back to detect to try to see which basically see, is anyone there? We would see if someone is there and start going down the path again, anew, you know, you kind of do a whole fresh link training, and it would just stop replying at a certain transition point that was an indication to enter, what's called compliance mode. So normally compliance mode is something you only enter because you specifically request it because you're at one of these like PCIe interop tests and you're trying to prove something to you and the PCI, you know, basically to PAS compliance, PCI level compliance.
• 1:41:33
  Bryan Cantrill: Mom is here right now, so we need to I'm I'm I'm setting the mom is here butt bit. We need to actually do something slightly differently.
• 1:41:43
  Robert Mustacchi: But normally you're not supposed to enter it by yourself. Like, you know, there's only a very few occasions that enter it of its own volition. So we would decide that we were entering it and that was just kind of confusing, You know, especially because we had the ability to see what the host side saw, we had it was not very easy to go there was no good way to go answer what was the t6 seeing, and getting a logic analyzer on that was going to be very challenging because we needed to get that on there, we needed to get it on for all 16 lanes, and that's the one downside with the chip down solution is that, you know, it's a little bit hard to get the logic analyzer on there.
• 1:42:24
  Bryan Cantrill: Just a little bit.
• 1:42:26
  Robert Mustacchi: So, then, you know, we started doing different experiments. I don't remember all the ones I did, but the one that surprisingly worked was, you know, we kinda said, hey. Everything's training to gen 1 just fine, and then it's going to gen 2 that's failing. So what if we just stopped at gen 1? Yeah.
• 1:42:53
  Especially since we're only in this manufacturing mode for a little bit. And when we come back
• 1:42:57
  Bryan Cantrill: And we actually don't care about that throughput in this manufacturing mode where we're just
• 1:43:00
  Robert Mustacchi: trying to get gen 1 and gen 2 is not gonna, you know, we're really bound by the by the time it takes for it to talk over spy more than anything else. Right. You know, the actual bandwidth is not gonna make or break us. So, shockingly enough, that actually worked, which is both great and a little dissatisfying. But
• 1:43:27
  Bryan Cantrill: Dissatisfying that we don't know what's going on. What's you know, we also learned in this that, like, as it turns out this mode, like, we think this makes a lot of sense to to to use this manufacturing mode even though on every boot. But it may make, it's it's a little more unusual from Chelsio's perspective. And Yeah. For them, this For
• 1:43:47
  Robert Mustacchi: them, it's a factory only tool. So any issues that they might have had there, you know, it's really just factory programming. You end up needing to, like, just, like, hit the thing twice
• 1:43:57
  Bryan Cantrill: and
• 1:43:57
  Robert Mustacchi: it doesn't impact normal operation, like, you know, that's totally fine in a factory context. Not fine, in a product context. So for us, but,
• 1:44:09
  Bryan Cantrill: This was hugely important because this these were, I mean, when we in our testing in manufacturing, when these things were failing to come into manufacturing mode, we effectively we'd, like, would not ship that sled because we didn't know what the issue was. And we didn't know, I know I know that was that the issue was, I'm sure, very relieved to get it back into, like, the software domain because, like, the kind of the nightmare is, like, you've got a manufacturing defect effectively, where it's like, yeah, that board needs to be scrapped. And that was not the case, which is a huge relief.
• 1:44:43
  Robert Mustacchi: Yeah. And, yeah. And I think, you know, this is another one where, you know, we ended up there, but we definitely didn't start with just software only. And so it's really about, you know, working together, brainstorming different ideas, and trying to think about, you know, what could be going wrong. And then more so, how do you make hypotheses that you can disprove, one way or the other to help kind of narrow the solution space?
• 1:45:05
  Because otherwise, you know, Joker's wild on different things we thought about. And there was a bunch of other electrical stuff that we did investigate around, you know, were we seeing power, you know, were we not ramping power correctly, other things, you know, could there be something where we're not draining enough while we're taking this a 2 lap? You know, and some of that were easy experiments to go to go run, so we did did do some of them just to disprove that. And, you know, just the, you know, negative results actually are important, which is, Yeah.
• 1:45:39
  Bryan Cantrill: Well, this is a very important point you're making, Robert, is there's, like, holistic design also means holistic debugging. And you've gotta get when the system is not behaving correctly, you've gotta get ready to cross a bunch of different boundaries that, that are historically difficult boundaries to cross. And you've gotta be able to go really get a team together from across the stack to go brainstorm an issue.
• 1:46:00
  Adam Leventhal: Not to quibble, Brian, but I I'd say it enables holistic debugging because it's no. No. As you're as you're saying There's,
• 1:46:06
  Bryan Cantrill: like, no. Yeah. Yeah. Sure.
• 1:46:07
  Adam Leventhal: Holistic design, you know, if if this were a more traditional system, you'd chase it to the border, and that's the best you could do. Right? Like, you could That's right. You could you could you could sort of imply that there was some other component that that required now examination, but there was really no trivial way or or straightforward way at all to communicate across that those arbitrary boundaries. Like, in fact, that Conway's Law I don't know.
• 1:46:31
  Conway's Law is when it applies to an entire industry, but you've got that in industrial super form of Conway's Law applied to this aggregation that has become the the modern server.
• 1:46:44
  Bryan Cantrill: Is that like Conway's wall? Conway's Yeah. Mister Conway, tear down this wall.
• 1:46:51
  Robert Mustacchi: Yeah. And I think that's actually a good point because in this, you know, we talked about the software changes in the host, but we also looked at stuff in the service processor ring buffers, you know, FPGA register readbacks. We added captured a bit more state along the way, and the fact that we could kind of use that to rule out things, the fact that we can easily see what is the state of this ext reset l pin, is hugely helpful. Or the fact that, you know, if we had had to do something much more, you know, invasive, we could have communicated to the service processor over the communication channel.
• 1:47:31
  Bryan Cantrill: Right. I mean, I think having all those kind of options at our disposal really does allow us to I mean, as you say, Robert, it gives us the opportunity to go holistically debug. So Right. Yeah. That that is awesome.
• 1:47:43
  Robert Mustacchi: And I think that's actually, you know, the ultimate part of Dogpatch, taking it back there, is these different things actually working together towards solving our problems versus, you know, fighting with one another and kind of hoarding information. And, you know, because we're doing all sides together, you know, some of the things in Dogpatch that were saying, you know, it has it can only be the OS. Well, that's actually a reflection of the fact that, you know, that's because that was the only thing we could actually modify,
• 1:48:13
  Bryan Cantrill: and the fact that
• 1:48:14
  Robert Mustacchi: we can modify not just the service processor, not just the host software, but actually the board design itself, gives us a lot of different flexibility in how we can approach different problems.
• 1:48:30
  Bryan Cantrill: That's right. And, actually, RFD 88 is actually already public, which is great. So that one I don't need to make public because it's already public, but that's another one to check out, which is an RFP that that talks about exactly kinda how we think about who does what in the system, and definitely a a consequence of this again, this kind of holistic thinking. Well, Robert, I think this has been a very exciting way to kick off Robert Maslachie appreciation week.
• 1:48:57
  Robert Mustacchi: Yeah. Let's hit that one in the bud.
• 1:49:04
  Bryan Cantrill: But we really, really, really appreciate it. I this is it's so great to kinda, obviously, walk through the origins of food for money, Friday. I mean, clearly, let's not
• 1:49:13
  Robert Mustacchi: let's not just
• 1:49:14
  Adam Leventhal: Let's not forget what's important here.
• 1:49:15
  Bryan Cantrill: Yeah. Let's not pray the lead here, but future historians will be very pleased that they have finally discovered on the the the canonical origins of this cultural phenomenon known as food for Monday Friday. But the the just just getting kinda your perspective and kind of the the way you've done what you've been able to do, which is, with a lot of collaboration, a lot of hard work, and being willing to dive into new things all the time. And then also being able to come up and think about, you know, how does this impact the actual user of the system? And I love your thinking about that kind of operator empathy, as a guide in terms of a lot of what you've thought about in terms of the system.
• 1:49:53
  But really terrific. So, thank you very much for, for joining and for indulging us, and for, taking us up, down, and all around.
• 1:50:04
  Robert Mustacchi: No. No problem. Thanks for having me. It was my pleasure.
• 1:50:08
  Bryan Cantrill: Well, good stuff. And then, Adam, next week, we're we've got a European friendly time, I understand.
• 1:50:15
  Adam Leventhal: Yep. 9 AM, in a week. 9 AM Pacific. And we got Orhan, and we're talking about Ratatouille, our one of our favorite crates. I know we talked about lots of favorite crates last week, but one of our favorite crates for riding Tui is gonna be really fun.
• 1:50:29
  Bryan Cantrill: It's gonna be a lot of fun. So, join us next time. I am looking forward to having you. And Robert, thanks again for joining us. That was a, that was a great discussion.
• 1:50:41
  All right. Thanks, everyone.