Latest / Oxide and Friends / When Nine Nines Isn't Enough

Transcript

When Nine Nines Isn't Enough · Oxide and Friends

• 0:00
  Bryan Cantrill: Can you hear me? I hope.
• 0:01
  Adam Leventhal: Yes. You sound great, Brian.
• 0:03
  Bryan Cantrill: Oh, thank you. I I'm like, this why am I asking this? Because this would have been Adam has got now has twenty five minutes to coordinate prank that I have lived in fear of, which is he will connect, and we will act like we can't hear him.
• 0:16
  Adam Leventhal: Okay. Here's what I'm gonna say. First of all, that would be hilarious. Second, I think since the pandemic, that has felt too mean. Because it's cool.
• 0:27
  I agree. Yeah.
• 0:28
  Bryan Cantrill: It's cool.
• 0:29
  Adam Leventhal: I Yeah. Yeah. Yeah.
• 0:45
  Bryan Cantrill: You know, like, I have one job. You you that's Yeah. Funny Yeah. Because my family doesn't like to leave that last bit as an implication. It usually is the Wi Fi is out.
• 0:54
  You have failed as a father and provider as kind of one sentence. It's it's delivered as a short object. Or what, you know, my my now 18 year old, but but forever 13 year old mischief maker in my mind. You know, the thing he used to do is any new anything that we would get, he would seize onto the parental controls before we did and then lock us out of it.
• 1:21
  Jordan Hendricks: Me as well.
• 1:22
  Adam Leventhal: Yes.
• 1:23
  Bryan Cantrill: Yeah. We got Jordan here.
• 1:24
  Jordan Hendricks: I'm in the litterbox.
• 1:25
  Adam Leventhal: Yes. Hey. Let's do a quick roll call just to make sure we don't do, like, mid show bullshit. Although that's fun. Nathaniel, can we hear you?
• 1:34
  Nathanael Huffman: I hope so.
• 1:35
  Adam Leventhal: Yeah. That's right.
• 1:35
  Bryan Cantrill: How about It's like a Mouseketeers thing?
• 1:38
  Adam Leventhal: Yes. Exactly. Eric?
• 1:41
  Eric Aasen: Yes.
• 1:42
  Adam Leventhal: Boom. And Alan?
• 1:43
  Alan Hanson: Yep.
• 1:44
  Adam Leventhal: Cliff?
• 1:46
  Cliff Biffle: I'm not actually here.
• 1:47
  Adam Leventhal: And finally, RFK.
• 1:50
  Robert Keith: I'm here.
• 1:51
  Adam Leventhal: Wonderful. Okay. I just wanted to make sure we had
• 1:53
  Bryan Cantrill: You're calling Alan?
• 1:55
  Adam Leventhal: I did call on Alan, didn't I?
• 1:57
  Bryan Cantrill: Oh, you know what? I got okay.
• 1:58
  Nathanael Huffman: Yes. I
• 1:58
  Alan Hanson: I You did. Sorry, Alan. Alright. Sorry.
• 2:01
  Bryan Cantrill: K. No. I'm doing I'm doing the thing that you you told me I do, Alan. I'm so sorry.
• 2:05
  Alan Hanson: Mhmm. Alan, you remember me. Right? I worked for you?
• 2:08
  Bryan Cantrill: Yeah. Is Alan muted for everybody else? I can't hear him. Oh,
• 2:15
  Adam Leventhal: and Jordan. Jordan's here.
• 2:17
  Eric Aasen: Jordan's here.
• 2:18
  Adam Leventhal: Why don't you why don't you kick us off in a in a less Mouseketeers fashion?
• 2:23
  Bryan Cantrill: Mouseketeers. Okay. So and I and I know Jordan I Jordan, I'm sorry. Just just to accept the fact that Martin Burt's gonna be doing this for our entire lives. Jordan is like my generational check for millennials of like, do millennials know this?
• 2:35
  Ask Jordan.
• 2:35
  Jordan Hendricks: Jordan, mouse of pressure.
• 2:37
  Bryan Cantrill: It is a lot but you know what? But you know you deliver under pressure, and that's why I call you. The the the Mouseketeers, is that a reference that you
• 2:44
  Jordan Hendricks: I know.
• 2:45
  Bryan Cantrill: Yeah. That's it's interesting.
• 2:47
  Eric Aasen: Because what about Oregon Trail?
• 2:49
  Bryan Cantrill: Oh. What's that, Eric?
• 2:52
  Eric Aasen: Oregon Trail. Oregon Trail.
• 2:54
  Jordan Hendricks: That I I do know.
• 2:56
  Bryan Cantrill: Oregon Trail, yes. Mouseketeers, no. I mean, kinda makes sense because Mouseketeers was old when we watched it.
• 3:01
  Adam Leventhal: True. But the Zoomers now like, son has watched Mouseketeers, like a new
• 3:06
  Eric Aasen: kind of Mouseketeers.
• 3:07
  Bryan Cantrill: I know. There's gonna be I mean, there's a bit of a there's a bit of a generational pincer move here where x and Zoomers both know something that millennials don't. And it was mean, I I'm sorry, Jordan. I did this again to you. It was super satisfying when I had lingo from the kids, and Jordan's like, what is that?
• 3:25
  Like, what do you mean? What is Loki? Like, what do you why I'm like, okay. I'm not gonna explain it to you.
• 3:29
  Jordan Hendricks: That I age too.
• 3:30
  Bryan Cantrill: Yeah. Exactly. It turns out. See, you are a great spokesperson for your generation. I that's that's perfect.
• 3:36
  Yeah. Alright. Well, Julie beloved, we are gathered here today for this is a whopper. This one has been two and a half years in the in the making, amazingly. I think that's right.
• 3:52
  Cliff Biffle: I swear it feels like just twenty five.
• 3:54
  Bryan Cantrill: Yeah. I know. I know. This one has been a I mean, this has just been an issue for a really long time. And Alan, I wondered if you might be willing to kick us off because you were the the first person to actually hit this.
• 4:12
  And it should be said, you know, and and normally, we would not disclose customer names. In this case, I've actually already spoken to and about the customer, actually. So this was an issue that we first saw at Jane Street, which was one a perished early investor and early customer of Oxide. Really appreciate Jane Street a lot in both those capacities. I mean, it's immeasurably.
• 4:38
  And Jane Street was one of our very first customers. So this is in the Owen, I think I'm not sure. Did you walk down memory lane on this one? I think that you first encountered this in the, like, 2023. Is that right?
• 4:51
  Alan Hanson: Yep. Yep. September 2023.
• 4:55
  Bryan Cantrill: And which is I mean, we know we shipped for the first time, like, only slightly earlier that year. So we had we had kinda shipped in our our first rack. I mean, RFK, you're here with us. The the RFK did that that picture make you famous? I'd like to think that that was a that was Little bit.
• 5:16
  Robert Keith: Some people that I knew from the past saw it and then posted some other pictures of me doing other things that I that I was that I was otherwise that I was otherwise famous for. But
• 5:28
  Alan Hanson: yeah. A little bit.
• 5:29
  Bryan Cantrill: Hey. Hey. Can I have my moment right now? But this is the
• 5:33
  Jordan Hendricks: Picture of him shipping the rack.
• 5:34
  Bryan Cantrill: Shipping the rack. Yeah. It was a good one. And the it was RFK in in the lab coat wrapping the first rack to be shipped. The Internet comment that really struck stuck with me is and admittedly, if you look at the photo, I actually and, RK, you will call me asking you about this because someone said, I I keep I stare at this photo, and I I'm so nervous that he's about to eat shit and and and trip on the crate.
• 6:02
  And you look at it, and I'm like, it does look like he's about to trip. And I remember asking you about it. You're like, I don't recall doing a face plant at at benchmark, but maybe I did. I don't know.
• 6:11
  Robert Keith: Well, we've we've been awake for quite some time at that point, so it's possible.
• 6:14
  Bryan Cantrill: That's right. I know. I know. It was a it was a haul, but that was in, like, that's in 2023. So this is not that long after that.
• 6:22
  And we are at a a one of our very earliest customers. And Alan, what did you you were, I think, handling one of our very first updates when you noticed that. Is that right?
• 6:34
  Alan Hanson: Yeah. It might have been an update or I wouldn't look through the issue. And it says, you know, while we were debugging some other problem, we noticed that one of the sleds in the rack had enough time of only four days. It was
• 6:51
  Bryan Cantrill: Which is not what you'd expect.
• 6:52
  Alan Hanson: Now and everybody else had had almost twenty days, and there's this one sled that that's it seems weird. And, also, looking back to this issue, I'm I'm really surprised at how how few tools we had to debug things back then and how I mean, probably a lot of a lot of tools came out of this, but I think it just how crude the stone tools we were using then to try to debug this are quite amusing.
• 7:25
  Bryan Cantrill: Yeah. It was a long time ago And and they were so what did we have from that initial symptom? I mean, we just had the that this thing had bounced, but did we have anything else at that at that time? I'm not sure we did.
• 7:41
  Alan Hanson: Well, we had a a core file, which I think it it was always like we we know we know we wanna have core files, like, colonel colonel dumps from the very beginning. That was sort of like a minimum minimum product requirement. And so we we had a core. We were just it was very puzzling. We were like, well, you know, you can't really tell much from this.
• 8:07
  Bryan Cantrill: And and so what were the symptoms in the actual dump itself?
• 8:12
  Alan Hanson: It was complaining about IO to a pool appears to be hung.
• 8:17
  Bryan Cantrill: That's right. Yeah. And so we if IO and I Adam actually looked up the history of this because I'm sure this thing has its own colorful history. If ZFS cannot push IO for what is like indisputably a pathologically long period of time. I think it's like fifteen minutes.
• 8:34
  Right, Alan? It is a very long period of time.
• 8:37
  Alan Hanson: Yeah. Don't
• 8:39
  Bryan Cantrill: ZFS will panic. ZFS will be like, I like something very bad is going on. And do you do you know the do you remember the history? Actually, Alexa, I just looked it up. Do you happen to know the history of it, Adam?
• 8:51
  Adam Leventhal: No. I don't remember that. And I'm I'm a little surprised that I don't because or I I I mean, I used to work at a company where we were connected to a sand controlled by the customer, and it would go out to lunch for updates. I don't recall us bouncing on that, but I'm sure we did.
• 9:09
  Bryan Cantrill: Yeah. I I think the I and we I I remember every time that we've hit this, the that we had hit this, I think I because we I I know we hit this from time to time at Joynt. But every time we had hit this, it's very tempting to you're like, oh, we'll just eliminate that thing. Like, don't panic. Why would you panic?
• 9:30
  It's like, No. No. That's their, like It's been a
• 9:33
  Adam Leventhal: long time. Just to be
• 9:34
  Bryan Cantrill: fair, man. Time. It's like Yeah. This is it's been fifteen minutes, man. I mean, it's like and every
• 9:41
  Alan Hanson: time
• 9:41
  Bryan Cantrill: you wanna blame this this time out, the VDev time out, you really need to look elsewhere. You need to look inward in
• 9:46
  Robert Keith: the words of
• 9:47
  Bryan Cantrill: Lori And Green from Silicon we there's always something deeper a mess. So, Alan, that I think is the only data point that we had early is that we had died. And then we begin and I don't know how much you were involved in that kind of initial investigation of like, what else could this possibly be? And you know, where do they kind of the diagnosis kind of proceed from there? Do you recall?
• 10:20
  Couldn't go on the spot. Yeah,
• 10:21
  Alan Hanson: like, it was- I think we were looking for some FMA stuff, but we didn't we didn't have that yet. And we were yeah. I think it was like it's gotta be somewhere in the host OS, a bug in the host OS somewhere. But we didn't we didn't. Yeah.
• 10:41
  Bryan Cantrill: Yeah. And I think at some point, it our colleague Keith realized that in going through these crash dumps because the thing is that we actually when we saw this, we had actually seen this a bit along the way. And we had seen this earlier, but not been able to debug it. So now we had another example where we were seeing it. And from the the the crash dump, Alan, just to your point about we really wanna get a crash dump every time the system panics because there there's a lot of information in there.
• 11:15
  And one of the things that we determined in particular is that the the the pool is hung and cannot make progress. It does not seem to be on a ZFS deadlock. Deadlock. Does not seem to be it it looks like like this is happening at the at the IO layer, effectively the device layer. And then at some point, we kind of realize that the the device state appears to be as if it is uninitialized.
• 11:39
  It like, the device state all of the device state, all all 10 of these devices seem to have reset themselves.
• 11:47
  Alan Hanson: You're you're covering a lot in that. Because there was a there was a lot of shade thrown on the the drive firmware. There was a lot of prior pain points we had we're expecting to find again, and it it wasn't any of that.
• 12:02
  Bryan Cantrill: Well, I've yeah. I felt that, like, the drive firmware was not off the hook at that point, but maybe the the the it felt like the drive firmware, it felt like and, Nathan, I'm gonna if you remember some these kind of early hypotheses about it felt like it was likely to be on the AMD part and part of, because the it's hot plug control and what were some of the, do you remember any of the theories there, Nathaniel? This is, I know this is a long ago. We should we should all put one another under hypothesis.
• 12:31
  Nathanael Huffman: I mean, I know, like, that that issue happened at the customer site in September, like, the September 2023. And I kinda where a lot of our stuff picked up was when we we turned that into a quality issue, you know, manufacturing quality one forty. And that was done by Alan and Keith in early October. So, I think, you know, up until the point where Keith had really dug through all that stuff, I mean, you
• 12:56
  Eric Aasen: know, we had a lot
• 12:56
  Nathanael Huffman: of options on the table. But, like, when Keith wrote wrote the issue in early October, I mean, he was he was pretty suspicious of the hardware at that of our hardware at that point. Right? Because we knew that all 10 front drives had had PRST asserted. And so, I mean, like, he had done a lot of work to get to that point looking through all of those dumps.
• 13:18
  Bryan Cantrill: So describe PERSD, Nathaniel, because PERSD is the
• 13:21
  Nathanael Huffman: So PERSD is the fundamental PCIe reset, and so there is a a physical pin on the PCIe interface that, just is like a global reset to the device. It's kind of the first thing that happens when, when you online a PCIe device, or in this case, when you, interrupt it in the middle and and re online it. So Right. And software is a little unhappy if you reset the device after done all of your configuration because it expects that until you tell it, you're gonna take it away, And that that configuration that it set up is there for the duration of kind of its life.
• 14:04
  Bryan Cantrill: Alright. So we don't expect purse to happen. And I I Nathaniel, have we talked about the the need for the double purrs? I mean, if we I mean, I know people
• 14:12
  Nathanael Huffman: will be I think we did when we talked about t six.
• 14:15
  Bryan Cantrill: We talked about t six. Okay. Yeah. So we talked about the the need most things need should only should have exactly one burst. Apparently, one device needs exactly two.
• 14:27
  But this is not something you expect in a running system. You do not accept expect PCI resets.
• 14:33
  Nathanael Huffman: I mean, yeah, I think you'll see them on on hot plug k so, like, our u dot twos are hot plug capable. So you'll see that when you yank. But, like, you're supposed to tell the OS that you just did a surprise hot plug or unplug, and, you know, there there are thing and so then it knows to, like, go reinit the thing when it comes back. So there are some cases, but it at least in our system, and I think especially at that time, I'm not even sure we had all of the hot plug machinery, in place. We're really not expecting to lose all 10 of the disk drives at any one
• 15:03
  Bryan Cantrill: point in time. Yeah. And I think, Nathaniel, this gets us to actually a really important juncture. And we've described this kind of juncture before, but we and we've we've had a couple of these in the company where we're we haven't seen this very frequently. We don't know how to reproduce it.
• 15:19
  It's not reproducible at all for us. We have some information, but we would like to have a lot more information. And I think it's really important to not throw up your hands at that point, but rather to ask ourselves kind of collectively the question, what information do we wish we had? And then what do we need to go do to modify the system to gather that information? I mean, you know, I kind of call this a half measure, but honestly, that information, that data can be very valuable anyway.
• 15:53
  And we had a bunch of ideas for things that we wanted to go do to and I think that like, you know, Keith had had talked about PCIe hot plug event counters, for example. And because what you ideally want are things that can really disconfirm a hypothesis where if you know, if the PCIe hot plug event counters are correct and are zero, then you know, that and you see this again, you're like, okay, that's not it. That we we wanted to go add some missing reports. We did want to go follow-up on the firmware side of it to figure out what was going on there. Maybe we had a firmware update, go check that those bugs.
• 16:36
  We wanted to get some we did wonder if there would be other witnesses to the crime of the I mean, at this time, I would say, I felt that like this is a this feels like a that somehow we are doing something incorrect because what we're doing is so gutsy with respect to the microprocessor and that we don't have a UEFI. We don't have a Giza. We don't have UEFI firmware. We've done and we've talked about this ad nauseam that we we have don't have a bias, and we we do our own low level platform initialization. But you're always thinking like any platform problem you've got, like, you're responsible for.
• 17:17
  And my personal thinking, and the thing I'll be in others what you're kind of thought was, but like, this is a stray purse coming to every drive because something in the AMD firmware is confused because we haven't done something correctly. It's kind of what I don't know. Think the do you think that was kind of like the the the collective sense of where this thing likely was? It feels like that way anyway.
• 17:42
  Nathanael Huffman: I I mean, I think that was certainly possibly on the scene. I mean, when I'm looking at the issue, I mean, we started we started probing shark fins. So we have a little board, that's kind of a right angle up off off of that, off the main board that plugs into a sem connector, and then the u dot two is all connected to that. And that board has a a little bit of circuitry there and it manages the, like, purst and everything. So I, like, I don't think we're actually using, AMD's, purst on these drives on Gimlet.
• 18:13
  Oh, And so but we do have we have kind of a a little bit, a thin logic layer there. I mean, a couple of gates and we have the max fifty nine seventy hot plug controller. And so basically when we turn the hot plug controller on and its power is happy, it fires a one shot that bursts, and then, and and then we should be done. So if burst is asserting and we have some kind of transient glitch, then we're thinking like, okay, Something's wrong on the shark fins or the max 59 seventies on all 10 of them.
• 18:46
  Bryan Cantrill: And so then we we determined that the 59 themselves were not seeing alts. So we we kind of eliminated some of that stuff. And then somewhere along
• 18:58
  Nathanael Huffman: line We have also a unit in our possession for a while that even exhibits this.
• 19:06
  Bryan Cantrill: That's right. And the Because, like,
• 19:08
  Nathanael Huffman: the customer has one, so we need to get that back. And we have, like, it looked like I mean, Keith maybe had, like, two other possibilities, you know, serial numbers somewhere, that might have exhibited this in history if we look back at, you you know, if we look back at some of the things. But I think it took us a while to actually get a unit that we knew exhibited this.
• 19:31
  Bryan Cantrill: Oh, definitely did. Well, so a couple of things on that. One, we we had so we got this sled and at some point and and Alan, maybe it was with the the next update. It took us a while to realize that like this sled, this one sled at Jane Street, sled 10 was seeing this multiple times. This was not and it didn't seem and I I again, I think it was that kind of next update where Alan may have realized that, like, okay.
• 20:02
  This thing is this one sled is does seem to be seen as somewhat reproducibly. Some point, we had a sled that had seen this, like, great. Put it on the bench. And like, we're just gonna portrait on the bench.
• 20:15
  Alan Hanson: That was
• 20:15
  Jordan Hendricks: the one in Colo.
• 20:16
  Bryan Cantrill: Right. I think that was at the one in Colo, Jordan. Yeah.
• 20:19
  Eric Aasen: Yeah. Yeah.
• 20:19
  Cliff Biffle: So we saw this in Colo.
• 20:20
  Nathanael Huffman: Cubby 10. Cubby 10 from the Colo.
• 20:23
  Bryan Cantrill: Was it Cubby 10 again? Right.
• 20:25
  Adam Leventhal: It was.
• 20:25
  Bryan Cantrill: But which unfortunately, like, look. We're you know?
• 20:30
  Alan Hanson: That was some great theories. Got some great theories
• 20:33
  Bryan Cantrill: out of that that
• 20:34
  Alan Hanson: were not true. But, yeah, that was like
• 20:37
  Bryan Cantrill: Yeah. And there really shouldn't be any differences in the Cubbies, but that yeah. That led to some great theories. So, yeah, George or Jordan, that was in and when did we was that that was in late twenty three as well that we're seeing that in Colo? Think so.
• 20:48
  Yeah. Yeah.
• 20:48
  Nathanael Huffman: It looks it looks like we we had it on the bench at the like, 10/21/2023. So And we did hit it out of Colo quick.
• 20:58
  Bryan Cantrill: And we did not see it on the bench. We we couldn't reproduce it on the bench, which is like, alright. And then we it goes into the rack and we are able to reproduce it in the rack. It sees it again in the rack. It's like, okay.
• 21:17
  That's interesting. And then
• 21:18
  Jordan Hendricks: And that was in dog food.
• 21:19
  Bryan Cantrill: That was in dog food. And then at some point, food being our yeah. Colo being the rack that we have stood up in a data center environment with the idea and not just the idea, but the the actual practice. If you are an Oxide potential customer and you have spun up a silo on the on the Colo rack, you've seen this thing, but this is something that allows people to actually try the Oxide rack before they actually have to buy one. It's great.
• 21:43
  Much cheaper to spin up with the Colo rack. Like, doesn't cost you anything, but then so it allows people to really get an idea of what the experience looks like. So Colo rack has been the Colo rack has been super important for us. Dog food, I would say, is not in a data center environment, to put it mildly, Jordan. We are in a the dog food is is in the next room here.
• 22:04
  And let's just say that there are we we do try to sweep the leaves out of the office on a somewhat regular basis, I would say.
• 22:12
  Cliff Biffle: There's usually the normal number of animals in that room.
• 22:17
  Bryan Cantrill: Well, okay. This if anyone has done experiments with an office cat, it's something we've been thinking a lot about over the years. That the I mean, we are in the litter box because a cat has lived here, but not an Oxide cat. And, I mean, we have I mean, Jordan, mean, you've you surely have thought about an Oxide cat.
• 22:34
  Jordan Hendricks: Bringing my own cats?
• 22:35
  Bryan Cantrill: Or bring well, I wanna bring I mean, we've got we we have a well known mouser in our house, but she doesn't she seems to be a mouser on her own schedule. I mean, whatever. She's a cat.
• 22:43
  Adam Leventhal: An oxide cat would be well fed. Let's just say that.
• 22:47
  Bryan Cantrill: Do but you think an oxide cat okay. Do you but serious question. Do you think and also did the I was very bullish on the oxide cat until Steve anthropomorphized the mouse. Steve calls the mouse Stuart. That that yeah.
• 23:01
  Cliff Biffle: And then now we
• 23:02
  Adam Leventhal: refer to it as the mouse. We did try
• 23:05
  Alan Hanson: a hawk.
• 23:06
  Bryan Cantrill: We did we did try
• 23:08
  Robert Keith: a hawk.
• 23:08
  Adam Leventhal: Mice are
• 23:09
  Bryan Cantrill: Well, we
• 23:09
  Adam Leventhal: did famously solitary creatures.
• 23:12
  Bryan Cantrill: I think It's like snowball. Say is to say a hawk was tried. Like snowball.
• 23:19
  Nathanael Huffman: Well, that's true. That's true.
• 23:21
  Bryan Cantrill: The just for, yeah, for we did have a hawk that managed to become confused. We believe they have chased his own reflection. Doesn't really matter how it and caught on camera bursting through a a window in the office. And, Alan, you do you wanna I'm sorry. We have to I guess we have to do our our hawk interlude here.
• 23:41
  Because, Alan, you were the one who discovered the hawk. Is that correct?
• 23:43
  Alan Hanson: Yeah. And there is a strange final feeling you get when you're you're walking into a room and you hear the sound of large wings flapping. Like, it it is a very weird thing where you're like, oh my god. They're talking about the mouth. I mean, yeah, it was really weird.
• 24:04
  Adam Leventhal: No. No. I mean, it I've seen the pigeon in the BART station, so I
• 24:08
  Robert Keith: can only imagine that this
• 24:09
  Bryan Cantrill: is the Yeah. Pigeon in the BART station I okay. Good example. And I've had, like, there's been, like, a Starling in Safeway or whatever. That's, like, alarming.
• 24:17
  Pigeon in the BART station, that's, like, next level. Hawk in the office is,
• 24:22
  Robert Keith: like, okay.
• 24:23
  Bryan Cantrill: We're now unfortunately, like, we're we're now a couple trophic levels up here in a way that is that is a little off putting. The and so, Alan, how did you ultimately resolve the hawk in the that was that you opened the door and then but the hawk, I think, also I mean, the hawk really needed to go into the blue tent. The hawk had been concussed, I think. We think. Right?
• 24:43
  I mean, this is, like, actually fighting's effect.
• 24:45
  Alan Hanson: It But we home crashed through
• 24:47
  Bryan Cantrill: the window and then was knocked out on the table according to security footage for some period of time.
• 24:53
  Jordan Hendricks: There was glass, like, 30 feet from the window. Yeah. It was going fast.
• 24:58
  Bryan Cantrill: It was going fast. What did it did break through the window, and it did I mean, it was it was wild.
• 25:06
  Alan Hanson: Yeah. We we got it out. We have a giant behind dog food, we have a giant roll up door. And I opened the roll up door and kept walking around the office until the the hawk took the hand to the left.
• 25:19
  Bryan Cantrill: Right. So I mean I mean and to just to bring us back on topic, dog food may have hawk droppings in it, which really tells you a lot about the environment that it's in. So
• 25:28
  Cliff Biffle: It's really
• 25:30
  Robert Keith: well, it's kinda sad because mean, we're not sad. I feel I feel dumb because it's funny in our safety testing. There's a hole in the top of the rack for where the power whips go in. And they told us, well, you know, in a data center, something could get in there, and we thought it's, like, eight feet tall. Who's gonna get into the top of the rack?
• 25:48
  And they're like, well, maybe a bird. And we thought, wow. That's the dumbest thing I've ever heard. Who brings a bird into a data center? And then this happens.
• 25:56
  Bryan Cantrill: This happens. Exactly. Well, you never know. Yeah. Exactly.
• 25:59
  They don't do it. They should do the Hawk test down for regulatory compliance. So we but in dog food, we are able to reproduce it. It and I think it it took like, Andy, I wanna say, like, I mean, it took a while, like, ninety six hours to reproduce it, but it did reproduce. And and somewhere and we you know, I only have, like, Robert in the issue referring to a conversation that he and I had of and that we must have brainstormed about about trying to use some other witnesses to kind of crime.
• 26:32
  And one of the things that we had on the MAX fifty nine seventy, and and what you had made reference to, the MAX fifty nine seventy has a little some functionality where it can tell you the minimum and maximum of the voltages that it sees. So and, you know, I don't know what we thought when we were instrumenting it. But we thought like, okay, well, that's that's another kind of half measure we could take. And we could go look at the the the and it you know, these things all run at 12 volts. So we basically expect them to see like 12 volts kind of all the time.
• 27:06
  So we could go and I added some code to hubris to actually go track the minimum maximum and then got it wrong a couple of times as one does, but got it to the point where we could we could determine the minimum maximum. And then to me, it was pretty shocking, the results. And Nathaniel, I think I was looking up where we kinda discovered this in chat. I think we were all shocked by this. Is that fair?
• 27:31
  Were we expecting
• 27:33
  Nathanael Huffman: Well, I I think because it's the 12 volt rail, we sort of expected there to be a lot more casualty than we than we saw in this case. But, you know, I mean, I I think through some realized that the max fifty nine seventy turned out to be the most sensitive thing on that rail and everything else is like good down to, you know, I don't know, eight volts or seven volts, six volts, something like that. And so, if your 12 volt rail dips, it turns out that the discs are just the first thing to
• 28:11
  Bryan Cantrill: go. Well, that's it.
• 28:13
  Nathanael Huffman: I think that was super surprising. Right?
• 28:15
  Bryan Cantrill: I mean, like, in particular, just to say what we saw in the 5970 is we expect I I again, I wasn't I don't know what I was expecting, but I was not expecting to see them see a minimum voltage of eight volts. Right.
• 28:29
  Cliff Biffle: Yeah. I mean, this was a low enough minimum voltage that my initial reaction is like, oh, we did the unit conversion wrong.
• 28:36
  Bryan Cantrill: Cliff, don't you think it would be we did the unit conversion wrong again? Because I feel like I had gotten it wrong a couple of times. So it's like I
• 28:42
  Cliff Biffle: mean, I didn't I didn't wanna say it. That's alright.
• 28:44
  Bryan Cantrill: I I mean, this would have been an extremely reasonable thing. The to but amazingly, it's like, oh my god. And so that I I I feel was a a real wake up call in a couple of different dimensions. First of all, as you say, Nathaniel, I definitely didn't think the 12 volt rail would be dipping. Do wanna describe what the 12 volt rail is, Nathaniel, and kinda where it comes from on the machine?
• 29:11
  Nathanael Huffman: So so we have, on a system, we we take 54 volts in from the bus bar, and we go through something we call the IBC. It's an intermediate bus converter. So it does a 54 to 12, and then that is essentially the main rail from which all other rails are derived in our system. So we we do have a we have a tiny little rail sitting off of 54 that keeps our ignition system up so we can turn the IBC off and turn essentially turn the sled off. But like, the CPU, the SP, all of our all of our rails for everything in our in our power domains, all the high power stuff, all the u dot twos, everything.
• 29:52
  I mean, on Cosmo, like, the DIMMs are on right off of 12, that kind of thing. So, basically, all of the power of the sled goes through that intermediate bus converter, and we do a 54 to 12 and then do point of load supplies and build our power tree from there. So there's and and especially, I think the thing that was the most surprise to us was that the the CPU stays up and running. Like, the big CPU stays up and running in this case. So we're not losing any rails that make the CPU unhappy, which are all, of course, smaller than 12, but they're all derived directly from 12.
• 30:24
  Bryan Cantrill: It is and I and that's because the all these things are we are still within their so they're expecting 12, but eight is within their input voltage range. They're just like Yes. I don't know. It's weird, but sure. Like, you you asked me to like, what you've asked me to generate is three three.
• 30:42
  And my job is to generate three three, provided you're within the input range. And eight is within the input range. So I'm like, alright. Yeah. I'll just go do that.
• 30:50
  I mean, they it feels like that was true for all of these things. We're just like, everything was fine with it except for the thing that actually has to generate 12. And that's the 59 seventies. The drives need 12.
• 31:04
  Cliff Biffle: With the benefit of hindsight, there was another clue that we picked up at this point, but I don't think we realized, which is that while it's true that a lot of these DC DC converters are happy with their input rail down to eight volts, the amount of current that they draw scales as their voltage drops for a given amount of output power. So the fact that things were happy at eight volts implicitly indicated that they hadn't been at eight volts for very long, which we
• 31:31
  Bryan Cantrill: would Interesting.
• 31:31
  Cliff Biffle: We would subsequently learn a lot more about. Yeah. Right. Right.
• 31:37
  Bryan Cantrill: So and RfK, this is where, you know, you and Eric begin to swing in because now we have the bus converter is now the eye of Sauron that we speak of that is kind of always whipping around around here. The eye of Soron has now moved to what I felt was a very unlikely place, namely the IBC. And now we need to know everything about the IBC. And and Cliff, at some point, had done the because we we had like the black box on there, but that had yielded nothing. There's nothing in the IBC that indicated it had any fault.
• 32:12
  IBC is like, I don't know. Why is everyone looking at me?
• 32:15
  Cliff Biffle: Yeah. In in hindsight, we need to actually probably figure out why that is because that seems wrong, but you're right. Like, so this this this DC DC converter has an internal black box memory by analogy to the black box on a plane that they used to record crash data. And it's always recording a bunch of signals in like a circular buffer, and when certain events happen, it captures it and we can pull it out. So we go to it and we get this black box data, and it says, yeah, here's here's my Black Box and everything's fine.
• 32:40
  What are you talking about? Why are you talking to me?
• 32:44
  Bryan Cantrill: And the Well, also think Yep.
• 32:47
  Nathanael Huffman: I mean, before we before we like we also have a hot swap controller in front of the IVC. And and so, like, there's a whole another, like, power switch out there with a bunch of stuff. And we we also have ignition control of the IVC. Right? So, like, if if the IVC is turning off, I think our first inclination is maybe our circuits upstream are doing that.
• 33:11
  Right?
• 33:13
  Bryan Cantrill: Right. Right. And what did we I think we had already fortunately, could instrument that as well. And I think we determined that this is we're not actually seeing a dip on the bus bar.
• 33:28
  Nathanael Huffman: Right. Yeah. Right. And and RFK did a I mean, a bunch of surgery on these things because none of this stuff is like super easy to measure instrument in the rack. Because this you know, our our neat our neat, like, blind mate sleds that don't have any cables out the front mean that we don't really have any cables out the front.
• 33:51
  And it's tough to get scope probes and stuff in there. So RK did a bunch of work there. Yeah. And and I think a bunch of work moving stuff around too. Like, it was like, oh, we didn't learn anything when this happened.
• 34:02
  RK, can you move some probes around?
• 34:05
  Robert Keith: Yeah. Well, what was kinda cool about the whole thing is we had just purchased these brand new scopes, which had like 10 inputs. And that was cool because it's really eight. But so that way, I'd be like, well, you know, if we're gonna have to move this around, at least we can sample a lot of things every time I go back to take all this crap apart because it takes forever to do it. It I mean, just to the point of not having any cables, you remove all the drives, run all these long wires out there, what roll and plug it into the rack, roll the scope over to the front of the rack, kinda like organize all this crap in some reasonable way.
• 34:46
  Let it run for three days, and eventually you get something happen and then go back, pull it out, unhook all or unsolder all these different from all these different points and decide which ones you wanna keep, and then resolder a whole bunch of them and run it all over again. It took a long time.
• 35:03
  Bryan Cantrill: And were you I mean, I was definitely worried that we we would have a literal probe effect here that we would
• 35:09
  Alan Hanson: Sure. Oh, yeah.
• 35:10
  Robert Keith: Oh, yeah.
• 35:11
  Eric Aasen: I mean,
• 35:11
  Robert Keith: I mean, we we we ran it got to the point where we we're not gonna not gonna say what we found yet, but we were running out of things to look for. And so we got the near field probe out, which is like when when you've gotten to that point, you're like, oh, you're not gonna find shit. Like, you guys are so screwed.
• 35:29
  Eric Aasen: Yeah. Your script's the bottom of the barrel.
• 35:31
  Robert Keith: Yeah. Okay. So We yeah.
• 35:34
  Eric Aasen: That that's like Listen. You're Is there some random electromagnetic garbage that's around here that caused something nebulous and sad? Something?
• 35:42
  Robert Keith: Right. Maybe. But, like, with what resolution? And you're in this noisy environment. So, like, are you seeing just the shit that's around anyway?
• 35:51
  Like, you just really it god. You've really run out of options.
• 35:57
  Bryan Cantrill: And what was the near field probe? What what does that measure? How does that work?
• 36:01
  Robert Keith: E field. It's just like a little, we hook it up to the, SA, and you've got a little amplifier.
• 36:07
  Eric Aasen: So noise antenna.
• 36:09
  Robert Keith: Yeah. Right. Test lab.
• 36:12
  Bryan Cantrill: I I remember this theory. This theory is like, oh, well, maybe, you know, there's, like, someone has got a a power saw out in the street that is that is
• 36:20
  Eric Aasen: Maybe maybe there's a little EMP that we accidentally designed into the board somewhere.
• 36:24
  Bryan Cantrill: Right. Right. Did I forget to say you guys put that one? I did. Right.
• 36:28
  Eric Aasen: You know, air burst, nuclear weapon, whatever. This is
• 36:30
  Bryan Cantrill: usually That's how
• 36:31
  Robert Keith: far out we're trying to look. We're out of options.
• 36:35
  Cliff Biffle: Usually how you can tell
• 36:36
  Nathanael Huffman: we're coming
• 36:37
  Eric Aasen: out of ideas.
• 36:37
  Nathanael Huffman: I mean, looking at the pictures, I know we had had some early units that had had like the plastic left on the thermal pad on the IBC at the manufacturer. So like we stripped the heatsink on the IBC and destroy the thermal pad and replace the thermal pad to make sure, like, that, like, maybe it's it's getting too hot. And it was not.
• 36:59
  Bryan Cantrill: Yeah. And actually, you know, that's a good because we had a whole bunch of theories around, like, maybe this is a bad solder on the IBC. Unfortunately, determining if it's a bad solder, the IBC is is one of the the more difficult parts to put down from a manufacturing perspective. Gather, but maybe you can elaborate on that. And it's definitely a one way trip to determine if that's a bad solder or not.
• 37:21
  Cliff Biffle: Yeah. You think it's hard to install, try taking one off. Well, we figured out here.
• 37:25
  Eric Aasen: You micro need blowtorch to get that thing off. There's so many ground ties. Because it, I mean, it upwards what? A 108 amps? Somewhere around no.
• 37:34
  Yeah. About about a 108 amps. And so needless to say, there's a lot of ground ties and power ties on that thing to keep it from desoldering itself.
• 37:44
  Robert Keith: Right. Having to do it a bunch of times, I suppose we've learned how to do that. We've learned how to make the connection better. We've a lot of lot of ancillary learnings around this one considering an opportunity.
• 37:56
  Nathanael Huffman: What? We definitely don't wanna give up our one unit that reproduces the problem either. So like, we're not
• 38:04
  Robert Keith: cut like, we're not
• 38:05
  Nathanael Huffman: cutting this thing off until we have learned everything we can from the sled.
• 38:10
  Eric Aasen: And then I think we found more.
• 38:13
  Bryan Cantrill: Yeah. That's it. I think that's right, Eric. I think that it was about then that we began to realize, and in particular, I think that we going back through our own logs. And again, you know, this is this had been seen on one of the one of the first racks we shipped kind of, you know, when we were still very, very early.
• 38:28
  And every rack was an adventure. I mean, at some level, you know, every rack continues to be a different kind of adventure, but every rack was was really an adventure then. I mean, it was and we were just discovering new things all the time. And I think at some point, Angela went back through the logs and realized, like, we saw this before the sled shipped. And it it was a you know, things were just we didn't kind of connect the dots.
• 38:59
  Jordan, I think it was about then because you you added something to our the manufacturing test when we realized that we had actually seen this before it shipped. And it's like, maybe we can we will actually prevent ourselves from shipping more of these, whatever it is.
• 39:14
  Jordan Hendricks: Yeah. So I was working on manufacturing at the time. We're about to ship to our third customer. I think the third rack. We're doing a bunch of things to get it out the door, and it occurred to me that we could check if there any of the sites had this.
• 39:31
  Bryan Cantrill: Right.
• 39:31
  Jordan Hendricks: Because we only had two units, I think, at that point.
• 39:35
  Alan Hanson: And we
• 39:35
  Jordan Hendricks: were very stressed as a company about this issue since it wasn't really understood. There was very little very few patterns, if any, about when it would happen. And so part of the, like, debugging data that was added to hubris and, I guess, the humility, hubris ring buff accessing with humility was something we were able to just build into our rack testing. And it was extremely janky. It was parsing unstable output.
• 40:03
  Bryan Cantrill: Right.
• 40:03
  Jordan Hendricks: It's like really horrible code, but I was able to hack it together in, a day. And then I found that two or three sleds in the rack we were about to ship in a few weeks had it. Yeah. And then we just started running that as a part of our manufacturing testing.
• 40:17
  Bryan Cantrill: And that was an eye opener. Eric, to your point of like, oh, this okay. Well, good news. Not the the I mean, I feel like it was one of these where it's like, is it good news or bad news? I mean, I think it's good news.
• 40:28
  It's good news. We're seeing this with with a this is not like only on one or two sleds. This is on I think it was it was somewhere between 35% of sleds were were saying that somewhere in that that neighborhood. So okay. That's good.
• 40:44
  Also, the thing that was sleds that had never seen it didn't see it, which said this sounds like a tautology. But we in other words, if a if a sled was going to see this, it was seeing it in manufacturing. We were not which is extremely which is a great belief because we really first and foremost, we wanna not ship any of these things. Is now the time to mention that we could not take the sled out of Jane Street? Because there is another Right.
• 41:12
  We Well,
• 41:14
  Jordan Hendricks: there there is one more debugging rabbit hole we went down in manufacturing, which I think you asked me to do. So as you we wanted to see if we could kind of get reproduced within a particular time frame. So for a while, RackTest had this very chaotic notion of whenever it saw this issue, it would then reboot the sled. And we saw that it I don't think we really got anything valuable from that, but it was a fun little experiment that kinda pushed the bounds of what RackTest, which is our internal manufacturing test software, could do.
• 41:46
  Bryan Cantrill: Yeah. Yeah. And then we so we also one thing that was, like, I think terrifying sorry, terrifying me terrifying all of us is that you can't for it seems first of all, it's like, why can't you take the sled out of there? Just like you guys talk about how easy it is to, like, remove the sled. Like, I'll go remove it.
• 42:02
  You I'll just go walk off with it. It's like, no. No. No. Like, obviously, you can physically remove it.
• 42:07
  The problem is that there's a distributed system running on that. And we had we and, you know, say, Adam, ring chime for all your episode with Dave on we had to have the ability to reconfigure the rack to reconfigure around the dead sled. So I think we marked the sled as nonprovisionable there to prevent things from provisioning on it. And then we were like, okay. We we don't have a way of removing it.
• 42:35
  We rebuilt the ability to remove it. So we did actually add in an update the ability to reconfigure around the bad sled and then remove the sled. And we ultimately got that sled home, but we had to develop software to do it. Kind of non obvious software. Definitely non obvious software.
• 42:51
  Very sophisticated software. So we okay. So now we we're seeing this on a couple of different ones. And then RFKU and and I think Eric take one of these, and now it's just like under the knife with this thing. As you say, we're measuring power in, power out.
• 43:11
  And I dropped your analysis, the PDF of your analysis in in the chat so people can kind of see what we because I think this supports this is a the the part is a it's it's from it's a flex part flex bought bought. This is actually the IBC originates at Ericsson, which, you know, you still see artifacts of, like, acquisitions in, like, the product IDs that these things have. So this thing has a four character product or manufacturer name that comes through as Eric, e r I c. And I'm like, damn, some dude named Eric decided to like make himself. And they're like, oh, no, no.
• 43:51
  Robert Keith: Wait a minute. No. No.
• 43:52
  Bryan Cantrill: I think this is Ericsson is what they're this is not I guess it is some dude named Eric, but he's an entire company, not an actual entire legacy anyway. But we so this is the the Flex BMR four ninety one is the part. And, Arkie, do you wanna describe kind of like collecting all of this data in the document? Because it was very important to us to kind of go to Flex with like, here is the the the totality of our investigation. We don't wanna we don't wanna drop this on your lap by any means, but we need your help.
• 44:21
  Robert Keith: We took select shots of kinda like the damning problem. We found this droop.
• 44:28
  Bryan Cantrill: And how long is the droop? You should describe the actual length
• 44:31
  Robert Keith: of the droop because that I mean, it's only, like,
• 44:34
  Alan Hanson: you know,
• 44:36
  Robert Keith: twelve milliseconds about you could kinda and, really, what was interesting about the droop is that you can tell from the from the waveform, it's it's a controlled ramp, which is which is different than you would see in something like a discharge or just because the controller is doing this purposefully for some reason, and it's in such a fine time frame that it's like, you know, we it it exceeds the, you know, restart time frame for the part. So that kinda, like, is weird. Everything's like, what it's really hard for us to induce this by you know, maybe we have noise on the the reset pin or, like, something like that. Was something weird. And we we've we've also scoped all those things too.
• 45:27
  And the answer is no. We don't do that. Maybe maybe Ignition's got some odd noise that, like, I we we really kinda poked all around, and we I've kinda got it labeled out on all these different inputs what what we were looking at. And it it kinda really put it together where it's like, look. This something is up with the IBC, not us.
• 45:52
  Yeah. Your your logs don't even really make a whole lot of sense. So we we really need your help because
• 46:00
  Bryan Cantrill: I think it's you. Right. Well, it's it's this is, like, three milliseconds down, four milliseconds ish up. So and you I mean, this is the kind of the amazing thing about like this droop is like not very long. I mean, thing is I mean, you know, this is, you know, I I, you know, I had had one of my three children who will remain unnamed did very poorly on a math test.
• 46:26
  And in talking to that individual about their performance of math test, they pointed out that they got half of them right. And I'm like, well, that is true. You did get half of them right. Yeah. Yeah.
• 46:36
  Total optimist. You did get half of them right. But the the problem is the other half. And I feel like the IBC would be like, hey. I've been I mean, I am I've been doing this all the time.
• 46:46
  I'm at I'm holding this 12 volt rail. And, yeah, like, I took a I took quick break. Yes. I stepped out for six milliseconds.
• 46:54
  Alan Hanson: How many nines of uptime is this is this rail have?
• 46:56
  Bryan Cantrill: So this is know, Alan, it's funny because I actually counted it across because, again, this is only three percent of parts. And if you if you if you have the 3% of parts having this once every twenty four hours, which is about as aggressive as we ever saw it, it was often not that aggressive. This is about nine nines of uptime for the part. So it it comes out to basically exactly nine nines. And nine nines is not enough.
• 47:22
  It's not even close to enough. This we actually need we actually need zero nines. We need one one and a bunch of zeros on on the the the the uptime of this part. And I do think that, like, the the thing that was very chilling to me about all this again, now that we've got this investigation focused on the IBC, it was it was chilling because it it was symptomatically so distant from the IBC. And if we you know, Jordan, I was kinda thinking back to our past lives when we were at Joint running on Dell or HPE or Super Micro.
• 47:54
  If we saw a similar issue, it would not even enter our mind that it was a issue that it is so distant with the power rail. And I remember also talking to Flex about this, and they're like, well, we have you. We've never seen a problem like this in the field. And we were like No. How would you know?
• 48:14
  Because it was kinda hard for us to know. Like, this it's not like the part is broken. It's like, it's got nine nines. It's it is basically working all of the time for any human definition of working. This thing is working absolutely correctly all the time.
• 48:27
  The problem is for a computer's definition of working, it's not exactly all the time. And I don't so I think they so There's
• 48:34
  Robert Keith: a lot of pushback on that just because I don't know. This part's been in the field for a really long time.
• 48:39
  Bryan Cantrill: I exactly.
• 48:41
  Robert Keith: It and so it I think when you it's pretty typical for to get that kind of response where you they they're like, well, this is a production part. I mean, this is same thing we got with the t six. Like, this is in cards that we use everywhere. How can it have a problem? We're like,
• 48:57
  Alan Hanson: well
• 48:58
  Bryan Cantrill: But, like, billions of customers can't be wrong.
• 49:01
  Robert Keith: I mean, other than the whole report that we just wrote you, we were pretty upfront about it. Like
• 49:08
  Nathanael Huffman: Yeah. I mean, that's where I think I'm the most proud of the team is, like, when you look at this issue I mean, you know, when we saw that it was a it looked like a commanded power down, then we're like, You know, we're glitching with ignition and we're turning it off. And then, like, we ruled that out. And, like, we went we really did go and try to rule out, like, absolutely everything about our circuit and how we're using it to to prove that, like, all of the inputs are stable and the output is still drooping for this little short amount of time. And and, like and we did all of that before even engaging with flex because, like, you know, we're, surely this is our problem.
• 49:43
  Right?
• 49:44
  Eric Aasen: Right. And and tellingly, like, we did see a v in undervoltage reported by the IBC. But when we measure the pins of the IBC, the input to the hot swap controller in front of the IBC, the bus bar, any other point, it did not show up anywhere. And we're like Yeah. This voltage is rock solid going into here.
• 50:08
  Robert Keith: And we provided all that and the and the report from the black box about their under voltage events, it to the point of, like, what what does your black box do?
• 50:21
  Alan Hanson: I I
• 50:22
  Bryan Cantrill: think Yeah.
• 50:22
  Robert Keith: I don't know what's happening.
• 50:24
  Cliff Biffle: It's it's like it's a black box.
• 50:28
  Bryan Cantrill: And this is where we get a lit it does get a bit frustrating because we are telling Flex, like, there is there's a crime happening on the IBC. We think that there is software that runs on the IBC that you that you, Flex, know something about, that we don't know anything about. And this software is is a witness to the crime and trying to get them to take the same kind of thing that we had done. Like, can you give us a half measure? Can you give us firmware that we can run-in a debug capacity?
• 51:01
  And we knew from you know, and, this is where, you know, you're a very good read of these, like, of these manuals and documentation where they will make, like, an accidental reference to, like, a debug firmware load. And you're like, okay. I I know. I like there is a way of getting debug firmware on here. I know
• 51:15
  Cliff Biffle: there is. I would I would like one of those, please.
• 51:17
  Bryan Cantrill: I like one of those. They're like, don't know what you're talking about. It doesn't exist. It's like, no. No.
• 51:20
  It says it right here. But no. No. We can't do that. And that was frustrating.
• 51:26
  I I and I don't I think and then at some point, we get to someone who was actually very helpful at Flex
• 51:33
  Nathanael Huffman: Yeah.
• 51:33
  Bryan Cantrill: Who had worked with Ericsson. It was very sympathetic.
• 51:36
  Robert Keith: Well, it was hard to get ahold of him. It but it was funny, like, to that, we had been dealing with some difficulties on our calls. And eventually because this guy did not work on anything related to this project anymore. He had moved to, like, a totally different team. So they had to go, like, dig him out of the, you know, closet and be like, where you know, we need you to think about this project that you did five years ago or whatever.
• 52:02
  And when he got on the call, he I guess, it was it was interesting because he was like, oh, it's my my assumption, this is always working great. And we're like, yeah. We thought so too. Do you have this ability to run it in debug mode? I mean, it was he actually could run it in a raw output so we could see what it was reading from the ADCs, which was very nice of him to provide us.
• 52:26
  Bryan Cantrill: And which is great, but ultimately not like, not helpful, really. And Yeah.
• 52:34
  Jordan Hendricks: Where are we in time right now?
• 52:36
  Bryan Cantrill: We are in early twenty twenty four in time.
• 52:39
  Jordan Hendricks: Still. Okay.
• 52:39
  Bryan Cantrill: Still. Yeah. Still. Those are really long. Yeah.
• 52:42
  When is this podcast supposed to end? They will be well, we're gonna fast forward through a bunch of time, but we kind of get stuck about here because now we are so we are detecting this thing in manufacturing. We're not shipping it to customers. That's good.
• 52:58
  Jordan Hendricks: That's early twenty twenty four as well.
• 52:59
  Bryan Cantrill: Yeah. So that's good. Now we are beginning to over the course of 2024, twenty twenty five, one, we are beginning to hear from Eric Anderson on our operations manufacturing team like, hey, this bone pile is kind of like, it's getting a little thick out here.
• 53:17
  Cliff Biffle: Right. Because whenever we find this in manufacturing test, we basically pull the machine, set it aside for, like it's like a cryogenic storage of someone with a terminal disease. Right? Hopefully, future doctors can fix this.
• 53:28
  Adam Leventhal: But and this is a fully manufactured server sled.
• 53:33
  Bryan Cantrill: Yes. We're ready to go.
• 53:34
  Adam Leventhal: You know, tens of thousands of dollars.
• 53:35
  Jordan Hendricks: It was in a rack being tested.
• 53:38
  Bryan Cantrill: It's the only pull aside. It only sees it in the rack. We we we know that we actually don't see it ever on the bench.
• 53:43
  Adam Leventhal: And and for context, how expensive is this part?
• 53:48
  Bryan Cantrill: The IDC itself.
• 53:49
  Robert Keith: Yeah. It's probably $10. It's not that expensive. I mean, it's as far as POS goes, it's expensive.
• 53:54
  Eric Aasen: Like, 50, $100. I mean, it's Oh, you can go jump change compared to the cost of a server.
• 53:59
  Bryan Cantrill: Right? Yeah.
• 54:00
  Nathanael Huffman: Yeah. And somewhere in 24, we did figure out how to, like, cut these off and we sent some to Flex as well.
• 54:09
  Robert Keith: Yes. Yeah. We also figured out how to reflow them off at great cost.
• 54:13
  Bryan Cantrill: And we have an analysis from Benchmark that is actually a really great analysis showing us, you know, our favorite results. No fault found. But it was but showing us X rays and and there's nothing that appears to be a manufacturing defect. The good ones don't look different from the bad ones. It's kinda how I remember that.
• 54:38
  Yep. The which it's like, okay. So now we are in the we're kind of in a bit of a holding pattern where so and Jordan, now we have to like and and the well, Eric, you and Nathaniel and and co, RFK are working on Cosmo. And importantly, and I I mean, this turns out to be correct. We have got without this thing root caused, we have no reason to believe that it's something about Gimlet, our Milan based sled that is causing this.
• 55:13
  And we have to believe that it's present on Cosmo. We're using the same part on Cosmo. And people may be like, well, why are you using that? Like, god, don't use that part. Like, clearly use a different part.
• 55:23
  It's like, yeah, you get out here and try to find another IBC. Like, they're not a whole it it's not like IBCs are there's exactly as Cliff is saying. It's not it's IBC. How much is gonna cost? $10?
• 55:35
  You know, Jordan does always fill up with joy whenever an arrest development or reference made Exactly. Around
• 55:39
  Cliff Biffle: There should
• 55:40
  Bryan Cantrill: be a separate Jordan type part. But the it's, and this is a part we know a lot about, and we know this pathology we're kind of, and we've been able to kind of work around it. So it's not, it's not as simple as like get rid of that part and and go get another one. And there's a lot of things we like about this part. So we are disinclined to do that.
• 56:02
  But we now have to assume this is going to be present on Cosmo. And Jordan, I know this is where we began to think like, okay, so what does it actually take? Because the other the other, like, meanwhile, in like another planetary system, the operating system is seeing these drives spontaneously reset. What if we could actually handle that in software? What if software what if this weren't pathological?
• 56:25
  What if software could just say like, okay, well, is a surprise. This is the first time learning about this that all these drives have been reset. But we actually have a mechanism in the operating system, hot plug drives in and out. So Jordan, do you wanna talk about that? But that's this is not the path by which we normally do hot plug.
• 56:40
  So what was kind of in what's involved in that operating system support?
• 56:44
  Jordan Hendricks: Yeah. So basically, of the drives are experiencing a PCIe reset at once. And it's roughly equivalent to just yanking a drive out, but except for it's all 10 of them at once. This is the kind
• 56:57
  Bryan Cantrill: of thing that I think you yourself would say would just say rude. This is just rude. This this is very rude. Yeah. Right.
• 57:04
  Jordan Hendricks: And it's also for a very short period of time. Like, if you were to do, you know, hot plug, remove a disc, that's gonna at least be a couple seconds, whereas this is on the order of, like, a dozen milliseconds. But when it does happen, we do get a hot plug interrupt, specifically the data link layer being down. Oh, respond to that in the interrupt path and treat it as a hot plug event.
• 57:28
  Bryan Cantrill: I don't think I realized that, that we actually get a hot plug inter interrupt when this happens.
• 57:32
  Jordan Hendricks: Yeah. I mean, it's it's a PCIe interrupt that gets kind of pushed through that code path. But Yeah. And that's where those counters that I think Keith either Keith or Robert Moustakhi added get incremented as well is when that interrupt happens.
• 57:44
  Bryan Cantrill: Okay. So then what's involved so how hard are they to base? Like, alright. Is the operating system like, that's it seems like the what's involved in getting the operating system to actually not bork itself when this happens?
• 57:54
  Jordan Hendricks: So in Illumos, we have a a hot plug framework for devices that support hot plug, and we can kind of push devices through a state machine. So instead of I forget the exact labels right now. But, basically, we take the device offline and then set it back up again once it comes back. And this will kind of do all the necessary cleanup of failing outstanding IO, of making sure different connections, let's say, to the device are cleaned up. And it's very similar to Hotplug except for that it happens very fast.
• 58:28
  And another thing that was cool while I was working on this is getting to use a lot of the existing debugging work that was built into hubris. So I wrote a little task of my own. That was my first time trying anything in hubris to just droop the rail for me so that I could simulate this on any sled without needing one that, was exhibiting the behavior.
• 58:49
  Bryan Cantrill: Yeah. That's great. And then so you could induce this effectively?
• 58:53
  Jordan Hendricks: As often as I want. I even put, like, a little window of time to droop it for however long I wanted to see how it behaved in different conditions.
• 59:00
  Bryan Cantrill: Yeah. Interesting. And able to get this like, on the road to getting mitigated. Yep. And what were they because the I know it's it ended up being like, we were able to get it mitigated, but we still saw some transient issue.
• 59:15
  Is that right? What was the
• 59:17
  Jordan Hendricks: Well, so what happened was I was working on this. And then in the meantime, I think we found I don't wanna spoil the ending, but the real fix. And then this work got deprioritized, so
• 59:29
  Cliff Biffle: I was switched
• 59:29
  Jordan Hendricks: to something else. But yeah.
• 59:31
  Bryan Cantrill: Yeah. And so we the so the the the first half I thought the first half outlanded, maybe not the the the oh, okay. The first bit of it didn't land. The okay. So actually, that's a good segue.
• 59:43
  But that that that work is involved is the point. It's like it's not that work is not simple.
• 59:47
  Jordan Hendricks: It's very involved in the the fact that it happened so fast was a huge issue just because the hot work Hotplug framework isn't expecting things to happen that quickly.
• 59:57
  Bryan Cantrill: Yeah.
• 59:57
  Jordan Hendricks: Interesting. And it's, yeah, it's just not built for that level of speed.
• 1:00:02
  Bryan Cantrill: Right. We it's it's it's hot plug, not like Nano plug or whatever we call this. It's like, what we're not expecting this thing to be out for for on the order of milliseconds. Yeah. Okay.
• 1:00:12
  Well, that this is a good segue to, that and and now we we I've and the thing did we see this in Cosmo in development? Because I don't think we did. Do you know if we did?
• 1:00:24
  Nathanael Huffman: No. I don't believe we did, but we probably didn't have a large enough corpus running anywhere due to like some shapes and other things. We certainly have seen this in Cosmo manufacturing.
• 1:00:40
  Bryan Cantrill: Right. No. No. Oh, no. No.
• 1:00:41
  I know we definitely saw it. I as we went to scale, we definitely saw it. Yes. And we needed to be pricing it. And then somewhere in here, Flex reaches out.
• 1:00:52
  I hate this.
• 1:00:56
  Adam Leventhal: Just cold
• 1:00:56
  Eric Aasen: drop a PCN at us.
• 1:00:59
  Bryan Cantrill: I hate this so much.
• 1:01:00
  Robert Keith: This reality sucks.
• 1:01:02
  Bryan Cantrill: But okay. But I love this. This is so they so Flex comes back and says, hey, that issue you saw. Maybe you weren't the only customer that saw that. Maybe another customer did see that.
• 1:01:17
  And they have, they got a fix. And this I thought was pretty wild. Eric, do you want to describe the the kind of the issue that they found? And then it guided us to some mitigation that we'll get into that clip. I want you to expand on in a second.
• 1:01:37
  Eric Aasen: Yeah. So on these supplies, they're isolated. So there's kinda two power supplies in one. There's an auxiliary power supply that fires up to control all the controllers essentially. So it powers up the primary side controller, and then that starts the the power converter up, and then the secondary controller is powered on the secondary side.
• 1:01:58
  And it turned out that the auxiliary power circuit had a not particularly great margin on part of it, which meant that certain combinations of component tolerance, could cause the cause the the BMR IBC. Sorry. The IBC to, think it has an input voltage that's low when it doesn't. And then it does exactly what we saw.
• 1:02:30
  Bryan Cantrill: Hallucination is not just for LLMs in terms of that.
• 1:02:33
  Eric Aasen: No. No. Hard work can hallucinate with the best of them.
• 1:02:37
  Bryan Cantrill: And, Nathaniel, I do remember me, like, going to you being, like, with a little bit of, like, despair of, like, what do you it's what do you we need the tolerance. And you'd be like, no. No. The problem is like, you design these parts and if the you're just getting natural variation effectively in the parts that are within tolerance, But the tolerance of the entire circuit then makes it set susceptible to noise.
• 1:03:02
  Eric Aasen: It was the tolerance of the design. Yeah. Right. Yeah.
• 1:03:06
  Nathanael Huffman: Yeah. I mean, it it was kinda funny because the the change is basically, like, we we changed a lot of our analog front end on the circuit. I mean, you know, they kinda list all the parts that they change, and it was a bunch of parts. So I think, you know, once they found the problem, it looks like they probably went back and did, like, a Monte Carlo or something on the circuit to make sure that the tolerances that they're, you know, they're designing to and the and the parts they may get don't stack up in a really, unfortunate way. But that does explain why we would see units that like fell into that corner and would reliably do that.
• 1:03:40
  And then the bulk of, you know, our whole design body didn't have that problem at all. Right? So it's and it really is a design corner.
• 1:03:48
  Bryan Cantrill: Right. And and it just it it's a it's bad luck stacking, basically. And as you say, Eric, it's like the the the circuit doesn't have the tolerance. And this is so the good news is that in knowing this, they had some ideas for like, okay, but here is what should be able to mitigate it. And, Cliff, do you wanna and Cliff, I'm gonna per Grand Oxide and Friends tradition, I am making an RFD publicly available for the first time while we're broadcasting.
• 1:04:20
  Cliff Biffle: Oh, alright.
• 1:04:21
  Bryan Cantrill: The the the RFD six thirty that you wrote on how to mitigate this. And because now we actually they've got a proposal for us on how to mitigate it that is that is actually something that we can actually go do in the field, which is exciting. Do you wanna describe a little bit about about their proposed mitigation?
• 1:04:43
  Cliff Biffle: Yeah. So, I mean, the problem with getting information out of most companies, particularly companies over a certain size, is that, like, you're never talking directly to the person that did the original work. Right? Like it's it's gone through several people on the way to you. So we got this doc that was a PDF or a Google doc of a word doc that contained a screenshot of a word doc that contained a screenshot of a hex dump from an internal only tool.
• 1:05:12
  Anyway, so it took a lot of reading and staring at, but we eventually worked out when and incidentally, the mitigation was provided to us as sequence of undocumented hexadecimal numbers we were supposed to poke into registers because of
• 1:05:27
  Alan Hanson: course it was. So so I had
• 1:05:31
  Cliff Biffle: to reverse engineer it and we eventually worked out what it was doing, which was because of the component choices on the analog front end, the part, the IBC was particularly sensitive to deciding that there was a voltage droop on the input when there wasn't really. And it has a a very nice designed in feature to cut power if the voltage droops out of range. In fact, it's a programmable feature, which is great Right. Because what the mitigation was doing, it turns out, was programming that to stop doing that or Right. Or concretely to turn off the entire under voltage detector.
• 1:06:06
  Right. So that's generally a bad idea because you were using that under voltage directory for detector for something. But in our case, it's actually probably fine because the mitigation turns off that, but then turns on some other features that kind of paper over it and mostly ignore it. But more importantly, we had our own upstream under voltage detector in the hardware that we had installed in advance of the IBC. It's one of these hot plug controllers we mentioned.
• 1:06:32
  So what we wound up doing is working all of their numbers backwards, figure finding a couple of unit conversion errors because math is hard. I mean, I do unit conversion errors all the time. So and then figuring out what the right numbers should be and plugging them into the IBC, and then the problem appeared to
• 1:06:51
  Alan Hanson: go
• 1:06:51
  Cliff Biffle: away. And then that RFD describes the actual math, but it's probably not very exciting.
• 1:06:58
  Bryan Cantrill: Well, and and so the great thing about that is, like, this is something that we can now do in the field, which is great. We can now hey. That bone pile, we can now like, we can okay. This doesn't this sounds worse than it is. We can ship the bone pile.
• 1:07:12
  Now we can actually, like
• 1:07:14
  Cliff Biffle: Oh, or at least we can we can recharacterize the bone pile. Right? We can take those machines. We can feed them back through and figure out how many of them seem like they work now and how many of them turns out have some other problem that this was masking.
• 1:07:26
  Bryan Cantrill: Totally. Totally. And with which is great. And then we are and then importantly, we can and Jordan this is where you know, you you were making progress on your work. Be like, actually, if this is not gonna be an issue, we've got it's not like there are not other hot
• 1:07:42
  Jordan Hendricks: There's nothing else.
• 1:07:43
  Bryan Cantrill: There's nothing else to do. Yeah. Exactly. You're just like, I've got like 8,000,000 other like super hot priorities. So happy to kind of put this one on the back burner if we can work work around it.
• 1:07:52
  I thought one of the things that I thought was really interesting and I don't mean Nathaniel, Eric, RK, maybe I shouldn't be surprised by it. But once the Cosmo population got large enough, we started seeing it at a basically the exact same rate we saw in Gen one. And again, I mean, I know I'm just like restating what we had already thought, but it was kind of mesmerizing to see it in practice that it's like, what yeah. Wow. It's exactly the same.
• 1:08:17
  Right?
• 1:08:18
  Nathanael Huffman: Yeah. Well, that didn't change anything. Bell curves are bell curves. Right?
• 1:08:22
  Cliff Biffle: Well well, but this was actually a really important result. And also we had a case where we shipped a bunch of systems to a customer without having applied this mitigation, without realizing that we had forgotten to include the mitigation, basically. And and we started sealing machines failing, and we were initially terrified, but then it turns out they were failing at exactly the rate you would you would expect given our previous data on the population of the IBC. So Yeah. This was an unintentional experiment that showed that, like, no, really.
• 1:08:53
  The the rate is pretty much constant across the fleet. And as soon as we deployed the mitigation to that customer, the problem went to zero, which Right. Means we might have done it right.
• 1:09:04
  Bryan Cantrill: Yeah. Yeah.
• 1:09:04
  Nathanael Huffman: And I I think what was interesting there is in manufacturing, we were actually frequently to this problem. We were actually using the prerelease software. So we were manufacturing it. Then when we, I mean, for this for this customer, they were going to get the fix in their in the next software load before they actually deployed. But what we actually shipped out of the door with did not yet have the fix that we were running in manufacturing.
• 1:09:33
  So like Yeah. It it But it was neat it was neat to see, like, we did all of our manufacturing and everything worked great. We put back the old software that doesn't have the mitigation and we start seeing failures. And then we release the new software, and it fixes all of these again.
• 1:09:49
  Bryan Cantrill: Yeah. It was neat. Do you have neat in quotes there? I mean, was neat. It is neat now.
• 1:09:53
  It's neat now. There was a moment where because this was in a preproduction environment at a at a customer, but the it was there was a moment of sheer terror. Kudos. Shout out to Eliza hopping in there just starting to, like, debug these things and realizing like, oh, these are all like, this this is actually our old friend. Yeah.
• 1:10:12
  Nathanael Huffman: And it And actually this tripping caught, I mean, you know, Eliza will have to do another talk about all of her bugs that she found as a result of this at some point. But, you know, this tripping exhibited a bunch of other neat bugs.
• 1:10:26
  Bryan Cantrill: Yeah. And I, this was because we could actually observe this too. This was also a motivating example to to pull together some of our fault management reporting because this is a fault that the service processor can observe. And Cliff, I know you and Eliza did a lot of terrific work on codifying how we actually see these errors and propagate them into the control plane. And there's it was very helpful to have an embodiment of an error that was out there that as a kind of a motivating example.
• 1:10:56
  Cliff Biffle: Yeah. Absolutely. And, like, this is still a work in progress, all this visibility stuff. But, like, the amount of work that we are willing culturally to put into discoverability and visibility and tooling and, like, to try to make sure that we are at least failing in new and exciting ways instead of repeatedly failing in the old boring way has been really critical in getting on top of things like this.
• 1:11:23
  Bryan Cantrill: Yeah. Absolutely.
• 1:11:25
  Jordan Hendricks: I do feel like this bug is so so Oxide. Else, I've seen here, like cross cutting a bunch of different teams. Like, it it's been going on for, what?
• 1:11:37
  Bryan Cantrill: Two and a half years.
• 1:11:38
  Jordan Hendricks: Yeah. Yeah. And so much of the debugging was built on top of each other. Like, I don't know. It it blows my mind.
• 1:11:44
  Bryan Cantrill: It is kinda mind blowing. And then, you've got and just the I also thought and actually, maybe this is a the kind of a question for the the the Eric, you and and Nathaniel Norfke. I thought we were never gonna get this root cause from the flex side. I had kind of like, alright. We're just this is something we're gonna live with.
• 1:12:01
  Robert Keith: Well, they dismissed it. And you when you get dismissed, like, four or five times, and then they stop answering your emails, you're like, oh, I mean, I guess we're
• 1:12:12
  Bryan Cantrill: done here. Yeah. I've been here before. I I know what this is like.
• 1:12:16
  Nathanael Huffman: Yeah. We we plan to ship a bunch and, you know, as we ramp up, we're going to get a big pile of these and like, eventually the problem will become unignorable. And so, like, that that doesn't mean that there isn't a lot of pain for us in that process. But at some point, it's like, look, we have, you know, 50 of these or a 100 of these, and they're all broken. Like, what's wrong?
• 1:12:41
  And so I I, you know, it's a it's like a long view on eventually this will get figured out. But, you know, I expected we would get there sometime, but not without maybe more pain than than we actually ended up experiencing in the limit.
• 1:12:55
  Bryan Cantrill: And Eric, was that what you were gonna say as well? What is that that you had faith that we would ultimately
• 1:12:59
  Eric Aasen: Yeah. Basically, the problem will get big enough that they can't ignore it anymore.
• 1:13:04
  Bryan Cantrill: That's right. Yeah. Interesting. Interesting.
• 1:13:06
  Eric Aasen: What are the You're gonna only gas at me so much.
• 1:13:08
  Robert Keith: That's exactly what somebody
• 1:13:11
  Bryan Cantrill: You know, but I okay. So here's what I'd like to believe the RFK. I think that our because I do think that they were they were like sympathetic with our problem. I mean, I didn't feel like they're like and we got them to understand that this is a likely there is an issue that is not understood here. And I would like to believe that when this was hit elsewhere, they were primed for, okay, actually, you're there is another customer.
• 1:13:38
  And the yeah. Mean, can tell this is
• 1:13:44
  Adam Leventhal: is great cold comfort. You're like when someone with more money came along, they were ready to claim victory.
• 1:13:51
  Eric Aasen: I've been on the other end And of this I get like when you see things like you you go through all this testing and you prove everything works right and then somebody comes in like well you know your your thing doesn't work right it randomly drops out and you're like yeah sure it does. Right. Because I mean there there is some, like, self delusionment, you know, whatever you wanna call it that is just you can't believe found some you haven't found already when you've made, you know, however many thousands of these things they had.
• 1:14:28
  Nathanael Huffman: I mean, when you look at the size of what presumably their install base is for these parts, like, three parts isn't really that interesting. Right? And so, like, I mean, at at a certain level, I can't really fault anyone for saying, oh, that falls into, like, something weird, and they're only your three parts. But you just, like, time, we're gonna get to a 100 parts or we're gonna get to 200 parts. And, you know, then and we'll spend more with you because our manufacturing is ramping.
• 1:14:55
  And so, you know, we'll just have a a better seat at the table and, you know, anyway, I and, you know, I think they they recognized, when they did find the root cause that it might be interesting to us and pass that along. I think there was some yeah. And it was interesting to us. Right? And and I but I think they were there was some recognition that like, oh, this this issue looks a lot like your issue as well.
• 1:15:16
  Eric Aasen: So
• 1:15:16
  Bryan Cantrill: Yeah. I know which is great. And they then so we've got and then then actually it it generated a clip had to split, but it generated actually an additional complexity for us because the good news is we now have a fixed version of the part. But now we actually don't wanna have the mitigation on the fixed version of the part. And it's like, don't wanna be like, okay, god.
• 1:15:33
  This is really make, could you just not fix this thing? It's really making our software complicated. It's like, no. No. No.
• 1:15:37
  Of course, we want this thing to be a better product in part. And because it and, Nathaniel, you and Eric looked at the circuit and were a better judge of this, but it seems like the circuit is like, it wasn't just like changing one part. It seems like they made it like much more robust. Is that a is that a fair read?
• 1:15:56
  Eric Aasen: Yeah. They they thoroughly rehashed that input. And it wasn't it wasn't just one component. There were, like, 12.
• 1:16:04
  Alan Hanson: Right.
• 1:16:05
  Eric Aasen: And they were not small changes. Like, there were significant changes done there.
• 1:16:11
  Bryan Cantrill: Right. Right. And I'm like, hey, I appreciate it. Like, the the over engineer. This was under engineer.
• 1:16:15
  Let's over engineer it. And then but then I'm always worried about like, okay, well, now they're gonna get the revision number wrong on the part or, you know, you've got all the other issues, but we've got.
• 1:16:26
  Nathanael Huffman: I mean, the mitigation is safe too.
• 1:16:30
  Bryan Cantrill: The mitigation is safe.
• 1:16:30
  Nathanael Huffman: The mitigation is safe. So if we did accidentally apply it to a new part, it it isn't that big a deal.
• 1:16:37
  Bryan Cantrill: Yes. Yeah. That's right. That that is fair. And the the cost, don't think did change.
• 1:16:44
  There's fortunately Sam Altman is not threatening to buy 40% of the world's IBCs, so we don't yet quite have the the the same run on on issues. Exactly. There's there's no in there. It is a very hot part, though. This this is definitely the source of of quite a bit of heat.
• 1:16:59
  We did learn it in an in an unrelating manufacturing mishap that install the heat sink on this thing. The sled is definitely gonna overheat. So definitely heat sink is important to this one. But yeah, Jordan, as you say, this is this is kind of as I mean, we've had a lot of bugs that are oxide bugs. This one feels very oxide.
• 1:17:17
  And to me was also the thing that again is so chilling about this is this is the kind of issue that when you have on a a machine where you don't have this kind of vertical integration or holistic control, this is just like bad stuff happened. A ghost was in the machine.
• 1:17:34
  Adam Leventhal: Yeah. Brad, I was gonna ask. So did you think about, like, you were a joint. Oh, yeah. One of the boxes that you were buying experienced this problem.
• 1:17:43
  I mean, you'd just be stuck, wouldn't you?
• 1:17:46
  Bryan Cantrill: Oh, totally. Stuck and in it's stuck in stuck in the wrong spot. Stuck looking under the wrong streetlight in the wrong city. Yeah. You know, for the you're because you'd you'd be like, okay, this is like a driver issue or this is a and I think that like you are ultimately the path you take is the one that Jordan was on in terms of working around this in the operating system, which is probably still not a bad change to do.
• 1:18:11
  I mean, Jordan, I think it's something that you would like
• 1:18:14
  Jordan Hendricks: I would like to finish that work.
• 1:18:15
  Bryan Cantrill: Just like I just like to get that on the record. Yeah. I think and we should. Right? And it will make the system more resilient to have that.
• 1:18:23
  But we also shouldn't have I mean, it is this could have been worse. Right? I mean, this could have this That's pretty bad. No. But in terms of, like, the droop could be could be actually lower and could affect more components in the system.
• 1:18:38
  Yeah. It could be longer. Cliff had the point about the about about current and hitting a some of the current limiters means that this is you don't wanna have a droopy IBC is what I'm trying to say. Like, we don't have a droopy IBC.
• 1:18:50
  Jordan Hendricks: If that had happened, it would have been more likely that Flex noticed it, though. Maybe. Maybe that's why.
• 1:18:55
  Bryan Cantrill: Yeah. Yeah. It might have
• 1:18:56
  Adam Leventhal: been harder to ignore. Like, it maybe this was just in this Goldilocks zone of, like, sure it's bad, but, like, it's how bad
• 1:19:03
  Bryan Cantrill: is it? And it's it's not it the answer is, like, it's not that it's nine nines of
• 1:19:08
  Cliff Biffle: Yeah. Exactly.
• 1:19:09
  Adam Leventhal: It's not that bad. Complain about nine nines.
• 1:19:10
  Bryan Cantrill: Come on. Yeah. Great. Sorry. I got it wrong one, you know, one one billionth of the time.
• 1:19:14
  I don't know.
• 1:19:15
  Eric Aasen: Oh, and the other thing is we run 54.5 volts, and there are other people that could be using this part that might run 50 volts or something a little lower, which might make it a little more more likely that this happens.
• 1:19:30
  Bryan Cantrill: Right. Right. Yeah. Interesting. Yeah.
• 1:19:32
  Exactly. And certainly, it's it's great to get this root caused. I do think it and it's also to me a, you know, and Adam, at some point we need to do an entire episode on the cash parody error at Sun.
• 1:19:47
  Adam Leventhal: Yeah. That'd be fun.
• 1:19:49
  Bryan Cantrill: Because I, we need to, that's always on my mind where the the failure of Sun and for those of you who are long time Sun customers whose eyes begin to twitch when the e catch parody error is mentioned. This was suffice it to say a it merits its own episode, ring chime for future episode, I guess. But where there were a lot of things that went wrong, but Sun also didn't handle that very well in that we did not connect the dots quickly enough. And we tried to kinda we as a company, everybody tried to kinda talk ourselves out of the the the the kind of individual instantiations of that. And I never I I wanna make sure that every failure is sacred even when we don't understand it, which I think is that actually that's when it's most difficult is when we don't understand what has happened and yet we still need to hold on to this information so we can connect the dot later.
• 1:20:44
  It's hard. It's just it's it's culturally hard. It's it's hard when you've got so much going on. But I think it's a real testament to the system that we've built that we have been able to find these I mean, because the other thing the other thing I think it's important, Adam, is like part of the reason that this issue stood out is because we've got such a robust system. I mean, you know, I think people are always shocked at the very small size of our double e mechanical team, but they've done such a terrific job with such a real we we have such a reliable thing that we are able to really and I think that in other systems, there are so there's so much that's unreliable that you you're kind of having multiple organ failure all the time.
• 1:21:29
  And it's really hard. And, yeah, it's very hard to diagnose these issues.
• 1:21:33
  Robert Keith: I mean, gotta give it to y'all for the instrumentation, though. I mean, we can put it all in there, but if no one pipes it up, it doesn't really mean much. So
• 1:21:45
  Bryan Cantrill: Yeah. Well, it's got it it hey. You know, even though our colleague, Megan, says that she vomits on her shoes every time I say it, teamwork makes the dream work. I I guess I guess Jordan either hadn't heard of that one or I can't tell Jordan is laughing derisively at me or
• 1:22:02
  Jordan Hendricks: I'm laughing at Megan vomiting on her shoes.
• 1:22:04
  Bryan Cantrill: Oh, because he me, we can both visualize that. Yeah. Yeah. Yeah. Exactly.
• 1:22:07
  Adam Leventhal: That's right. That's right. That's right.
• 1:22:08
  Bryan Cantrill: But this is a good one. Well, thank you to you all. And Adam, you'd asked like if you needed to do any read up, you could see why you mean that. Like you could just be on this one. You could just kinda strap in.
• 1:22:20
  Robert Keith: Absolutely.
• 1:22:22
  Bryan Cantrill: Although I we should go look at the the history of the Vita. The it was ultimately that that Vita of time out that that was the that told us without the VDF time out, I'm not sure we would have seen this honestly. Or we would it would have been very differently. So the oh, I have to look at the history of that one. Well, thank you all for joining us.
• 1:22:42
  Really appreciate it. Thank you, Nathaniel and and RFK and Eric. Thank you, Cliff. Thank you, Jordan. Thanks for thanks, Alan, for being the the the initial first responder as as Alan is so frequently.
• 1:22:53
  First one on scene and really appreciate everyone. I think, Adam, we are out for the next two weeks, I believe. Correct?
• 1:23:01
  Robert Keith: Two weeks. Yep.
• 1:23:02
  Bryan Cantrill: Two weeks. And then so we'll be back in three weeks. We got a lot of great episodes that we've got germinating. If you have ideas for episodes that you would like to see, please let us know. I think we love your suggestions as well.
• 1:23:18
  So definitely throw them in there. Clearly, we are we've got our own content generation over here at the computer company. The exactly. Hit hit that subscribe button, Jordan. That's right.
• 1:23:33
  Adam Leventhal: Well and and drop a drop an issue in the GitHub or or show up
• 1:23:37
  Bryan Cantrill: Great idea.
• 1:23:38
  Adam Leventhal: Discord and let us know. But if you have an idea, sure.
• 1:23:41
  Bryan Cantrill: Yeah. That's a great idea. Open open an issue on the Oxide and Friends GitHub. That'd be that's actually a great idea. You know?
• 1:23:48
  And I I'm sorry that I said actually, I know I I really don't
• 1:23:51
  Adam Leventhal: like It's people to fine.
• 1:23:52
  Bryan Cantrill: Okay. Or will you edit it out or will you I you know?
• 1:23:56
  Adam Leventhal: Who knows?
• 1:23:58
  Bryan Cantrill: Who knows? Only the editor knows. Like, well, thank you in advance for doing whatever you do on that one. And thank you all, and we will see you next time. Thanks all.