Latest / Oxide and Friends / For Want of a Resistor

Transcript

For Want of a Resistor · Oxide and Friends

• 0:00
  Adam Leventhal: Bryan said he was ready and then I
• 0:01
  didn't respond. Well.
• 0:03
  So
• 0:04
  Robert "RFK" Keith: Alright. Let's look at him.
• 0:06
  Adam Leventhal: Yeah. Waiting to be invited up.
• 0:08
  Bryan Cantrill: You know? I just I I I just wanna be wanted. You know? There's there's nothing wrong with that. I just wanna be invited in.
• 0:14
  I'm not gonna invite myself in to your, you know, wanna let a little bit of decorum. Nothing wrong with that.
• 0:19
  Adam Leventhal: Good song. Makes sense.
• 0:20
  Robert "RFK" Keith: Yeah. I want you.
• 0:22
  Bryan Cantrill: And there
• 0:22
  Robert "RFK" Keith: you go. Me.
• 0:25
  Bryan Cantrill: And I got a I got a guest in the litter box today.
• 0:29
  Adam Leventhal: Who's who's with me?
• 0:30
  Steve Tuck: I heard someone I heard someone talking about clock buffers and
• 0:33
  Bryan Cantrill: Did someone say supply chain?
• 0:35
  Steve Tuck: Come on. How do I resist
• 0:36
  Bryan Cantrill: that? Exactly.
• 0:37
  Steve Tuck: I may I may have kicked the door in rather than a surprise guest.
• 0:42
  Adam Leventhal: And for those of you who don't recognize the voice instantly, Steve Tuck, CEO of Oxide Extraordinaire.
• 0:49
  Steve Tuck: Who Supply chain assistant.
• 0:51
  Bryan Cantrill: A supply chain assistant? Yes. Who I mean, Steve I think actually part of our conversation today should be Steve going through the different social venues that he has inappropriately brought up PCIe clock buffers to people just trying to make small talk, actually.
• 1:09
  Scott Tagwerker: Again. It's like, look, I hate to
• 1:11
  Adam Leventhal: ask, but do you have, like, in your pockets or, like, back at the
• 1:14
  Bryan Cantrill: I'm not sure he ever he ever prefaced that with, hate to ask. I'm not actually like, I don't remember anything like that, actually. That would've been I think actually someone would've liked that actually. That that degree of self acknowledgement would've been welcome. No.
• 1:26
  I think it was much more likely, I'm gonna ask, even though you're sending me social cues that it's that your offer to help was actually not in earnest.
• 1:34
  Steve Tuck: Yeah. Sir, this is a Wendy's.
• 1:36
  Adam Leventhal: It's like the
• 1:36
  Bryan Cantrill: Yes.
• 1:37
  Adam Leventhal: When the ball when the ballers guys were on, asking if anyone knew any pitchers. So same kind of thing.
• 1:43
  Bryan Cantrill: Similar vibes? Very similar vibes. Yeah. Yeah. I'm I'm looking for a left handed PCIe clock buffer if you've got one.
• 1:51
  But, Adam, before we do, we just well, a couple of things. One, I gave this keynote at ResistorConf last week, which is great. There's some reaction online to some of the draft names of the company. And then you, in an attempt to show that there were real veracity, there's real veracity to what I was claiming. You posted a screenshot of a text message from me.
• 2:18
  Adam Leventhal: Yes, true. And
• 2:20
  Bryan Cantrill: I, you know what I was reminded of? I was reminded of when we had the episode where, we elected to go in to how to castrate a bull and the Dave Lightman fan blog. Oh, yes. Oh, we're doing that now. Oh, okay.
• 2:36
  I didn't realize we were going to, oh, that that water on my knees is the Rubicon that I guess we're crossing. Okay. Good to know. I felt I I saw that and I'm like, oh, that's the profile photo that Adam has of me. Well, maybe the internet won't notice, which is never that's never a good strategy.
• 2:54
  Adam Leventhal: I like the that the Oxide official account retweeted it because I was like, you know, I'm not sure how Bryan's gonna feel about this photo that I have used as your profile photo on my phone since the day it was taken.
• 3:08
  Bryan Cantrill: Okay. So the okay. So so this is that is kind of question number one. Because obviously, I mean, like, this is like you you're you're never on like someone's phone seeing I mean, some ways it is it is our innermost sanctum is the profile photo that we choose for our
• 3:24
  Adam Leventhal: Have you heard that Josh Josh Johnson bit on this? Like, he he had a a female friend of his, like, at his house, and that like, she had lost his phone. So he calls in, finds the phone, and he sees that the name that's coming up is just nope.
• 3:38
  Bryan Cantrill: Nope. Exactly. Well, so I get so I I hear what you're I that no. I hear what you're saying is that I should be actually be I should take it as a huge vote of confidence that it was not nope that you were screenshotting. It wasn't like, see
• 3:50
  Adam Leventhal: you a loser. Right?
• 3:53
  Bryan Cantrill: The so okay. So we there was that. So you posted that. Then again, I was like, maybe the Internet's not gonna notice. With each passing retweet or repost, I'm like, it feels like we are and at some point, like, the the dam broke.
• 4:16
  And people were asking, what is this profile photo? And do and is it a violation of the Geneva Convention? I think is the the the question that is really next to
• 4:26
  Adam Leventhal: yeah. Exactly.
• 4:28
  Bryan Cantrill: I said a lot. I don't know if we're gonna explain that other than it's quite old. It is clearly incriminating at some level and it involves Super Muddy Friday, which probably merits its own episode at some point. Yeah.
• 4:43
  Adam Leventhal: I mean, it was it was at Fishworks. I mean, if if you remember exactly what was going on, but it was I
• 4:49
  Bryan Cantrill: remember what exactly what was going on. Yeah. Of course. Okay.
• 4:51
  Adam Leventhal: Just just
• 4:52
  Bryan Cantrill: checking. Yeah. No. I the the absolutely. Yes.
• 4:55
  I I am I'm attempting to eat chips without my hands. Actually, just I'm I'm now realizing that, not going into more details is actually much worse than going into details.
• 5:05
  Adam Leventhal: That's
• 5:05
  fair. I wasn't gonna say it.
• 5:06
  Bryan Cantrill: But sure.
• 5:07
  Adam Leventhal: Yeah. That's true. Yeah.
• 5:08
  Bryan Cantrill: Right. So I was attempting to eat chips without hands for money. I'm not does that make that better or worse? Unclear. Okay.
• 5:16
  So that we we wanted to get that underway. That was kind of like Internet drama kinda chapter one. Internet drama chapter two, and it's actually a little bit of a it is a dovetail into, like, the the subject matter of the podcast. Okay. You know, I we all know that you I was gonna say you live a cleaner life than I do, but you did read Yudkowsky's book supposedly cover to cover to cover you claim and I know that you believe that.
• 5:45
  And I that's actually like like that that that's actually important. It doesn't actually matter. Did you read every page or not? Who who is to say? Who's to say really?
• 5:55
  All that matters is you believe that you read, if anyone builds it, everyone dies cover to cover. Apparently not in exchange for not admitting wrongdoing in some sort of criminal activity. I I don't actually don't know. I don't need to ask the question. But broadly, you do live, I think, a cleaner life with respect to the Internet.
• 6:14
  But I got to believe that some of this doomerism that left the the fire line from Silicon Valley into the mainstream media in the last week, this surely reached you.
• 6:25
  Adam Leventhal: Yeah. Yeah. Yeah. No. I got it.
• 6:26
  I I got that that Hawking radiation even even through through family members and stuff.
• 6:32
  Bryan Cantrill: Through family member. I was gonna ask because I and so would you like to what were your family members asking you? Because I think they were asking me and asking us all the same question.
• 6:41
  Adam Leventhal: I mean, in short, like, you worried? Why shouldn't I be worried? Is this true?
• 6:47
  Bryan Cantrill: And did you in your conversations with friends, family, and loved ones, did you ring a chime for our podcast episode from three years ago?
• 6:55
  Adam Leventhal: I I mean, I did. Like, that was my response to my dad, on on that subject.
• 7:01
  Bryan Cantrill: Was, you know
• 7:03
  Adam Leventhal: No. I just I just sent him the link. He he he just he doesn't need the preamble.
• 7:07
  He just he'll just take
• 7:08
  Bryan Cantrill: the link. He just just spare me spare me your preamble. Give me the link. Exactly. Did you relisten to that episode by chance?
• 7:15
  Adam Leventhal: No. I didn't.
• 7:16
  Bryan Cantrill: It's good. Okay. Cool. Sounds so forlorn. Why do you sound so no.
• 7:20
  This is us, like, joining up to sign the light cone resistance. This was a It's a classic. It's a classic. It's some classic tracks in
• 7:28
  Adam Leventhal: there, I feel bad because every time you're like, hey, did you, you know, do your homework?
• 7:33
  Bryan Cantrill: I'm like, oh. This is not homework? No. This homework. I forgot about no.
• 7:37
  This is not homework. Was that was just going back and reliving some of the some of the greatest hits. But I want to say that that thing that's from three years ago and then over three years ago.
• 7:47
  Adam Leventhal: It's called Okay Doomer, right? That was great.
• 7:49
  Bryan Cantrill: It's called Okay Doomer, which is a great title. Is that is that what you were? Is that what you were fishing for? No, was great.
• 7:55
  Adam Leventhal: I think it's your title.
• 7:56
  Bryan Cantrill: That is a great title. That's an amazing title. One of our best titles actually, honestly. Ages very well. In fact, I think given that we can all acknowledge that the last three years have been like thirty years of technological development for all of them, it ages even better.
• 8:13
  That's like listening to a podcast from 1970 and having it still age well. And it ages really, really well is what I On a record, I
• 8:21
  Adam Leventhal: like this. Plugging the eight track from
• 8:23
  your podcast.
• 8:23
  Bryan Cantrill: It is, you put in the eight track with that podcast episode on it and get in your T Bird and, you know, roll the windows down, man. And because it crank that one, that is, it is a classic track is what that is. No, it has aged very well. And I would also like to say that I gave then that in part based on that episode, that conversation kind of inspired me to give this talk at Monctoberfest, also aging very well. I it was very nice to be able to point concerned folks, concerned energy to give people the context and I would never read the tweet.
• 9:00
  An exanthropic employee said that he felt there was a greater than 10% chance that we would have a human extinction at the hands of AI in the next ten years, which is just it's bonkers. So, know, I feel like when we were having that discussion three years ago, you were kinda like, are we is this like we kinda tilting it when I was like, does anyone really believe this stuff that we are? But indeed they do. So anyway, I I just wanna say we did. That was terrific.
• 9:30
  I I really it was it was a great conversation and it ages very well is what I have to say. And it also dovetails to today's discussion. Because the that discussion, my talk at Munktoberfest was all about the ways in which we still have very much have humanity in our engineering. And I think that there's a lot of humanity in our engineering in just in our brains. But there's absolutely humanity in our engineering in terms of the physicality of the thing that we make.
• 10:01
  And there is a huge gulf between the kinds of things that we're asking an LM to do in terms of language or code and actually and I mean, can design, which is great, but when you have something that you are that's not working properly, that is physical, it's great to have an LLM to brainstorm with. But ultimately, it's gonna be a human that is very much using that as a tool. So that is gonna be a bit of a theme for today is the, the kinds of things that we would still expect, an LM to, to not actually to be of assistance of, but this is not something that an AI is gonna do of its own. And this is still, like, pretty basic stuff. This is not a bioweapon.
• 10:47
  Although there are times when it may be felt that way, but this is, this is actually, pretty elementary stuff from the perspective of extinction. If if your goal is to subjugate all of humanity, you really need to be able to make computers pretty reliably, I would say.
• 11:06
  Adam Leventhal: Yeah. I think that's right.
• 11:08
  Bryan Cantrill: And, of course, before that,
• 11:09
  Adam Leventhal: a reliable HTTP HTTP SDK generator. But yes, please continue.
• 11:15
  Bryan Cantrill: Yes. Are you would are are you you would like to clarify to our robot overlords that you could be of great use to them in their in their robot Just saying.
• 11:22
  Adam Leventhal: For sure. Just saying.
• 11:23
  Bryan Cantrill: Just saying. You need a human to to I just I for one would like to welcome our clanker overlords.
• 11:30
  Adam Leventhal: True.
• 11:31
  Bryan Cantrill: Oh, okay. Actually, you know what? This actually does bring up one sorry. One last thing that I wanna So did you see my did you happen to see my blog post on this that where I kind of quarreled over over the weekend?
• 11:41
  Adam Leventhal: I did do my homework on that one.
• 11:43
  Bryan Cantrill: You did do it over again. Alright. Yeah. So I I boiled over. I tried not.
• 11:48
  I did try not. Okay. I tried not to boil over. And this is where I can just channel you where you'd say like, did you try? Wasn't thinking that.
• 11:58
  Write down three things that you did. How did you try? Write down in your notebook three ways that you tried not to blog this, but I actually did try. And I failed because I was just being And I wrote this blog entry on The Caucasian of Fear. And then that blog entry, because I do think that one of the things that was bothering me about this whole thing when you're talking about the extinction of humanity, I mean, you're actually, I mean, that's not abstract.
• 12:28
  You're talking about my death, your death, Steve's death. I mean, the death of our children, of our parents. I mean, it's like, these are like people that are important to us. And I I had a a a line that these ghoulish claims strike brazenly at the hearth, you know, a a line that I liked. Someone online is like, you know, I actually had to Google that line to see if it was a classical literature reference.
• 12:54
  I'll take that. That feels like praise. And then with the Simpsons appendant. And so I'm like, oh, that's kind of funny. I mean, obviously, like I made it up.
• 13:04
  So it's like, it's definitely not a classical literature line and like, it's not a Simpsons reference. And so I Google it with these ghoulish claims strike brazenly at the heart of The Simpsons. And and Google and Google's AI assistant is like, yep. During the Springfield mayoral debate on channel six, sideshow Bob dramatically delivers the line. These ghoulish claims strike brazenly at the hearth of Springfield while turning the debate against mayor Joe Quimby.
• 13:33
  Instead of denying his criminal history, Bob exploits Quimby's record and boldly tells the audience they secretly long for a cold hearted Republican to rule them like a king. I was like, what? And mean and do you feel you know that debate and that episode well enough to be like, I know that that's false?
• 13:52
  Adam Leventhal: No. I I I feel like I I it seems unlikely, and I'm like, oh, like that really stuck deep, maybe wedged deep somewhere in the brain.
• 14:02
  Bryan Cantrill: Yes. Yeah. I mean, little bit of a panic attack. Yeah. Like if I mean, if I am having what I feel to be original thoughts that are in fact just like Simpsons lines that are regurgitating from thirty years ago, like that's troubling.
• 14:15
  That's like But how are you not gonna know? CT. Yeah. How am I not gonna know? Go image that.
• 14:19
  Is there a part of my brain that we can Is there a part of my brain that's dead or that we can zap? Can we kill it? I don't know. The And then so Google folds when I I'm like, okay, it takes me, of course, like I have to go watch the whole thing. And then I'm thinking like, well, maybe I'm getting a trimmed version that doesn't have it.
• 14:36
  And then I'm finally, can you give me the whole script? And it's like, oh, no, no. No, it doesn't. Actually, this is not in the Simpsons at all. It's in this blog entry by Bryan Cantrill yesterday.
• 14:44
  Come on, man. And I'm like, okay. Are you fucking with me? And then, like, well, that seems like that seems pretty bad. Like, what happened?
• 14:51
  And then Google is like, what happened here is a classic AI hallucination that it puts in quotes and bolds. Because the phrase you shared used highly stylized melodramatic language, ghoulish claims, comma, strike brazenly at the hearth, two quotes, my system misidentified the tone as belonging to Sideshow Bob, who is famous for speaking at exactly that kind of griniloquent Shakespearean style. I'm like, fuck you. Did you just like I mean, mean, am I wrong to, like, over am I overly personalizing this, first of all? I just feel like, you're how are you you're name calling me.
• 15:33
  I this is, like, this is my fault somehow that you're hallucinating.
• 15:38
  Adam Leventhal: Gaslit into thinking you had plagiarized it.
• 15:41
  Bryan Cantrill: Yes. And
• 15:41
  Adam Leventhal: and then sort of this backhanded compliment.
• 15:44
  Bryan Cantrill: It's a
• 15:44
  total a total I'm not sure there's anything compliment about it. I think it's just like backhanded. I don't know. Don't know. Just a backhand backhanded.
• 15:52
  Alright. Yeah. Alright. We're done. I feel like that sorry.
• 15:56
  That's the I I I just wanted to get all of that out there. Good. Feel better now. Yeah. So okay.
• 16:02
  You gave Scott time to get to the stage. I I gave Scott time to, to download the Discord app and click through all of the, the the various offers that he's being yeah. All the sorry. Scott, welcome. Hey.
• 16:13
  It it is great to have you here.
• 16:16
  Scott Tagwerker: Yeah. Thank you. Thank you, Bryan. A long time first There
• 16:20
  Bryan Cantrill: you go. Okay. So this I don't know. And Scott, maybe you between you and Steve can tell me exactly when this odyssey begins. But it's gotta be, like, a couple of months ago at least when we get when do we get the old road map update?
• 16:37
  Or I guess it's a decommit that we get. Right? What how does this start?
• 16:40
  Steve Tuck: This this starts sorry, Scott. I'll let you you have got better information on this. But this starts over a year ago.
• 16:47
  Bryan Cantrill: Oh, really? Okay. Wow.
• 16:48
  Steve Tuck: So wind the clock back. And it is, over a year ago because this is like summertime last year. And a, you've got only a handful of companies that make these power electronics parts at scale in the industry. And we, we rely on lots of different parts across the system. And in this particular case, the clock buffer is one of I think eight or nine parts that we were getting from this particular partner.
• 17:24
  And, this particular partner, like many in the industry, was seeing a lull in demand. And when you have a lull in demand, that is like a perfect opportunity to move to your next generation part. Oh, yeah. Interesting. So they had taken the opportunity probably already a couple months later than they would have liked, but seized on the opportunity to move from and I won't get this right.
• 17:49
  This is where Scott, you can correct me, but it was like ninth generation part to tenth generation part. Well, when you make that cut over, you are signaling to your fab upstream, like this is where we are going next, and you're not entirely, but largely gonna live on inventory until you are all the way up and operational with that new part.
• 18:09
  Bryan Cantrill: So Okay. And in this case and so they're doing that with all their parts or they're it or just
• 18:15
  Steve Tuck: I don't know.
• 18:16
  Yeah. All I know is that we we definitely postmortem the hell out of this one. Yeah. Which what you're thinking about was like a couple months ago when we were crisis mode. Figuring out, like, how do we get here?
• 18:25
  Bryan Cantrill: Oh, this is, you this is great to know that all these crises that I actually decide just don't rise to the I mean, it's like, Adam finds out when we do a podcast. So he finds out, like, when the it's like, well, I guess we lived.
• 18:37
  Steve Tuck: Just to give you the quick version, so they, as soon as they had like made the hard cut with their upstream, partner who is like the, most popular fab on the planet right now and, they the AI infrastructure boom like lit them up less than thirty days later.
• 19:00
  Bryan Cantrill: Wow.
• 19:01
  Steve Tuck: And every single part of that ninth generation was being was in demand and getting hoovered up as fast as possible. And that like, know, end number of months of overlap of inventory until next gen is online. It was already starting to look extremely, extremely thin. And then the knock on effect of that would be you know if you have any disruption whatsoever between an already shrinking inventory, then scarcity sets in and market dynamics take over and, these particular parts that are in very high demand from lots of industries, largely evaporate. So that's where we landed a couple months ago.
• 19:42
  Bryan Cantrill: And then where do we get to Okay. I want to talk a little bit about the specifics of the part and then Scott, I want you to kind of get kind of when you came into this here. But Robert, could you describe this is a PCIe clock buffer. Could you describe a little bit about what the And this is what the part that we're gonna talk about today is this particular clock buffer, one from Renaissance, one from Microchip. But could you describe this clock buffer in terms of what it is, what its role is?
• 20:11
  Robert "RFK" Keith: So PCI needs clock. And well, I guess it doesn't necessarily need the clock itself. PCIe
• 20:21
  is complicated
• 20:21
  and there's a couple ways you can put it together. You can do it with no clock and it uses the data path to acquire a clock. We don't do that in these parts.
• 20:32
  Otherwise you need a
• 20:34
  clock and the chip, your processor provides reference clocks. And in order to get those to use, but you only have so many, right? So you want to split that up into, you know, four new clocks from one clock. And in order to do that, you need a buffer. And what that does is it takes the input clock or the output of your processor into the input of this chip.
• 20:58
  It takes that reference, replicates it with a little bit of skew, and then, you know, poops it out to your new targets. Okay. That's all fine and good. And usually, they're they're very, very simple parts, and really, they classify them not so much by how much they're going to alter the signal because it's largely an analog process, but by the delay that you get between your reference and your output.
• 21:30
  Bryan Cantrill: And these are parts that like these do not have, I mean, we have like parts that have got a big complicated software interface or have them like these do not, right? These are parts that are that the software is unaware of that we kind of like lay them down and there's no programmability to them whatsoever. I don't think. Right.
• 21:50
  Robert "RFK" Keith: No, no. And you love it like that. They have an enable. And the output and like a few like other ancillary things, like really not a lot going on here at
• 22:05
  Bryan Cantrill: all. But they're very important because So they're gonna take these reference clocks, a reference clock from the CPU, and then they are gonna generate these very important clock signals. These clock signals need And then the whole system is gonna kinda, at least from a PCI perspective, is gonna roll from there. Yeah. Okay.
• 22:29
  And then the So, okay. Do you wanna describe a little bit what the and I think we'll get Eric is gonna join us as well so we can go into a little bit of the actual the the the logic that our particular part actually views.
• 22:53
  Robert "RFK" Keith: So Yeah. Yeah. The the difference between the low power HSCL termination.
• 22:58
  Bryan Cantrill: The low power HSCL
• 23:00
  Robert "RFK" Keith: termination. And there are other terminations. You can get things like your LVDS termination. But the difference between these chips, which we did not see at first, was the difference between a low power HSCL driver and a regular HSCL driver and how a differential signal behaves due to the output driver. So, this largely simple part, they can have different types of output drivers.
• 23:28
  And some output drivers require no ground reference and some output drivers do. And that's the difference between these two things. Some of them, and in, and you can really, it's just the type of termination they use. So like these are 100 ohm signals and so you have like 50 ohm differential termination. And in one of them, you need to provide that and the other one, you do not.
• 23:56
  So that therein lies a critical small difference that God you wish you
• 24:06
  Bryan Cantrill: saw. Totally. Yeah, we'll get to that a little bit. So Scott, you were joined Oxide. When actually did you join Oxide?
• 24:22
  Because it feels like five years ago.
• 24:23
  Scott Tagwerker: It wasn't as
• 24:24
  Bryan Cantrill: well.
• 24:24
  Scott Tagwerker: It was November. Yeah. It's
• 24:27
  Bryan Cantrill: been a while. November 2003. Is that right? Mean, I that feels like that would make more sense. Yeah.
• 24:32
  Scott Tagwerker: Yeah. Exactly. Exactly. Aged enough.
• 24:35
  Bryan Cantrill: Oh my god. Okay. So you are thrown into an absolute cauldron where we are trying to scale everything, ramping up on all sorts of parts of this bomb. And then at some point, it sounds like actually maybe this had been lying in wait for you, but you begin to learn that this particular clock buffer is gonna be a problem.
• 24:56
  Scott Tagwerker: Yeah. And I think I'll I'll set the stage and says it's like, Steve did a great job on the industry over wide. But, I mean, as as I'm sure most, if not all listening know, like, late you know, mid to late last year, you start to see the signs on memory storage, CPUs, etcetera. And so we were just throwing so much of our effort after I joined in through q one, really, you know, covering down on that from the the memory and storage side. But you knew, you know, there were some other ones you're watching in terms of Murata and some flex parts.
• 25:27
  But you knew there was, you know, something else necessarily lurking there. And I will say, you know, when you join procurement roles, you usually come into a lot of fires, and I would be remiss to not mention the work that Kirsten had done and CJ and everyone else who was working to cover these gaps, you know, kind of in that that first real ramp did an incredible job where there weren't too many of these, you know, initially right after I joined. But I will say, you know, this was kind of in that transition of we had built at least quantity wise enough to cover what that, you know, kind of first that first big order was. And so then this is like the, okay, the material that needs to follow on after that with a shorter lead time. And then also as, you know, the the constraints are ripping through the rest of the supply chain beyond some of those, you know, most notable components like DRAM and NVMe.
• 26:21
  And so I started hearing, I wanna say, mid to late q one, just rumblings of Rensis. And I think we had four parts from their, from their power portfolio and three parts from the timing side. And it became very clear very quickly, you know, that this is gonna turn into a problem.
• 26:41
  Steve Tuck: So we
• 26:41
  Bryan Cantrill: got the timing portfolio and the power portfolio. And Yes.
• 26:44
  Scott Tagwerker: It became clear pretty early that power was gonna be rough. Was gonna be rough for a a given period of time. And we need to go do something about that. And so for, you know, a lack of a better term, we went out to, you know, the the open market and almost, you know, bought our way at least some runway in terms of the power side. But we just weren't finding anything out there for the clock buffer.
• 27:06
  And so as we were successful in finding under rocks and nooks and crannies and working on the power side, there was just nothing to be had on the clock buffer side. It was clear if we didn't go solve this, there was going to be potentially months of challenge. And so at the same time, as we're really working through this and the supply challenges, the Resistor team that we start to work with says, oh, by the way, heads up, we've sold the timing business to Cytimes. And, in in a way that was, like, on one hand supportive of, hey. We'll make sure you got the right contacts, and we'll make sure that goes that goes well, but also We're making
• 27:44
  Bryan Cantrill: sure you get your get to a good home.
• 27:46
  Scott Tagwerker: Yeah. Also, like, that's we're not controlling that anymore. And so it was shifted over to to a full new. And it it was clear, and then, you know, I think they'll admit this is, you know, those are always challenging. And from a systematic execution standpoint, there was just a lot of challenges in terms of understanding when supply was coming at all and how much was there.
• 28:06
  And so that was really the situation as, you know, and we try to minimize these, and I'm sure as Robert can attest, there's a lot more of these requests than any of us would like. But this was certainly one where going and finding it all was was a high priority to keep, you know, keep builds going, keep us from going down for months.
• 28:23
  Bryan Cantrill: Well, and it's like this knife edge. Right? Because we on the one hand, we I mean, for a bunch of reasons, alts can be are expensive. We gotta go find something that ideally is is compatible. For some of these things, the I'm not sure if if you were struggling on the power stages or on the actual power controllers.
• 28:40
  I mean, those power controllers, we gotta go get those because I'm sure like, I'll just channel Eric here to like, yeah. Yeah. Those are those you need to just go like, you know, obtain the using whatever dark arts in the broker market you need to go to because those parts were really they we we've got are really, really critical. We don't have a drop in replacement for those. So on the on I'm glad that we're able to screw that on power side.
• 29:06
  Around the timing side, feels like, well, maybe there's some drop in. Mean, in replacement's a little bit tough. But I mean, you must be on this kind of knife edge of like, okay, how do I kind of let the double e's know that we need to start scoping some of this out? And then that urgency getting kind of increasing is like, oh my God, we're gonna actually run out here and we're gonna be we'll be we won't be able to make, make machines. We'll be bad.
• 29:30
  Scott Tagwerker: Yeah, absolutely. And I and I will say, like, there's a there's a like you said, there's a delicate balance there because sometimes on the operations procurement side, especially when it's we're not in the weeds, there's a or at least, you know, if Benchmark is doing some of the or ADCM is doing some of the procurement, that that's the first ask from their side.
• 29:51
  Eric Aasen: Hey, can
• 29:51
  Scott Tagwerker: you just go find an alt? But, you know, the our technical team, our engineering team has already, you know, has plenty on their plates that does not involve going in and qualifying and finding alts for us
• 30:03
  Eric Aasen: in any
• 30:04
  Scott Tagwerker: situation, especially in a market like this. And so you really got a sanity check that we've done everything we can to find any any possible parts before you go make that ask. But this one certainly got to that that price.
• 30:14
  Bryan Cantrill: Well, totally. And there's always like this back and forth too being like, okay. Because it means parts are like, not equal. And, you know, some of them we've got a lot and I do remember Eric, do you remember when Robert, you probably remember this too when at some point during, like, a Gimlet era, we were during the the first supply chain shortage and the kind of post pandemic, we were really struggling to source the TPS five forty six b twenty four a, the the buck converters that that that we use, the TI ones that we all, like, I think pretty much love. And CJ's like, what if we can't get this anymore?
• 30:45
  And it was like absolute rebellion. He's like, okay. Okay. Settle down.
• 30:50
  Robert "RFK" Keith: I'll get there. Some parts there are some things where you're like, you can probably find something close, and then there are other parts where you can't. And so if you bring up the ones that you can't and you ask for a replacement, the answer is no. Yeah. Yeah.
• 31:04
  I'm sorry. You must find them. That is Right. Right. This is not a this is an optional thing.
• 31:11
  Bryan Cantrill: And because we've a partner, we've got a software support that works and so on. It's like, boy, if we if we if we really don't wanna have to design that one out.
• 31:18
  Scott Tagwerker: Yeah. I think there was one of these recently where Eric told me, like, we should just go by five years of of that and have no no chance of running out.
• 31:25
  Bryan Cantrill: I like it. And Eric, what what what part was that? Was that was that I I almost wanna play, like, part beta on some of, like what's that?
• 31:32
  Eric Aasen: The IBC.
• 31:34
  Bryan Cantrill: The IBC. A 100%. Yes. Yeah. Yeah.
• 31:36
  Yeah. For sure.
• 31:37
  Robert "RFK" Keith: Well, that's a big one.
• 31:38
  Bryan Cantrill: Yeah. And yeah. That's a great example. So the the intermediate bus converter, ring the chime actually, you know, welcome to Oxide and Friends where when when someone mentions the IBC, we can ring the chime on a previous episode. But, yeah, we just we know we're gonna be having that part for a long time and just go buy all the inventory we can.
• 32:03
  Eric Aasen: Yeah. I mean, things where you have super deep supply chains, the IBC, the I even the fans have a deep supply chain.
• 32:10
  Bryan Cantrill: And
• 32:12
  Eric Aasen: every time you have a deep supply chain that is opaque to us, we can't possibly help figure out how to fix supply chain issues. I mean, we've had some success in the past helping our suppliers fix their own supply chain issues, but some of these parts, just don't want to be there.
• 32:29
  Bryan Cantrill: You don't want to be there.
• 32:30
  Eric Aasen: The rectifiers, the IBC, you want to have enough safety staff that you can weather the storm.
• 32:37
  Bryan Cantrill: And so Eric, at some point, Scott is because I think he's either just coming to you or Robert, he's coming to the team anyway being like, hey, we really need to get alternatives here and start looking for alternatives. And so when you are looking for an alternative, because it's not like, oh, we just need a PCIe clock buffer. It's like, no, no, no. There are lots of constraints on this. Do you want talk about some of the constraints when we're looking for something that we can substitute in?
• 33:02
  Eric Aasen: Yeah. I mean, ideally we would have something that's footprint compatible. So that's your first filtering criteria is what can fit in the same footprint because making an interposer board for something like this is painful, to put it lightly. And one of these is actually underneath their fans in a little gap that is between the fans, but is otherwise dead space on our board. So we put low profile things like a clock buffer in there, which is great until you have to stack it on top of an interposer.
• 33:32
  Bryan Cantrill: Oh man. Yeah. Did not even realize Oh
• 33:39
  Eric Aasen: Yeah. So there's okay, is it footprint compatible? Cause like we can't, I mean, we could just change the design quote unquote, right? But the lead times on these circuit boards are like six months. And so we have like six months of supply chain in queue and in process being built and changing something like this means, okay, yeah, we'll get it, at this point next year, you know.
• 34:03
  Like, well, okay, so ideally footprint compatible. And then after that, it's like, okay, is it footprint compatible? And then does it have all the inputs and outputs, right? We individually enable our output clocks. Some buffers don't allow that.
• 34:18
  There's that how many clocks does it output because we have, you know, four. It's a one to four buffer, so we have one input and four outputs. And even if you didn't have a footprint compatible part, you still have that one input and four outputs, no matter what, or something to get you to that. And there not a like there are variety there's a variety of options, but none of them are available from distribution. So even if we wanted something that wasn't footprint compatible, nothing's in distribution for stuff like this because it's all hyperscaler getting sucked up by the hyperscalers and the big PC vendors and everybody and their mothers sucking up these carrots because everybody needs them and there's only a few vendors that make so even if we wanted to change to the newer technology, three by three millimeter part, you know, that'll go up to PCIe Gen seven, lock jitter specs, we can't because they're it wouldn't help us.
• 35:15
  Bryan Cantrill: Right. We
• 35:17
  Eric Aasen: have to find something that actually is available.
• 35:19
  Bryan Cantrill: And so, and this is a four millimeter by four millimeter part to give you the because the actual size of this thing is gonna become important later. So this is like, I mean, it's a small part. I mean, it's a small part. We are And it's around this. So Eric, you're beginning to look at We're not finding things in the broker market.
• 35:41
  Are you finding things that you're kind of like bouncing off of Scott and Scott's like, go fish or how is that kind of working in terms of determining the availability?
• 35:48
  Eric Aasen: I mean, it's a combination of me searching for things because you never know quite what you might find versus somebody else. There were some suggestions from our Centimeters and there were some suggestions from our distributors. And one of those suggestions was this microchip, which at a 10,000 foot view looks absolutely perfect, minus one quirk where one of the pins that was a loss of signal indicator, which means, hey, if I lose my input clock, I assert some signal, which we don't really care about, on our current part would become a power pin on this new part, it's like, oh, well, okay. Maybe we can figure that out. There's already a pull up resistor there, and maybe we can just start connecting it, and it would probably be okay.
• 36:31
  But otherwise, like, the footprint matched perfectly. Like, it was literally the same package. It's like, oh,
• 36:37
  Bryan Cantrill: that's Yeah. That's amazing. Okay. Yeah. So so no interposer.
• 36:41
  That's great. The part is available, I guess. I mean Yeah. And to give an idea
• 36:47
  Scott Tagwerker: of like, Bryan, it was like, it was like a hand wave, the availability in the sense that I think we found like, like 3,000 of these. And then there was a, Hey, we think we have a line on another, you know, which, which I really is only, I want to say, yeah. I don't know. Yeah. 20, you three racks or something like that.
• 37:05
  I feel like we we use a lot
• 37:07
  Bryan Cantrill: of these.
• 37:07
  Scott Tagwerker: We need to go back and look at it. But it was it was something that would hey. It would it would get us started, but we really needed to go see, like, okay. We still gotta go chase this other one down, but it's certainly a lot better than the, than the original.
• 37:19
  Bryan Cantrill: Right. And then given the, I mean, relatively low cost of these things as it is, mean, it's on the order of like a $3 part, Scott, is that right? I mean, something like on a couple of bucks.
• 37:28
  Scott Tagwerker: Yes. Yes. That's correct.
• 37:30
  Bryan Cantrill: So, I mean, are you at what point are you like, I'm just gonna start buying these because this is gonna be close enough and we because you've also got to be, mean, for sure there is, certainly in the pandemic supply chain crunch, one of the things we were contending with was a lot of panic buying on some of these old smaller parts. And all of a sudden, you'd go to the, someone would have a lot that was available and then that would all of a sudden disappear because there's a lot of panic buying going on. You must have been concerned about the same kind of stuff happening here.
• 38:04
  Scott Tagwerker: Oh, yeah. For sure. And I think, you know, I think when we when we looked at the previous part, like you said, it's probably a $34 part, and it started we we found it would find some at, you know, $20 and then $30. And then by the end, you know, this was up around $70, I think, before it it just completely ran dry. And it just, you know, I I think that, like, we've talked about this past week as we've been, you know, just making sure that we we cover down on all options.
• 38:31
  You know, these things got to $7,001,000 dollars in terms of just like when there's the lot the last few leftover remaining. And so it's that balance of
• 38:40
  Bryan Cantrill: I mean, how
• 38:40
  Scott Tagwerker: do you go? I mean, and and in some cases, okay, relative to what the impact is on your overall supply chain versus people sitting around just the, you know, the the loss of the ability to go build. Sometimes those still make sense, but you really gotta just balance what that urgency is with the other options over in in flight.
• 38:57
  Bryan Cantrill: I did like especially as, like, the the market was really running out of parts. Steve would be Steve and I was on these calls with Steve and you, Scott, where Steve's like, money is no object. And you'd be like,
• 39:09
  Steve Tuck: Steve, I That was that was at $20.
• 39:11
  Bryan Cantrill: It was at $20? And then
• 39:13
  Steve Tuck: It was like, buy every part you can get on the planet. And then Right.
• 39:16
  Bryan Cantrill: Scott would be like
• 39:18
  Steve Tuck: 62 yeah. Hey. By the way
• 39:20
  Adam Leventhal: By the way
• 39:20
  Steve Tuck: week later, it's now $62, which is wild. 20 x. And I was like, go well, keep your eye on as many of those parts as you can.
• 39:31
  Scott Tagwerker: And then it was into
• 39:32
  Steve Tuck: the hundreds is just, I mean, insane.
• 39:35
  Scott Tagwerker: Yeah. And then that was Steve, when it when it started reaching those prices, was okay. So wait. When are we lying down? What else is?
• 39:42
  Bryan Cantrill: Right, right, exactly. So, okay, so we get the microchip part. Eric, it sounds it requires a little bit of a stencil change for this control pin that we want to pull up. Is that right?
• 39:59
  Eric Aasen: That was just a bomb change. So that was one ten ks resistor to a zero ohm and it worked well, you know, works fine. Oh, perfect. Oh, Yeah. Got There was an initial, like like, we looked at this, Tom looked at this, and Tom noticed that the the way they were showing the input termination was different than the old part.
• 40:21
  It's like, okay. I looked at that and it's like, oh, no, they're just showing the bias resistors or something that puts the input to a known state when there's nothing driving. Okay, fine. Well, that doesn't matter. And there's no termination on the previous part.
• 40:36
  So like, there's no termination really on these blocks. And it's for someone like me, I come from a traditional, like, this is not how you do PCIe or any differential pair. You always have termination at the end. So you always have like honored own diff or a same termination structure for LVPECL or LVDS or whatever.
• 40:56
  Bryan Cantrill: Describe what termination is a little bit and what Yeah. Gig is
• 41:01
  Eric Aasen: So you any signal that's high speed or has really fast rise and fall times has a lot of high frequency content. You remember back in the old days of your RF class or your electromagnetics class, anytime you have an impedance, which is a combination of the inductance and capacitance along the line, say, just super simplify it, but the impedance of a transmission line, anytime there's a change, there's a reflection. And so if you have an impedance change, it causes reflections which cause your signal to look like garbage, and that is generally not preferable. Especially when you have something like an open circuit, which is what the input of these clock numbers looks like, and all PCIe devices, all their clock input pins all have this structure where it's basically just open circuit. It's like it's driving nothing.
• 41:58
  You know, the wire is just infinite impedance. And that caused
• 42:02
  Bryan Cantrill: When say open circuit, you mean there is it it's not coming back to ground through that?
• 42:07
  Eric Aasen: Yeah, there's no like intentional impedance, resistance, whatever to ground or power or any of anything else. It's just, you know, the gate of a transistor essentially. And traditionally, the way that you deal with that is you put a termination, so then the end of your transmission line looks like the rest of your transmission line and none of the signal reflects. But PCIe is a little different, and for the clocks, they are all source terminated. So the way HCSL, high speed current steering logic, which is what you call the output structure of all these clock drivers.
• 42:52
  The way it works is it has a 14 milliamp current source and that 14 milliamp current source is switched on to either the positive or the negative signal depending on what you're looking for. And so whether you want the positive side high or the negative side high for your thing. You have a 14 milliamp current source, or I guess they show 15 near, but some low milliamp current source and there's a couple of transistors that switch that on, and so then you have these resistors to ground, and those resistors to ground are what actually create the voltage that the receiver sees. And if you don't have those, it turns out that that 15 milliamp current source just pulls it right up to VDD, and that's not the way our old drivers worked. Our old drivers are what's called low power HCSL.
• 43:48
  So instead of a 14 milliamp current source, they have this LDO or whatever with a 0.75 volt source, and then there's more transistors here, kind of like you see on LVDS or many other, like, voltage source transmitter designs, differential transmitter designs. Yeah. It's it's lower power, so you don't have to source 14 milliamps into a resistor to generate your voltage. You just generate the voltage and then just switch it onto your your transmission line here. And these termination resistors, RS, are to slow the edge down if you want to.
• 44:24
  Though we do have massive impedance mismatches, those RS resistors can help you kind of slow down and filter out some of that high frequency content so that the reflections aren't as bad.
• 44:35
  Bryan Cantrill: Oh, interesting. Okay.
• 44:36
  Eric Aasen: Yeah. Yeah. Yeah, so we designed it for the low power HCSL on the right.
• 44:43
  Bryan Cantrill: Into the low power HCSL, which does not have the resistor to ground is that's like the newer generation, That is like the, that is the that is the kind of so when we built Gimlet, right? Because this was the we had this thing on Gimlet. So this has been true for a while. The we were, I mean, designing that way that kind of, you know, made sense. And one of the things that we didn't realize was that this microchip part was this effectively older generation, probably explains its availability, I would imagine.
• 45:19
  Which actually it consumes a little more power because it it actually has this connection to ground, and has resistors there.
• 45:29
  Robert "RFK" Keith: That has more parts. It
• 45:31
  Eric Aasen: And you have more external parts and that's generally not desirable in applications, so a lot of these parts integrate as much of this stuff as they can. But in this case, they didn't. We were caught with our pants down a bit because we didn't realize this and then built.
• 45:51
  Bryan Cantrill: Well, and it should also be said that because we actually, we got this on a board and you measured it, right?
• 45:58
  Eric Aasen: Yes, I measured it. And I did I did clock jitter measurements, right? All the right things. Okay, fine.
• 46:04
  Bryan Cantrill: Looks great.
• 46:04
  Eric Aasen: When you measure clock jitter, you're using an eight gigahertz scope, which automatically means you're terminating at the ground with 50 homes. So the scope was providing the required termination for this driver chip.
• 46:18
  Robert "RFK" Keith: Literal probe effect.
• 46:19
  Eric Aasen: Probe effect. Literal probe effect.
• 46:21
  Adam Leventhal: To test this, I'm always fascinated by your rework. Do you, did you take like a raw board and that had not had parts on
• 46:30
  Bryan Cantrill: it or did you go rip it apart? Yeah, Good question. In terms of like testing that first one, whether this is a Yeah, sensor at great question. Think you run one through, don't think we did, we didn't manually rework No,
• 46:42
  Eric Aasen: yes we did.
• 46:43
  Bryan Cantrill: We did. Oh, wow.
• 46:44
  Eric Aasen: Okay. Well, we had them rework it. Mean, I can't.
• 46:46
  Bryan Cantrill: Yeah, right. Of course.
• 46:47
  Eric Aasen: I looked into a rework machine that would be able to do things like this and it is ludicrously expensive.
• 46:53
  Bryan Cantrill: Oh, That's something I was going to ask. A rework machine. Okay. Yeah. They do exist, but they are very, very expensive.
• 47:02
  Robert "RFK" Keith: Yeah. Like a DRS. They're expensive.
• 47:04
  Eric Aasen: Yeah. DRS. It's our board. We need for our class of boards is around $250,000.
• 47:12
  Bryan Cantrill: Wow. But,
• 47:13
  Adam Leventhal: yeah, eight sticks of DRAM. That's crazy.
• 47:15
  Eric Aasen: Know by the way
• 47:17
  Bryan Cantrill: 250 clock buffers. Exactly. That's right.
• 47:20
  Eric Aasen: And that that assumes you have an x-ray machine to inspect those hidden cider joints you just made. Yeah. So, you know, there's another, 200,000.
• 47:28
  Bryan Cantrill: Yeah.
• 47:29
  Eric Aasen: So yeah, this was we had two boards reworked by our Centimeters and I went over to Minnesota and tested them with our eight gig scope. He performed beautifully. It's fantastic performance because the scope was terminating them right at the end until the waveforms all look beautiful. Fast forward.
• 47:53
  Bryan Cantrill: Fast forward. Yeah, exactly. So we head into the wilderness now, confident that we have a drop in replacement. And then we discover that these machines are behaving. We get our kind of our first machines with this microchip part and the symptoms were kind of bizarre.
• 48:13
  Eric, what would they do?
• 48:15
  Eric Aasen: So like it wouldn't net boot, but then we would, when we logged into them, we could see drives. So that means PCIe was enumerating. However, when we had them load an M. Two, which is our boot drive from a board that had already been programmed. So this drive is normally programmed as part of our manufacturing flow, but we couldn't get to that point on these boards, so we just stuck one of these that had already been programmed from another board into this and booted it up, and it panicked.
• 48:49
  But it panicked because of a front U. Two drive, not because of the M. Two. We're like, okay. So let's pop all the u dot twos out.
• 48:58
  And then we popped all the u dot twos out. And then it panicked again on the m dot two. It's like, oh, okay. Crap.
• 49:04
  Bryan Cantrill: Okay. Oh, boy. Oh, boy. Okay.
• 49:07
  Adam Leventhal: And the like the brain surgery was not successful.
• 49:11
  Bryan Cantrill: Yeah. The, the brain, well, they're successful. It's just like, he makes inappropriate jokes now and he does it all the time. Exactly. It's like, well, can we muscle him?
• 49:26
  Well, we tried that. And then he writes them down and puts them in front. Okay. Okay. So yeah.
• 49:32
  This is getting and so Okay. So you get on there. And I think at this point, I know we were It's kind of going back and forth with our folks in manufacturing. So we're getting I mean, like, we're we got a we're cranking on manufacturing these things. So it's like, oh.
• 49:50
  Eric Aasen: We have a few of them made at this point.
• 49:52
  Robert "RFK" Keith: We made like 800 or something.
• 49:55
  Eric Aasen: Yeah, 100 something. Yeah. It wasn't like 300 server boards.
• 50:01
  Bryan Cantrill: It's many. And so you, at some point, Eric, I mean, are out in our facility in Winona there with our manufacturing these to get hands on there to figure out what's going on and actually see this thing on the scope.
• 50:19
  Eric Aasen: Yeah. So we go out there, we get our scope on there and we look at it and we're like, oh, no. I think before we even went out there, we identified we identified that Resistor yeah. We we saw that it was weird because we had one of their production folks who was fantastic. He got their eight gig scope out and looked at some things, and things didn't seem right.
• 50:50
  And this the what we saw was the clock just sitting high all the time and went, oh, crap. The clock sitting high all the time stirred some memory where I'm like, wait a minute. HCSL only pulls up. It's like
• 51:04
  Bryan Cantrill: Oh my gosh.
• 51:04
  Eric Aasen: Cursed iceberg. It's like ultra high speed iceberg c. Right. And so it's like, oh no, it's always pulled up. That's not good.
• 51:14
  That means
• 51:14
  Bryan Cantrill: That's not good. Okay. So you have like the realization that you're like, and I'm sure like you realize it all at once that like, oh my God, it was the probe effect that was allowing this to look good and oh shit. Okay.
• 51:27
  Eric Aasen: Yeah, that massive sinking feeling and you go, And so then the scramble was on like, okay, we need four zero two, 49.9 ohm resistors or something close to that.
• 51:39
  Bryan Cantrill: Okay. So, with the let's just pause on that. So, 49 ohm resistor, very small, obviously, because I mean, the that's why these things obviously don't get to me, but very small resistor there. And then an four zero two, can you describe how physically small this thing is? Because these things are tiny.
• 51:59
  Eric Aasen: Yeah. So four zero two, if you look at the metric designation, I mean, English designation also makes sense. But the metric one is more intuitive. The metric version of an o four zero two or the designation is 1,005, and those digits mean 1.0 millimeters by 0.5 millimeters. And no, Adam, I'm not doing it miles or furlongs or whatever, Ordinizing.
• 52:21
  And
• 52:23
  Bryan Cantrill: He has he has CHAPT to to do the the furlong.
• 52:26
  Eric Aasen: In English units, o four zero two is 0.4. It's really 40 mils, so point o four inches by point o two inches. So that's an o four
• 52:36
  Bryan Cantrill: zero My god. That's the o four zero two.
• 52:40
  Eric Aasen: Yes. And o two zero one is point o two thousandths of an inch. Point o two inches by point o one inches. So 20 thousandths of an inch by 10 thousandths. So to give you an idea, a sheet of paper is about four thousandths of an inch thick.
• 52:57
  So this is, like, four sheets of paper by two sheets of paper.
• 53:02
  Bryan Cantrill: Mean, that's That's It is insane. This is I mean, these things are I mean, the the the line that that that I think Aaron had heard the grain of pepper as the description.
• 53:15
  Eric Aasen: Yeah, like ground pepper is about the right size. So it turned out that we had this via structure where we send these signals to the inside of the board. And when we do that, these two differential signals go into the inside of the board and there's a few vias around them that act as return signal via, so they're ground vias. It just so happened that the spacing between our signal via and our ground vias was just about perfect for an o two zero one. So then the mad scramble for
• 53:50
  Bryan Cantrill: I mean, so in the okay. So first of all, that is like major a break in our favor. Right? I mean, it feels like they mean, correct me if I'm wrong, that is that was just kind of like lucky.
• 54:01
  Eric Aasen: Yeah, it was. Yep.
• 54:03
  Bryan Cantrill: Yeah. That was very long. Thank you know, sometimes get unlucky, sometimes get lucky. Gotta just just be thankful for the luck, man. Okay.
• 54:12
  So we have got We've got it exposed via the the length and we are gonna be able to basically potentially rework this grain of pepper on there.
• 54:23
  Eric Aasen: Yep. So we asked, we didn't have any in our BOM that we used already. So like we didn't have any inventory in our Centimeters for a resistor that size and that even close to that value. The only O201s that we use are AC coupling caps for PCIe because dealing with O201 sucks.
• 54:45
  Bryan Cantrill: So we
• 54:48
  Eric Aasen: asked our SAM like, hey, do you have anything within any reasonable value of 49.9 ohms in your inventory that you could sell to us? You know, I mean, replacing these is not part, right? You can get them by the real 15,000 digit people, like, $120. But it was like, can we get them today instead of tomorrow?
• 55:12
  Bryan Cantrill: Right,
• 55:14
  Eric Aasen: yeah. And they looked and they said, oh yes, we have one, and we looked in and we're like, okay, think it was 45.3, I remember right? Right. And so we're like, okay, great. 45.3, do it, rework.
• 55:30
  And so they reworked the rewards.
• 55:32
  Bryan Cantrill: And so just to give you the rework here, because I think I just am, I mean, I've just so the dexterity here is so extraordinary because this is not actually one of these that we need per board. It's think 30. 30? Is that right?
• 55:49
  Eric Aasen: 30.
• 55:50
  Bryan Cantrill: Yeah. Okay. So we
• 55:52
  Eric Aasen: The rework people are insanely skilled and they work under a microscope, literally looking through a microscope with the finest tweezers you could imagine in their hands for hours on it.
• 56:04
  Bryan Cantrill: They're They're amazing. They have very steady hands and really, really good hands. And I I mean, think it's like, I mean, look, we all know that I've just got like just absolute I got like feet for hands. I mean, feel I've got very clumsy hands. And it's always because I always hold I mean, Robert, you I mean, like, Robert, you you've got surgeon's hands.
• 56:26
  I mean, it's just like you're very dextrous with soldering. Whenever you guys are like, oh, no. No. This stuff is gnarly. This is this is like next level.
• 56:36
  I get like, I it's, total reverence for how small this thing is. This is real and and the skill required for those folks that are doing the rework.
• 56:46
  Robert "RFK" Keith: Yeah. I mean And again,
• 56:47
  Steve Tuck: it's a real say this work, this rework and this working was highly consequential. Steve's contribution to the call.
• 56:56
  Bryan Cantrill: Like, hey, I just wanna unmute myself to say that this is, yes, highly consequential.
• 57:02
  Steve Tuck: I mean, well, just because we mean, the alternatives were- wind down. Try to find thousand dollar parts such as you can go find them out there.
• 57:10
  Bryan Cantrill: Right. Yes. Yeah. I mean, we really, really, really need this to work.
• 57:14
  Scott Tagwerker: And then I think the, the other side of that is we got them in these alternates on a timeline that was more favorable, but I think we ran into another challenge with like a TI part. And so we really were like, this was going to be very impactful from a schedule standpoint with very, very firm commitments that had been given.
• 57:33
  Bryan Cantrill: Very firm commitments. But the good news is
• 57:36
  Robert "RFK" Keith: Just wanna let you know.
• 57:37
  Eric Aasen: I'm just gonna let you all know we're counting on you.
• 57:39
  Bryan Cantrill: Yeah. We're all counting. We're all counting. Well, yeah. Total total airplane.
• 57:43
  We're all are all counting on you. But we have so but we This is honestly great because like, okay, we we know what's going on. Eric, you've got a I mean, the hypothesis in terms of like, oh, shit, we we we've got this resistor because this is not LP, HCSL, it's HCSL. I mean, it just like fits with all the data. So this is like, and and oh my god, we got this lucky break.
• 58:09
  We can get this resistor in there. They've got with they've got a resistor that's roughly the same size like or the the roughly the value we need. The specific value is not as important like 45.3 or whatever it was is like fine. Great. They're gonna rework them.
• 58:24
  We're gonna power these boards and like, Crisis averted, like feeling really like and we we I feel like we're pretty good about not getting too optimistic. Maybe now is a good time to just talk a little bit about just a brief interlude about who was ultimately to blame for this because we had had an all hands where we got a little, we got a little heady and
• 58:53
  Eric Aasen: Somebody got a little out of line.
• 58:54
  Bryan Cantrill: Somebody got a little out of line talking about how like, oh, like Oxide always ships and like, we've got a great track record for shipping. And then Steve, I feel like you
• 59:03
  Steve Tuck: Let's get that person on the podcast. I'll come to answer this.
• 59:05
  Bryan Cantrill: You know, Steve, what do you have to say for yourself?
• 59:09
  Steve Tuck: I will say generally, am among the twitchiest and Yes. Least prone to confidence around things that are not well in hand occurring.
• 59:21
  Bryan Cantrill: Yeah. And I would say that like in general, I mean, twitchy I mean, obviously, we all honor your twitchiness, but your twitchiness is generally such that like, we I I think I pride ourselves on being a company where we can bring bad news to one another. Bringing good news is a little more challenging because often good news is meant like, oh, well, I'm glad that that's problems resolved. Meanwhile, do you wanna know this may may I talk to you about all the other ways that this is gonna go wrong?
• 59:43
  Steve Tuck: Alright. So because this is turning into my performance review No. It's fine. I do wanna argue a few points that led me to this point of of undue optimism. We had navigated over the prior twelve months.
• 59:55
  First of all, a crippling DDR5 crunch in the industry.
• 59:59
  Bryan Cantrill: Yes.
• 1:00:00
  Steve Tuck: And being able to navigate that in ways that we'll be able to talk about, soon was a just was felt like we had really, you know, lucked out and also done some things that put Oxide in really good spot. So navigating that wild crunch was step one. Then the NVMe crunch shows up, then the IBC crunch shows up, and we've kinda like knocked these things down.
• 1:00:29
  Eric Aasen: All of
• 1:00:29
  Bryan Cantrill: them. All of them.
• 1:00:30
  Steve Tuck: So we are now like we are on the path to being able to ship at, you know, in in many regards, like, 20 x the volume per month than we had ever done before.
• 1:00:40
  Bryan Cantrill: It was it is amazing.
• 1:00:42
  Steve Tuck: So we're, like, on the kinda glide path there, and then we hit this crippling clock buffer issue. And then as we just described over the prior, like, fifteen minutes, the numerous oh shit moments and getting footprint compatible, getting pin compatible, being able to get down to a part that not only worked but was in enough of a supply to keep us going.
• 1:01:04
  Bryan Cantrill: So it was just like kind of like No, it's a lot of I mean
• 1:01:07
  Steve Tuck: Bullet pass, bullet pass, bullet pass, bullet
• 1:01:09
  Bryan Cantrill: wouldn't be filled with heady optimism. Now, I think in so you kinda got the phrase Oxide ships kind of mentioned this that Told to me. Told to you. Told to you. And then
• 1:01:20
  Steve Tuck: By by, you know, a user of Oxide.
• 1:01:23
  Bryan Cantrill: Yeah. And and I think in your defense, it was not your idea because in the all hands, we got the all hands and the chat on the side. It's got the a very sure that, you know, the very Oxide kind of a thing that there's always a, someone in the chat is like, we should get that on a T shirt.
• 1:01:36
  Steve Tuck: And by the way, like, where the reason I the reason I even brought it
• 1:01:39
  Bryan Cantrill: up is that And you didn't say that. I just wanna be We thank you. Thank you. I I just wanted you to hear that from me. I know that
• 1:01:44
  Steve Tuck: you Important hear it from you. No.
• 1:01:46
  Bryan Cantrill: No. I know. I because I feel because It is.
• 1:01:48
  Steve Tuck: Was a different time.
• 1:01:49
  Bryan Cantrill: There was a different time. Well, listen.
• 1:01:51
  Steve Tuck: We My picture was on the wall.
• 1:01:52
  Bryan Cantrill: Well, we record all meetings and you were able to like, sometimes it's important to go back and listen to a recording of the all hands. Because you're like, you know, that I wanna make sure that I understand exactly what I said. And then it is true. Like, you had really just parroted back something else that a customer had told you, namely that Oxide ships.
• 1:02:07
  Steve Tuck: Yeah. That kinda that kinda ran away.
• 1:02:08
  Bryan Cantrill: And then, I mean, no.
• 1:02:09
  Steve Tuck: Mean, it's like was proclaiming we should get T shirts made that Oxide always ships.
• 1:02:13
  Bryan Cantrill: Dropped a lit match into chat and, in high wind, and it got things got carried away. And, yeah, next thing you know, people were talking about T shirts. At this point, like, the w's like, no. No. No.
• 1:02:22
  No. We're not putting I know. We're not putting like, I mean, there's optimism and then there's hubris. We Stop. We Optimism is a value.
• 1:02:28
  Hubris is an operating system. We really try to stay
• 1:02:31
  Steve Tuck: on of high consequence. And I think the the point I was trying to make to the broader company was, like, we are not alone. Like what what Scott Tagworker is seeing out there in the market and what we are hearing from companies that are much much much larger than Oxide is that, you know, for want of a clock buffer or a resistor or a I mean, these like these these smaller parts that are often overlooked are stopping or cutting back massive allocations to customers from companies that are in the hardware space. And, and what we had heard from a customer of ours was like, hey, just want to say we don't get to see all the diving catches. You know, obviously we know, there's been a lot of hard work that has been done and, just want to let you know, like Oxide always ships.
• 1:03:18
  So that, you know
• 1:03:19
  Bryan Cantrill: Yeah. I got it. Still feels like now you're not I know. But now that you're repeating it back, it's like, god, everyone's frenetically knocking on wood. I'm gonna knock on wood right now.
• 1:03:26
  I mean, needless to say, the gods were dialed into that all hands and and the gods did not like what they heard. The gods were just like, these idiots. And that the all hands was on a Tuesday and we had a board that did not work on Wednesday night. Within and then Or probably Wednesday. No, no, because then we have the breakthrough because Eric got Eric's like, okay, know what's going on.
• 1:03:48
  We rework it and Eric, now we're optimistic.
• 1:03:50
  Eric Aasen: Right,
• 1:03:50
  Bryan Cantrill: right, We're going into the into like Friday being like, we've got this thing reworked. We're going to power it on. We think it's going to work. This is clearly the problem. I felt like I mean, a lot of confidence.
• 1:04:03
  And then Eric, what happens?
• 1:04:05
  Eric Aasen: Yeah. Well, so much like Neo and his first, his first example of dodging bullets in the matrix eventually one nicked him and this one nicked us.
• 1:04:16
  Adam Leventhal: So they, yeah, they
• 1:04:20
  Eric Aasen: reworked them. They put them in the programming station and they didn't work. In the
• 1:04:27
  Bryan Cantrill: exact And
• 1:04:27
  Eric Aasen: it's same in one of these where you're
• 1:04:28
  Bryan Cantrill: just like, you're just like, maybe did it work like a little bit better? It's like, no. Basically the same as before. And this is one of those where, you know, the crops have failed. And I mean, the crops have failed a lot around here, and I feel like we all have we all do.
• 1:04:44
  This is like, listen, we may have a problem with with unbridled optimism. When the crops have failed, I feel like we're all pretty like, okay, crops have failed. The the the Robert, this is when we were like, okay, we actually now we really need to darken the skies with double e's. Robert, you get on a plane. Eric, you're going out there.
• 1:05:04
  We were basically getting everyone to go out to Winona because now we just don't know what we're I mean, goddamn it. Like this hypothesis was such a great hypothesis. Yep.
• 1:05:16
  Steve Tuck: I mean
• 1:05:16
  Bryan Cantrill: Oh, man.
• 1:05:17
  Robert "RFK" Keith: At this point, it's like where the normal, where it seems so obvious what the problem is and then that doesn't work. And given what you're working with, you think, oh, like this is going to be really gross. Like that some something is very not cool right now.
• 1:05:35
  Eric Aasen: There's gonna be some horribly weird interaction that we didn't see or anticipate because it's some weird Yeah. Or some undated undocumented thing or whatever.
• 1:05:45
  Robert "RFK" Keith: Exactly. It's on a clock signal and you're
• 1:05:47
  Eric Aasen: like, oh, in PCIe, you're like, oh, god. Like Maybe there's some sort
• 1:05:51
  Robert "RFK" Keith: of power noise
• 1:05:51
  Eric Aasen: or something.
• 1:05:52
  Robert "RFK" Keith: Good. Not nothing about this is gonna be cool.
• 1:05:56
  Eric Aasen: We're like bringing the kitchen sink. I'm bringing my VNA, Robert's ordering a signal source analyzer to look at clock jitter frequencies. Yeah,
• 1:06:05
  Robert "RFK" Keith: it's getting
• 1:06:05
  Eric Aasen: We're ordering all the things and bringing all the things we can to bear.
• 1:06:09
  Adam Leventhal: All the
• 1:06:10
  Bryan Cantrill: things we can. And CJ is like, I mean, is like, we need to go hit up electrical rent to get all this equipment. And we're kind of like, I mean, we're just like, and now, A you know
• 1:06:18
  Robert "RFK" Keith: huge shout out to CJ on that one. Woah. Like, calling people directly, keeping them up later than they should.
• 1:06:26
  Bryan Cantrill: That's then getting Microchip lit up and like getting a contact in there. Like, we don't think that Microchip is at all to to blame, but God, we would like just love to their consultation. And Steve's dinner party conversations have shifted from, do you know where I can get a PCIe clock buffer to who do we know at Microchip? And that's over. So then, okay, then you guys all go out there and now everyone's out there on Tuesday morning, the holiday weekend.
• 1:07:01
  Robert "RFK" Keith: That's great.
• 1:07:03
  Bryan Cantrill: Yeah, we all Friday and
• 1:07:04
  Eric Aasen: early Monday night, but we get another, you know, Tuesday morning, and we're looking at it, and I bring all the things for probing, all this stuff.
• 1:07:12
  Robert "RFK" Keith: Set it all up. Looks really nice. It great.
• 1:07:17
  Eric Aasen: And we clock just sits hot.
• 1:07:20
  Robert "RFK" Keith: Yeah. I think that was the that was the really good point where, like, hold on. Why don't we just measure this part? Just make sure that we're all on base We
• 1:07:35
  Eric Aasen: turned on the clock input or the input termination because if you limit the bandwidth to 500 megahertz, then you can see as a high impedance input to the scope, you can see the signal. And that was solid high. And as soon as you turn on 50 ohms, as you go up to eight gigahertz, then it goes and it's perfect. I'm like, are you kidding me?
• 1:07:56
  Bryan Cantrill: Yeah. And
• 1:07:57
  Eric Aasen: so Robert measures them.
• 1:07:59
  Robert "RFK" Keith: Yeah. So get the DMM out. Like, you know, no big deal. You put you touch both sides in this very teeny tiny part.
• 1:08:06
  Bryan Cantrill: I'd I'd also like to say that I have been like when the whenever we've like blown anything out or whatever. Robert, I've been with you when the crops have failed and you're just like, well, DMM comes out. I mean, it's just like, okay.
• 1:08:18
  Robert "RFK" Keith: It's like broken. So what are you gonna do? Are you just like, poke around? Like, whatever. Yeah.
• 1:08:25
  And, yeah, you're right. The DMM comes out and you touch both sides of tiny little part and it reads like 25.6 ks. You're like, oh, that's not, you know, like close to 50. Wait, wait, wait, wait, let measure another one. We're just gonna be like, sure that we're not, you
• 1:08:46
  Eric Aasen: know We just happen to be measuring the wrong phone.
• 1:08:49
  Bryan Cantrill: Right. What do mean?
• 1:08:50
  Robert "RFK" Keith: Like, that that's obviously wrong. That's that's me for sure. Like, that's weird. Right. Measure, like, another one, different clock, different output.
• 1:08:58
  It's like 25.6 k again. And I think I look at Eric at this point, I'm like, oh, oh, we this could have
• 1:09:07
  Bryan Cantrill: been Okay.
• 1:09:08
  Robert "RFK" Keith: So We've we've done something wrong.
• 1:09:11
  Bryan Cantrill: Okay. So are you thinking like overwhelming relief at that? I mean, because that's gotta be like
• 1:09:16
  Robert "RFK" Keith: Well, it's like, yeah, but also
• 1:09:20
  Eric Aasen: also How? Like Right?
• 1:09:22
  Robert "RFK" Keith: Because it's like you pulled this part from stock from an inventory system And you're like, no, that's gotta be for sure what we told them, because it's a managed inventory system. They pull reels from these things to make everything. You know what I mean? You don't get the wrong number on accident. So there's like that fear.
• 1:09:46
  It's like super not cool. And then also the, well, this problem might not be as bad as we thought. So dichotomy.
• 1:09:56
  Bryan Cantrill: And it turned out to be that it was not a 45 ohm resistor. It was a 44 ks ohm resistor.
• 1:10:08
  Scott Tagwerker: And
• 1:10:11
  Eric Aasen: there is one important digit missing in the description that was copied to us. It's not even a digit.
• 1:10:18
  Robert "RFK" Keith: It's it's just the letter. That's what all letter.
• 1:10:20
  Bryan Cantrill: It's all it's just a one little letter. How important can a letter be?
• 1:10:24
  Eric Aasen: It's If he orders a magnitude Yes. Between Amazing. It was like a cacophony of failures on all sides, The description was copied incorrectly, but the part number that was shown right above the description was correct and had the K in it. Oh yeah. So we told them to use these parts, and they did, and they were the parts they told us they were going to use that we requested, but they just happened to have the K in them.
• 1:10:58
  Bryan Cantrill: Well, I also I feel like, you know, we're asking them to, you know, rework things. I mean, it's like there's people are scrambling on this. Right. So it's like, it's kind of understandable you've got.
• 1:11:08
  Eric Aasen: They had reworked 30 of these over the weekend.
• 1:11:11
  Robert "RFK" Keith: Right. I With the wrong value. 30
• 1:11:13
  Bryan Cantrill: times 30 grains of pepper.
• 1:11:15
  Eric Aasen: Yeah. 900 grains of pepper replaced.
• 1:11:19
  Bryan Cantrill: Yeah. Okay. So you, I mean, you get, so we're like, we're gonna take one of these boards and rework now off these, the resistor that is three orders of magnitude, more resistance because it's effectively like no different than not having a resistor there for all practical purposes. Getting this, the resistor in there with the proper resistance and then it all worked. Is that a That's what it looked like from Chad anyway.
• 1:11:48
  If there were other drama along the way, it all worked as expected at that point.
• 1:11:53
  Eric Aasen: Mean, as expected, know, given that, you know, anything that can happen bad, it will. The yield that we have from this manufacturing site is phenomenal. The first pass yield is above 99%. And we, of course, end up with two boards that have a quirk in them that is not really the right, but make them fail our tester. And we're like,
• 1:12:16
  Bryan Cantrill: oh, shit.
• 1:12:17
  Eric Aasen: What happened? It's like, oh, no. There's just something else wrong with these.
• 1:12:20
  Bryan Cantrill: There's like a literally unrelated failure. I they Completely different.
• 1:12:26
  Eric Aasen: Completely unrelated. Well, one of
• 1:12:28
  Robert "RFK" Keith: them was the best one was we're freaking out so hard that one of them had failed. They didn't even read the like the failure tag that someone had already put on there. Like they they already identified what the problem was.
• 1:12:39
  Bryan Cantrill: Oh, that's funny.
• 1:12:40
  Eric Aasen: We we were like,
• 1:12:41
  Robert "RFK" Keith: why isn't this working? And then like, oh, that's not bad. Yes. We did. Yeah.
• 1:12:46
  And it's like a SolderBLOB or something in there. And you read the tag, like, after we got done all about failing the test. It's like, SolderBLOB in in four, like, oh, okay. Well, we're just really dumb.
• 1:12:55
  Bryan Cantrill: Okay. Whoops.
• 1:12:57
  Robert "RFK" Keith: Maybe we should read
• 1:12:58
  Bryan Cantrill: it. I and then so we, mean, took advantage of having the equipment in there today, like, to to really measure it and to be able I mean, we we we took advantage of but I I mean, you must have felt this way as well. I mean, an overwhelming sense of relief that like we are we're gonna live. We've got all the the because when the line is down, I mean, is the the the great thing about making hardware as a product is like, it is God's own open source revenue model. Like people pay for hardware.
• 1:13:29
  The bad news is when the when the line is down, the revenue is zero. Until like, you know, not to be as Steve nods emphatically like the you need to be the line to be up. So That's not great. Yeah. It's not great.
• 1:13:47
  But this was I mean, I just felt like this was heroic work. And so now, I mean, Scott, we've been able to, I think, additionally be able to secure inventory now from what is now CyTime because they've business is going from us on CyTime. And we've got this alternate that actually works.
• 1:14:08
  Scott Tagwerker: Yeah. So very a couple of very positive developments. But yeah, we it was interesting timing of very shortly after that, we finally, you know, got some news and the new contacts we've developed at at CyTime, we're making progress with, come and said, hey. Good news. We're gonna get you what you asked for in in q four, and the linearity looks like it's gonna be relatively soon.
• 1:14:27
  And it certainly didn't, make the the exercise or, you know, this qualification unnecessary because it's still gonna be you know, that would would have resulted
• 1:14:35
  Bryan Cantrill: in a month,
• 1:14:36
  Scott Tagwerker: you know, with three weeks of wind down.
• 1:14:37
  Bryan Cantrill: Yeah. We would only be lined down for six weeks. Like, over yeah. Right. Exactly.
• 1:14:41
  Scott Tagwerker: It is a nice way of, you know, a brow wipe to say like, hey. Now we've got a couple of options and, you know, hopefully, can avoid something similar in the future.
• 1:14:50
  Bryan Cantrill: Yeah. And I thought it also an interesting object lesson in terms of like, you know, an alternate is like very I mean, there's a lot of like, the details really, really, really matter. And it's just really easy to have something like this. And then in terms of debugging it and being able, I mean, it was And you know, one of the things that I kind of identified, Adam, to get us back to the top in the talk that I gave him Oktoberfest. You know, I went through some of the near death experiences.
• 1:15:25
  Eric and Robert, you were involved in quite a few of them when we were building Gimlet. And, you know, the times when like a single resistor was the difference between life or death. Robert giving you nightmares for the strength of a pull down resistor needing to be and termination resistors and so on. But the I I one thing is I'm pointing on that talk is when you have something that should work and doesn't, that's when you've got to like really, you know, check all of your priors. And we had something in that reworked board that really should have worked and didn't.
• 1:16:00
  The oh, one kind of actually final bit I wanted to to circle back on is in terms of actually like making this change because we obviously don't wanna rework every board that comes off the line. I mean, it's it's enormously expensive. We are so the other thing we think we got gotten lucky about is being able to make a stencil change. Is that right?
• 1:16:17
  Robert "RFK" Keith: Because so you have a stencil for when you do reflow for a board so they can apply paste, the solder paste that you need to the board in the correct places and not anywhere else. Right. So what's nice about that is can, they're usually like quick turn shops where you can like go and make a new stencil or where the factory will have their stencil made. And what's great is we can send them a modified CAD of our board, and then they can take that where we can outline exactly where we want to put the two zero one resistors, Right? And so we can put that into our design, send that to them, the factory, and then they can go and give it to their little quick turn shop where they can put a whole new stencil together.
• 1:17:05
  And that way they can start because of the way that these resistors can be placed over the vias, they can just run this process through normal reflow, which means that you don't have to have people out there putting these things down by hand, which is what they were doing, which is ridiculous. No one should be having to do this 30 times for how many thousands of boards do we need to make. That's an asinine, right? You want this to be part of reflow or else the fix really it's like, yeah, it's a fix, but that's not cool. That's not like a real manufacturable thing.
• 1:17:40
  Because we were able to make a stencil and they got the stencil back in like two days, something like that. That means the time that we were figuring this out, they were able to send it out when we got there, get the stencil back, and by the time we validated it, they're ready to run-in a regular manufacturing flow with this fix applied, which kind of solves the whole throughput problem that we're now faced with. After we fix the problem, we're like, Oh, guess what? Now you're behind. All that time you spent that you were supposed to be making boards, you need to do all that, but way shorter because you gotta ship back
• 1:18:20
  Eric Aasen: to that. Right.
• 1:18:22
  Bryan Cantrill: Yeah, right. And I did when you did break through on this, I mean, you know, I know the Internet says that we are actually a podcasting company using computers for content generation. And boy, we have some things like this. It's like, I think they might be right. Because I mean, Adam, this is amazing content.
• 1:18:45
  I mean, this is like Crazy. Extraordinary. So, of course, I'm like, hey, can I get a picture of this as we're kinda getting? And they're like, okay. Yeah.
• 1:18:53
  You want a picture for the podcast episode? I'm like, maybe I do. So what? Yes. But so Robert sent me this picture.
• 1:18:59
  I I I just put it in the chat. So the that that is a tip of a pencil pointing to those two zero ones. Adam, if you can see that. Yeah. And so the one way to think about that is you see that there is a there's a diameter there to that.
• 1:19:19
  That's obviously under a scope. That diameter is about the diameter of a double a battery. And the actual part that we're talking about that microchip part, can see up at the top of the frame. So that's four millimeters by four millimeters. That's the ZL4264 is the microchip part.
• 1:19:37
  So it's just, the whole thing is just amazing. It's the I mean, it is truly for want of a resistor. You do not have a computer company. You got like you've got, you know, Steve, we keep thinking we're kinda like past the point of like, oh, a resistor is actually standing between us and death. But we we were definitely on the operating table seeing the light on this one.
• 1:20:07
  So this was really just extraordinary work. And I mean, you know, we we say this over and over again, but God, the teamwork from everybody working together. And, you know, Scott, I I I the so much of what you're doing is this like a delicate balancing act between like you're like half commodities dealer, half doubly, half like I mean, you're you're balancing all of these different things in this problem that is not just over constrained, but is whipsawing dynamically all the time. And it's just And the Yeah. And Eric's saying in the chat, Centimeters, Benchmark Electronics, huge shout out to them.
• 1:20:50
  Honestly, terrific through all of this. And their rework techs are extraordinary. Then also, honestly, Cytom, Renaissance, Microchip, everybody wants to get us through this. Like, everybody is trying to everyone's rooting for us. So No.
• 1:21:09
  Steve Tuck: They all were. And, again, I think this was, it is hard to overstate just how remarkable the execution of the teamwork was over this like five day period. Yeah. And we talked about the stencil, like the team had the conviction and kind of the disposition to get those stencils ordered without waiting around for a bunch of things to get sorted in the event and that's why it took that's why it was so quick.
• 1:21:34
  Bryan Cantrill: Yeah.
• 1:21:35
  Steve Tuck: And whether it was, you know, rental companies that we work with for scopes, all the different teams across benchmark that were just exceptional. And and then obviously across the Oxide teams. A huge shout
• 1:21:49
  Bryan Cantrill: out to Amy too. Was not Amy on the Oxide team was the kind of quarterbacking all this and went on. I'm just like, it was, really heroic.
• 1:21:57
  Steve Tuck: Yeah. It is it is very much one of the benefits of having built like a very integrated trusted relationship with Benchmark.
• 1:22:06
  Bryan Cantrill: Yes. And I think
• 1:22:06
  Steve Tuck: you there there are many relationships out there where folks kinda treat the Centimeters as very much a contract manufacturer to go do the bidding. And I think we we you know, this is just another case of like really appreciating the the the depth of the relationship and kind of the trust across the teams because you just saw so many different folks in different places marshaling and ran the gauntlet.
• 1:22:32
  Bryan Cantrill: Ran the gauntlet. Really, again, terrific work. And Robert, thanks for hopping on a plane. And Eric, thanks for hopping out there. I mean, it's just like everyone just did whatever needed to be done.
• 1:22:43
  And think we know that now that the gods are lurking in the all hands and we just, it's on all of us to keep our optimism bridled. So, that's not, I'm looking at myself. I'm looking at myself as much as anybody.
• 1:23:01
  Steve Tuck: Shared endeavor.
• 1:23:02
  Bryan Cantrill: Shared endeavor, exactly. All right. Well, want On the way to extinction, the AI is gonna have to figure out all this shit. So good luck to them. Good luck, buddy.
• 1:23:13
  Yeah, exactly. Good luck. Meanwhile, we're going to be there, know, reworking your boards with the wrong resistors. See if you can figure that out because you very much relied on not just human hands, but extraordinary ones. So thank you everyone.
• 1:23:31
  We are going to be off next week, Adam. We are not off. We are off on the spectrum of podcast, I think, because we we've got Oxide Con, so we're going to have the team here. So but back I think in two weeks. Perfect.
• 1:23:46
  Awesome. All right. Thanks everyone. See you next time.