Latest / Oxide and Friends / The Power of Proto Boards!

Transcript

The Power of Proto Boards! · Oxide and Friends

• 0:00
  Speaker 1: Alright. So we got a bunch of folks here, and actually before we get going, I want so, Adam, I've got a photo album that has, all of these boards that, I'm gonna share in the chat. And that can help guide some of this. But and, and actually, go do that. And so if if you're catching this in the podcast, you're gonna need to, we'll put it in the show notes, what have you.
• 0:28
  But this will be, an album that will allow you to see all of what we're talking about. And maybe, we can actually kick it off. I I I kinda wanna Adam, I don't know if we wanna go in in chronological order or reverse chronological order in that. Do we wanna start with Nathaniel's demo on Friday and then jump all the way back to the cliff at the beginning of time? That's what
• 0:51
  Speaker 2: I was gonna say. Let's Let's let's start with the end, and then and then, and then get back to the beginning and work forward. I love it.
• 0:57
  Speaker 1: Perfect. So, Nathaniel, would you mind you're here. Would you mind describing what you demoed on Friday? And I would love to get we're gonna have to get a video of this up because it was pretty neat. Sure.
• 1:07
  Speaker 3: Yeah. So, what I demoed was, a pick and play an open source pick and place in my house, like, running behind me in my electrical lab here, in the middle of Wisconsin. And, I was building 1, like, a series of the small protoboards that we had originally had a a supplier build for us, back, before I even joined Oxide, so, you know, more than a couple years ago. And those have, you know, had a great life and been consumed in various fashion. And, it turns out that those, that's our our ROT chip.
• 1:46
  And the ROT chip that we had at we could source at the time didn't have the secure pieces, which is what we actually need for our ROT. So we're building a new set of those to go, distribute to the team with the chip that actually has the secure stuff that we have managed to, you know, finally be able to buy. And so in in my lab, I have, I've got a stencil printer that I use to put paste down, and, one of my coworkers, Cliff, had, had this board panelized for us. And so we have a 9 up panel. This is a small little board.
• 2:18
  It's like an inch and a half by an inch maybe. And, but it's got about 40 parts on there. And I've done a number of prototypes, for oxide here, in the last couple of years with, mostly, like, my own pick and place. So, like, me with tweezers setting each individual part down. And at a certain scale and a certain BOM count, that kinda gets annoying.
• 2:44
  And so this this is one where you got about 40 parts on the BOM, and, you know, we wanna do a couple of panels of these. And so, you know, you're starting to look at, like, nearly 800, manual placements. And so, I have separately been working on getting the Lumen PNP, which is, an open source pick and place by, Opulo up and running, in you know, it's kinda like grown up Legos putting all the stuff together with, like, aluminum extrusions and 3 d printed parts. And so, the inaugural run for that was, doing these ROT carrier boards, and so I showed that on Friday.
• 3:19
  Speaker 1: Which was just mesmerizing. I mean, a pick and place machine, I think, is always mesmerizing. It just it so the they always kind of almost slack job watching pick and place machine, but Nathaniel, that was great to watch at in I mean, right behind you, and then you were you were showing kinda leading up to that, showing how you could kinda go to any ref desk or you could go to any location on the board and watching the camera whip around. It was just really, really cool.
• 3:44
  Speaker 3: Yeah. And having robots is fun. Right? So Having
• 3:47
  Speaker 1: robots is fun. It you know, I'd like I I think this is, like, where my inner Calvinist really shines through where I'm, like, not into robots for things of, like, for, like, company. Like, I'm not into, like, the Jetsons vision of a robot, but I'm definitely into robots doing, like, hard physical labor.
• 4:09
  Speaker 3: I think it's amazing. And I think, like, repetitive boring stuff or I mean, you know, like, I could have hand placed this board. I mean, there's nothing that that huge on there. But, like, that's hours of time where instead, I can just, like, let the thing go and put all the little resistors down and, you know, not worry about it. So repetitive, boring things robots are really good for too.
• 4:29
  Speaker 1: They are really
• 4:30
  Speaker 2: The this pick and place machine, this is it OpenPnP? Or
• 4:34
  Speaker 3: Yeah. So OpenPnP is the software. So that's like a Java app that is running, and it's that's multiplatform.
• 4:40
  Speaker 1: You can get it on a
• 4:41
  Speaker 3: bunch of different platforms. And that's been around for quite a while. And then, kind of like, you know, if you look at, like, the 3 d printer industry, there's been a lot of, like, community development and trying to get, you know, various printers, be it, you know, like people doing mods on an ender or people doing the Voron printers or what have you. And so, Steven Haws, who's the guy who founded Opulo, started this off kind of as a side project from as I understand it, and, wanted to like build an open source, pick and place piece of hardware. And so they have custom hardware.
• 5:16
  They have, a custom circuit board that runs the thing. It runs Marlin, which is a a 3 d printer firmware that's open source. And, they 3 all they did all the CAD, all the CAD is up in free CAD, so you can go download it and modify it to your heart's content. And it's it's out there.
• 5:34
  Speaker 2: And, Anthony, I know it's gauche of me to ask to talk about money. But about how much is like, what are we talking to build this thing?
• 5:41
  Speaker 3: I think you can buy their kit right now that's probably back ordered for about 1700, and that comes mostly assembled. So you have to put, you know, like, 3 or 4 subsystems together as I understand it. I was an early adapter, adopter, and so I spent less money but got the opportunity to put it all together myself. And and and you do have like, if you bought the early ones, you actually had print all your parts too. So all the parts are printed.
• 6:10
  And so I had to print all that too. And, you know, the that isn't very expensive, but it's, like, time consuming, especially if your printer bed is fairly small.
• 6:17
  Speaker 1: So you say adult Legos. You mean it's like step 734. Print the following 3 pieces. Yeah. Yeah.
• 6:24
  Okay.
• 6:25
  Speaker 3: Pretty much. I mean, they they have a a bomb, basically, with a whole bunch of STLs. And so the STLs are the, like, the 3 d data, basically, with dimensions or with, you know, like, an assumed dimension, basically. And so you run those and you drop those into a slicer, and, you get, g code.
• 6:43
  Speaker 1: And how hard is it to get the software to work in all this with? I know. Because I I do feel like, Adam, Gary, you know, are proud of, like, getting, like, clear audio working meanwhile. Yeah. It was building up fast.
• 6:53
  Speaker 3: I mean, it's, it's hours. I mean, it's it's many, many hours. You know, if you think about, for people who have played with a 3 d printer, a lot of times you spend a lot of time kinda getting that dialed in. You so you have all the same kinds of motion axes here, but then you have, vacuum and vacuum sensors, and you have, additional lights and then a whole vision system. And, you know, these the vision system uses these little, either like little USB cameras that have a fisheye lens.
• 7:22
  And so you have to put them in and do calibration so that you, like, defisheye the image because OpenPnP uses vision a lot, which is kind of the big difference between the 3 d printer's world where vision isn't used quite as much. This thing has a top camera and a bottom camera. And, you know, so it can look at where you're where you're picking the parts up and where you're putting them on the board, and also it can fly them over the bottom camera and look at how the part got picked on the nozzle. So there there is a good bit of messing around trying to get things dialed in. And, you know, there's like the mechatronic system and then also the vision piece.
• 7:58
  And so, OpenPnP is pretty cool, and there is, you know, there's a helpful Discord channel that the Opio guys are running. And, lots of people have been doing mods and help in there, so it's, you know, a useful place to reach out. But it's mostly just time and distance.
• 8:14
  Speaker 1: Yeah. Right. Well, that is really cool. And, of course, like so aside from the fact that it is extraordinarily cool, why do I care about this? Why do this?
• 8:24
  Speaker 3: Well, you know, you get you can kinda do things on demand. So, like, that's a useful thing. And, you know, like, I mean, as we'll get into, I think, tonight, it's not an infrequent for us to decide we want to make something and then, like, sort of want it the next week or the week after. And and some of that maybe is poor planning, and some of that is also maybe just, you know, we recognize the need, and it's like you wanna turn that need into a fulfilled reality. And so, I mean, I feel like, know, we we'll go through the boards, I guess.
• 8:56
  But, like, Gimlet is a good example of that. I think Cliff, you know, had kind of an idea one afternoon. And then, you know, 24 hours later, it's out for fab. Right? So
• 9:08
  Speaker 1: It is remarkable. And and the ability to go out for fab, so that means to go I mean, with the you know, you you actually haven't done your own fab, so that still needs to go to a to a PCB house. Right. But and you get turnaround on that can be, you know, a week. It's something like that.
• 9:22
  Speaker 3: Yeah. It's I mean, you know, it depends on how much you wanna pay. Right? You can do it stateside, and you pay more. Or you can, you know, ship it off to a PCB way or a JL PCB or whatever.
• 9:31
  And, you know, like, I think Eric got, like, 50 gimlet boards back that are 4 or 6 layers with controlled impedance, and, you know, you get, like, 50 fabs back for $300 or something. It's not that expensive.
• 9:43
  Speaker 1: God. It's amazing. And then but then there's this assembly piece, and that's what you're automating with your own TMP machine. And but the ability to then not not for these small boards, we can go from ideation to hardware in hand, like, functioning in a remarkably short amount of time, a week Right. Change basically.
• 10:03
  Speaker 4: Yep. And Yep. Go ahead.
• 10:06
  Speaker 1: You know, when you when you when you get that tool in the arsenal, when you begin to, like, have that superpower, there are a bunch of problems that it becomes really tempting to solve that way because it's like, okay. What I actually now now there are a bunch of problems I actually didn't realize I had, but now that we can solve it this way, let's go solve it.
• 10:21
  Speaker 3: Well and I think something to remember, especially about, like, our organization is we're not all in the same, you know, region or physical space. And so, like, walking down to the lab to go play with the prototype is not actually an option for a huge portion of our company. Yeah. And so, and, like, if you look at our equipment, I mean, like, a Gimlet is cool, but, you know, it takes 54 volts, and it doesn't have a NIC interface, and it, like, it's not it's not something that's, like, that useful to somebody sitting on their bench, and it's noisy. And, may with the 2 k fans, it's not so noisy.
• 10:57
  But, you know, it's it's like it's not some and then they're also kinda expensive. So, you know, you're you're looking at, you know, many 1,000 of dollars just for, like, processors and whether you want, you know, SSDs or DIMMs and all of that. And for, like, a large portion of our company, the people who are working on a subsystem, they don't actually need all of that other stuff. And so, you know, we've been able to be pretty successful with some of these little boards to to, like, break out subsystems into these boards, and then they're relatively inexpensive. We can build them ourselves even in some cases and, you know, ship them all over the country very cheaply.
• 11:30
  Speaker 2: Is this a good point to rewind back to some of our our first proto boards, our first development boards?
• 11:36
  Speaker 1: Yeah. And I I would so I was gonna take us back to Gemini, and and if you if you got that album, I did learn the hard way that I cannot be at least not easily, I can't be in Discord on the laptop and on the phone at the same time. So, I did drop the link, but then I had to I had to run to get back over here. But hopefully folks have got that link. The and the the the first photo in the album there should be, Gemini, which is the first board we did, and, Cliff, you led the charge on that, and I think that our experience with Gemini informed a bunch after it.
• 12:09
  Do you kinda wanna talk about the the genesis here and and how what we learned from doing Gemini?
• 12:16
  Speaker 4: Sure. So I think of building these little modules, like, it's sort of the hardware equivalent of unit testing. You wanna write little things and see if they work in isolation, and then if they don't, you throw them away, you make a new one, and you repeat this until you've got a library of useful things. So when you're when we set out to build a server from scratch, you don't really wanna go straight to building the whole server motherboard. Particularly because at the time, we had a total of, 0 full time electrical engineers at the time.
• 12:48
  So there's a sort of there's a process I learned from robot people at a previous job. Some called a roadkill build, which is where you get all of the parts that will wind up in the eventual thing, but instead of being tightly integrated, they're all spread out over a bench with nasty cables between them so that you can probe them and poke them and replace them as needed. And so we thought it'd be a good idea to do that for the server. So we started with the part that was simultaneously the best understood and also the scariest, which is the root of trust slash service processor initial few instructions that run on boot end of the system, which we had at this point recently decided to do with a combination of 2 off the shelf microcontrollers when all of the previous things we tried fell through. So, and Gemini worked okay, but honestly, I think the main thing we learned from this is that it is entirely possible to bite off more than you can chew in these prototype circuits.
• 13:48
  In particular, Gemini was just too highly integrated. There were too many moving parts. There were too many committed IO pins to certain purposes. It was there was just too much on the board, and we could've turned it around faster, and it would've had a better longevity if we hadn't built all of those things in there as fixed functions.
• 14:06
  Speaker 1: Yeah. I think that was a big point of education. And, I mean, to be said, the Gemini worked. I mean, it was amazing. Sure.
• 14:12
  And the, the actually, this Gemini, this is this Gemini is always I wanna be buried with this, like, Adam, by the way. If I so if should she survive me,
• 14:21
  Speaker 2: you know, the on the show notes?
• 14:24
  Speaker 1: Yes. Please please bury this board with me because, the the the this board is special to me for a lot of reasons. It's serial number 1. It's the first thing we made at oxide. Cliff had done the bring up of it.
• 14:34
  Cliff and I were in the office doing the bring up. It's 51 degrees in the office because the heater was broken. And this particular Gemini, which and and the software came up. I I mean, it was great. The software came up.
• 14:44
  The hardware came up. Basically, everything in this board works. So that but Cliff's point about the Tite integration was it just took a long time. It took a long time to to get it reviewed and right, and, you know, we found a bunch of things in review that were really important. We wanna get them fixed.
• 14:59
  But this particular also was if it looks reworked, it's because it has been reworked and not because of any fault of the board. Mhmm. So the, you'll note the the the header, the the the header, was exposed pins before. There's a header at the bottom. The the we're never gonna need this header, was exposed pins.
• 15:23
  And, over shortly after I had this thing at home and was working on it and had some confidence in its robustness, We did have a moment where we were petting the cat, talking to the family over the holidays. This is in holidays of 2020. And at some point, I moved this board to be further away so it didn't get knocked off the table. And then only later realized that the board was no longer responsive, right, which was rather upsetting, and the I had actually managed to zap the board. And so Cliff not only reworked the board to kinda solve the UX problem of changing these exposed pins to a header that couldn't be zapped, but that that s t m 37 the the s t m 32 h m 53 in the middle there has been reworked by Cliff.
• 16:07
  It's just amazing that that that you reflowed this thing back onto place. Incredible. Well, I mean But I think you just
• 16:18
  Speaker 4: static Sorry. Go ahead. Oh, the
• 16:22
  Speaker 1: It's good.
• 16:22
  Speaker 4: And it's it's also worth noting. I think Brian is the only one that got a, a board with the, quote, unquote safety header at the bottom. Just because the rest of us either don't have cats or have grounding straps at our workstation.
• 16:42
  Speaker 1: Fair. I feel like a couple of these been reworked, but maybe that's true. I still, anyway, this board, I still am very emotionally attached to, and not least for, I love all of, cliff and Rick's 2 delightful kind of jokes and and wordplay on the board is is terrific.
• 16:58
  Speaker 5: Wait. Wait. Certainly, like, going back to the we put too many things on this board. Like, the the concept that we ran into was, hey, an x86 server with all of its management and and everything is way too complicated. Let's slim it down.
• 17:13
  And so we initially, we're slimming it down to just the management subsystems. Right? Like, this is intended to be the service processor, and the things that it would talk to and what we expected are plausible choices for gimlet or x86 server. And it just turns out that even that's too much, right? And and by having it be all on one board instead of being pluggable modules, it meant that we committed to a lot of design choices where, in fact, what we were trying to understand was are these actually the parts that we plan to use?
• 17:45
  And it turns out, in a lot of cases, the answer was no. Right? And at which point more and more of the system became less useful. For example, the 2 Ethernet jacks on there never actually used.
• 17:56
  Speaker 1: Right.
• 17:57
  Speaker 5: Because we we changed our philosophy on how that was gonna work before we ever actually got boards back from being manufactured.
• 18:06
  Speaker 4: Yeah. I did manage to switch traffic from one port to the other once, but I don't think it ever went to the SP.
• 18:15
  Speaker 1: So and then the the the next image in there is part of what I also loved about this, and this was like that a little bit of modularity where the the on this first image, the Root of Trust is actually not on here. The Root of Trust is in a separate carrier board, that, you'll see on the on the next image, which I think was some of that budding modularity. Rick and Cliff, I know, Rick, you you had kinda taken this piece, but it was, I think we saw some of the power of the modularity by actually having separated that out. Even though we did it, I think at the time we did it because we were convinced we were gonna be bricking a lot of these LPC 35s.
• 18:53
  Speaker 5: Exactly. I mean, the root of trust, part of the part of the the design goal there is that you have first, instruction integrity. Like, I know what the first instruction is going to be, and I know how I got there, and I can trust that implicitly. And so that means, in this case, setting up the secure boot and permanently locking the device, which meant you actually have to test out that functionality. And so that was the, Hey, we'll just make these as little add in cards that are low pin count.
• 19:23
  You can't really use it outside of the system. Right? Like, it has to be plugged into something to get power and and other useful signals and, like, the debug header isn't or the the, SWD header isn't even there to be able to attach a programmer module to it. You you have to do that through the Expansion Connector, but it meant it was a very low complexity board. It was very simple, so that we could make a bunch of these very inexpensively.
• 19:50
  We also could rework them if we needed to as we destroy destroy parts. And like I do actually have a small collection of bricked parts from when we later did do exactly what we thought we were going to do. So you know, that that was the initial thought was more on the, like, I'm gonna have to do destructive testing, where everything else on the Gemini board was a there's no reason we would do any destructive testing. We just think that this is probably what we're gonna use. Oops.
• 20:21
  That's not what we actually ended up using.
• 20:23
  Speaker 1: And, also, I ended up doing destructive testing accidentally.
• 20:26
  Speaker 4: Right. Yeah. I was I was gonna point that out.
• 20:30
  Speaker 1: So, but these were both, very, I think, educational boards and really beginning to see some of the power of being able to iterate quickly. Then starting in early 2021, we had, Cliff mentioned that when we did these first two, we I had no full time double e's of the company. We were added a bunch in, in early 2021. And, Eric, I feel like the the this next board, which is the the the power module for Get More. I feel like this is I'm probably getting the chronology wrong here, but I feel this was definitely one of those first things where you feel like we have to have the ability to power this thing on from a stock power supply.
• 21:07
  We cannot rely on a bus bar to power this thing on.
• 21:10
  Speaker 4: I I do see what
• 21:11
  Speaker 1: the history of this part is.
• 21:12
  Speaker 4: I think there's one additional piece of context here that's important, which is that if you think about the years that we're saying so January happened in late 2020, real engineers real electrical engineers showed up in 2021 and, both of these years were pretty much the worst electronic supply chain disruption, I've ever seen. And Yeah. So, like, in the case of Gemini, we were redesigning the thing repeatedly right before it went to fab because the parts were we were playing whack a mole trying to find any parts that were in stock and would stay in stock. Like, we couldn't get connectors. It wasn't just integrated circuits.
• 21:49
  So, yeah, it's it's been really entertaining.
• 21:55
  Speaker 1: Yeah. And then it was a big constraint on, obviously, that that we had to be and it it was nice to have that agility to be able to switch things up, but unfortunately that we had to had to take advantage of it.
• 22:05
  Speaker 6: Quick shout out to some of our ops people who really went the distance with getting some of those parts and just, like, doing magic. Mhmm.
• 22:14
  Speaker 1: Yep. Yeah. And if you haven't yet, great one in the the back catalog of Oxide and Friends is, Kate Hicks on all of the magic that she and her ops team pulled for, solve some of the supply chain issues. So, Eric, at this board, as I recall, like, this was something you did very shortly after arriving at oxide. Am I remembering that correctly?
• 22:34
  Speaker 7: Yeah. It was, I kinda looked at the the gimlet board design. I'm like, How are we gonna power this on a bench? Does anybody is there, like, some dongle somebody has? Or it's like, oh.
• 22:47
  Oh, okay. I'll I'll go make something.
• 22:49
  Speaker 1: Okay. I'll make one of those.
• 22:50
  Speaker 4: How about that?
• 22:51
  Speaker 7: And so I picked the, yeah, the the mating connectors for these particular things. Like, they're very inexpensive, but they're insanely hard to get sometimes because they're basically all bespoke ordered from Amphenol or whatever. And so you have this, this lovely lead time you have to deal with. But, basically, I threw this thing together of, okay, I'm gonna put a right angle connector on here, and I'm gonna put the biggest freaking Phoenix terminal block I can get my hands on apparently. You know, it always looks smaller in the catalog.
• 23:23
  Right? And I didn't bother making a 3 d model of it. And then I
• 23:27
  Speaker 1: put a
• 23:28
  Speaker 7: current sense resistor and soldered it all together, and I'm like, oh, good. Now we have a way of powering up the boards.
• 23:34
  Speaker 3: I I think my favorite for this for this board generation though is the fact that every one of these had to be trimmed.
• 23:40
  Speaker 7: Oh, yeah. Yeah. So it was originally designed to have a straight connector on it, but then I realized I couldn't really get the straight connectors very easily, and we want it right angle. And the right angle bore or connector hangs down from the board edge a little bit, and so that interfered with the board. And so I took some, some metal snips, some tin snips, and sheared the edge off of every one of these boards by hand, which is why they're looking all a little janky if you look at it.
• 24:07
  Speaker 3: Yeah. So yeah. Every one of these on the Power Blade side of it has a not quite straight edge.
• 24:15
  Speaker 1: But this is a good example of a board. I mean, this was a a rather essential board for us, and this is a
• 24:20
  Speaker 3: small This may be the board we made the most of.
• 24:23
  Speaker 7: Yeah. The
• 24:23
  Speaker 1: most of. Most bearing.
• 24:25
  Speaker 3: Yeah.
• 24:25
  Speaker 7: Yeah. Turns out.
• 24:26
  Speaker 3: Yeah. I mean, every gimlet we powered up until the point where we got them in Iraq has used one of these. Right? So
• 24:33
  Speaker 7: I Totally. A newer version, which is much more svelte, which is later on in the album, but, yeah, this one was the it has it has served its purpose well.
• 24:43
  Speaker 1: It had and it has been it it to me, it's like one of these where the ability to make our own board and to do it quickly was really important. Like, I'm not sure without this board, it's I I don't wanna contemplate it. It just gets it gets nasty. I mean, you're dealing with, like I don't know what we would have done. It would have been a big mess.
• 25:00
  Speaker 3: And, I mean, I remember sitting there, during bring up in 2021 in October at the, at our the facility, like, hand soldering these things together so that we had we had things to power up our first units.
• 25:13
  Speaker 1: Yeah. They're that important. I mean, they are this this board is is, every get wet up until very recently was, had one of these attached to it. So a really, really important board. And then, in the next image, is that the same rev?
• 25:28
  Eric, you just got the
• 25:30
  Speaker 7: I got a little, it was late night, and I got a little loopy with the, the markings. Because, of course, I got the silk screen wrong. So the plus and minus, which you can see very faintly blacked out on the board next to the connector or backwards, so I had to, draw it on top of the connector. Which is
• 25:47
  Speaker 1: why I'm at least That's great. Well, so and this is another thing that
• 25:50
  Speaker 2: I love. So so we have an idea for it, and then we throw it into KiCa KiCad or OrCAD.
• 25:57
  Speaker 7: That was KiCad. KiCad. Yeah.
• 25:59
  Speaker 2: And then, it sounds like in some cases or in all cases, maybe are we we send it to one place, we're getting back the PCB, and then soldering it ourselves, or are there other cases where folks are kind of putting the chips down as well?
• 26:13
  Speaker 3: We've we've done both, and we'll get like, some of the boards coming up were assembled somewhere. Yeah. But the but these these like, these and the spy mux that we'll get to next are were all hand assembled by us.
• 26:25
  Speaker 4: This actually came up in chat and, like, one of the points that I made there was it takes a surprisingly large amount of effort to prepare all of the data that a third party assembly house needs to make a board for you. So if your total quantity of boards is like a dozen, and you're not putting 1 of the 4,000 ball BGAs that's coming up in the photo album, but I you know, we'll we'll get to that when we get to that. It's almost faster to just solder it yourself if you've got, you know, people that are willing.
• 26:57
  Speaker 7: Yeah. These take me about 10 minutes.
• 26:59
  Speaker 3: Yeah. And that's I mean, that's where, you know, as we go through these, you'll see some of the more complicated ones with higher BOM count. We had somebody else do because there's a kind of a tipping point for sure. But, like, a lot of these are, like, 8 parts or 10 parts, and it's just easier to just stick you know, you can buy whatever you can scrounge off the Internet and, you know, stick them in your lab. So
• 27:21
  Speaker 4: And the exciting thing about the pick and place,
• 27:22
  Speaker 1: which show I
• 27:23
  Speaker 4: is it changes the sort of tip over point for this.
• 27:28
  Speaker 3: Right. Correct.
• 27:30
  Speaker 1: Yeah. It it gives us a lot more complexity that we can do on our own effectively without going to a supply house and without going to a to a board assembly house at all. I also love this, and, you know, a theme that you definitely saw on on Gemini, and we will see this throughout these boards. Even part of what none of these boards will ever ship to a customer. It's supports that we only use for ourselves.
• 27:51
  And I have to say, I love, like, the jokes and the art and the, you know, I I I think it's it kinda reminds me of, like, the nose cone art. During World War 2, it's kind of like the I I I really you see a lot of personality on a lot of these boards. I certainly see it here. Speaking of personality, I think we'd be this and, Nathaniel, I think this this next board, I think,
• 28:20
  Speaker 3: somewhere along the design path, we realized that, we wanted to, we needed a multiplexer. We had this, like, multiplexer circuit in, that the SP, which is, you know, our the little ARM processor, and then the AMD processor are going to share a flash part so that we can do attestation and that kind of thing ourselves. And so we have this, like, you know it's not that complicated, but there's, you know, an analog mux in there and, you know, some level translation and some other stuff. And so, we we really wanted to be able to prototype that. We also had changed the flash from what was used on the Gemini board, at some point.
• 28:57
  And so it was like, we really need something to to actually prototype this. And so so we built we built this guy in order to do that. He kinda had the the circuit that that we intended to put on the gimlet. So I I kinda took the gimlet schematic out of ORCAD and dropped it into KiCad and, you know, put a few other little things on there so we could, you know, get in with jumpers and change things and, and do that. And and, like, I think this turned out to be a fairly valuable, piece because we realized, like, there were some issues with that circuit.
• 29:35
  And, so you you can see, you know, if you go to the next picture, Rick had Rick Rick had, a lot of opportunity to do some rework, and, I got, you know, some feedback that 0402 resistors aren't fun for everyone. And so, that was but, you know, like, on this board, for example, we could own we couldn't find, leaded flash parts, so we had to put a BGA on. And so, like, the, the u probably u 4 there, kinda in the middle is a square BGA part. And that's the only flash part that we could buy that was in the same family, and the right size. And so we just like, we did that.
• 30:19
  And so I did that in my lab with a stencil and, you know, and a a little toaster oven reflow.
• 30:25
  Speaker 4: And I
• 30:25
  Speaker 1: think, I mean, in this board, I think, is is a really interesting example where we're really trying to explore something very, very specific, like one very, small but important implementation detail that we want to explore different alternatives. We want to aim and the ability to get this thing spun quickly was really important. And then it was really important because it would also allow the software to be developed. And so, in addition to exploring the electrical characteristics and so on, we could actually get the software developed long ahead of time, ahead of actually having an actual game on hand.
• 30:55
  Speaker 5: Well, in this case, it wasn't even the software that we were concerned about. It was, like, what sharing a QSPY a QuadSPY, Flash part has some interesting analog challenges, because some a fair number of the the IO lines become bidirectional. And so how you actually mux that, how you do level translating on it, when in our case, I think it's the AMD side uses 1.8 volt, quad SPI signaling, and the SP uses 3.3, and you wanted to switch between them. We we just had enough questions about how it works and would it work correctly that that's why we built it. And then I took, like, and and actually hooked it up to an AMD reference system.
• 31:44
  I I removed the actual flash, like the the BIOS or EFI flash part from an AMD reference board, and then brought that out as to the AMD Quad Spy connector on this board. And that's why mine has all the wires on it is because, ultimately, we discovered it didn't work the way we thought it did. And, you know, we bodged around it until we actually understood what was happening at the analog level, at and made sure the digital level, and, like, we got to a point where we said, okay, it'll work at 33 megahertz. Most of that is actually limited by the fact that I've got this running over a whole bunch of extra wire that won't exist in the real design, but we're confident enough that I can actually boot the x86 machine through this mux, and I can switch the mux and have the SP take over. It was very valuable to, you know it seems like an almost trivial amount of logic and and parts to put together, but it saved us a huge amount of effort in terms of we would have had to respun Gimlet Yeah.
• 32:46
  I was gonna say it did higher server board.
• 32:49
  Speaker 1: Yep. This saved us a board spin because it would have been really grisly to get this wrong.
• 32:53
  Speaker 3: Well and I think, notably, the the gimletlet, which we haven't gotten to yet existed at this point, which is where, like, this became one of the gimlet let's, you know, first couple of peripherals.
• 33:05
  Speaker 1: Yeah. So let's go 2 images ahead. We'll come back to the neck in a bit, but I I do wanna go to the Gimlet Let because this is a really important board for us. So the the this green square board is, and and, you know, this is one of those, I think, Cliff, where some of the things that we've been kind of, you've been saying, we've been thinking like this board really expresses kind of all of that, that wisdom where you were like, I know what I need now. I need a board that has, like, I don't wanna have all this other stuff that Gemini has.
• 33:36
  I don't need any of this. I really need a service processor, and I need IO, and I'm just gonna go design this thing. And, Kupp, how long did it take you to go from an initial idea to having this board out to the board house?
• 33:51
  Speaker 4: Oh, I'm sending it out for fabrication, I think, was within well, so this is so there was an earlier gamelet lit that was designed to fit on a Nick test circuit that that only existed for a very brief period of time. This was the follow-up when I sort of, rage drafted the board because the thing I wanted didn't exist, which is how I wind up doing most of my things. But this was probably within 24 or 48 hours, I think, and then we had to wait for it to ship, of course, back to us from the board house, and then and then we had to wait for them all. Because I I I can only fit one of these on the $8 hot plate I use for reflow. So, we had to, had to wait for those to go through.
• 34:30
  But honestly, if I know we were gonna be making more than I mean, these boards are everywhere at Oxide now. And if I had a phone that that was going to happen, I probably would not have set the part number on the, inventory label to a long
• 34:43
  Speaker 1: no. Right. Yes. That's right.
• 34:47
  Speaker 3: I
• 34:47
  Speaker 1: And actually, it's I and I'm pretty sure that Gimlet went too, which is later on here. I'm not even sure that it's got a different silk. So, yeah, it is pretty funny. I don't think it knows. Right.
• 34:57
  But this board and I I mean, this board is so useful, Cliff, just to your point. So this is a photo of where I was doing some work as a temp sensor down there that we're not using, but I was using this test and other stuff. These things are workhorses for us, and we are so if you go back to that Spymux board, the Spymux board was designed to plug into this thing. And you that that's gonna be a theme where we are, designing things where we can and you'll see that that we've got lots of gimlet. Gimlet lets doing all sorts of things.
• 35:28
  The the photo of the the gimweltlet, later on, the black silk screen that maybe I don't go into because we actually, had we we did have that mass assembled because we did the cliff. You'd assemble this one on your own, but, we it became clear that these are gonna be really useful, and we're gonna use them in lots of places, and we're gonna make a ton of them. It felt like the $8 hot plate, maybe we could, save you the burden there of doing that. But these things are just in use all over the place.
• 35:56
  Speaker 3: And and notably, they're in heavy use in our lab right now, just acting like a relay to turn gimlets on and off. Yes. Because we have because, like, we they exist, and we have a full software stack and debugger all, like, you know, targeted to this. So it was easy. Right?
• 36:12
  Speaker 1: It was easy. And the the photo of the the gimbal 2 years later on is the one that I was using today to do some some work, the LTC 42. I'm gonna literally, like, had to, like, get everything unplugged so I could take a photo of it. These things are used so frequently. And the other thing the other thing I love about this is the LEDs on this thing.
• 36:28
  Because there's still the one thing you kept from Gemini is there are 4 LEDs on this, which are actually I find very useful for communicating purpose. So the the the gimlet lets that we use to only as power supply controllers have a different flashing sequence than just our stock demo.
• 36:46
  Speaker 4: I mean, I feel like you can tell people that what a done internal level bring up on a system by how many LEDs they design in that don't have very many drivers between them and the CPU that have to work before you can turn the LED on. Right. Because, like, this is invariably the first thing that happens when we're bringing up a board. And, the other thing I wanted to point out on this, just in terms of, like, you know, if if any of y'all in the audience wind up doing something similar, If you've done better electronics before, you might recognize the connectors. Those are p mod pinout connectors.
• 37:18
  That's a standard ish that the gentleman developed a while ago for FPGA boards, but, after Gemini, which had a bunch of custom connectors and pinouts and fancy stuff, that never had anything plugged into them because we don't have time to make cables for every new connector under the sun, we used standard pin outs here, which means that the sensors at the bottom of the gimlet that or gimlet that Brian's plugged in there, those are just off the shelf eval boards that ship with the pin out. You just plug them in. So, that, I think, was a big win. Huge win. Not the best.
• 37:53
  Huge win. There's and the the pen outs are you know, it's a it's a standard. So somebody designed it that it wasn't you, and it probably doesn't exactly meet your needs. But it's been probably good.
• 38:03
  Speaker 1: The only and I just view this as a lovable tick of this board that if you're holding the board right side up, such as the handwriting is right side up, the pinout for the I squared c is actually upside down, which I think is so in other words, that pinout that you're seeing there is actually invert if you look at I squared c 4, I squared c 3 on the bottom, that is the that's the inverse of the actual pinout.
• 38:26
  Speaker 4: Yep. When's at the legend, one of the pins in the legend is square, and that's intended to do the square thing on the connector, but that was way too subtle, and I should have done something
• 38:38
  Speaker 1: different. Well, I mean, I get I think that you did this you were saying 24 hours. I think you did this in, like, 10 and a half. It was I just jaw dropping how fast this was and to get it back. And then I also love, Cliff, the fact that you not only did all the IO, but then you've got these GPIOs in the middle there that are super is it that's what makes this I mean, Cliff, you described this when you did it, like, this is gonna be a Swiss army knife.
• 39:01
  And it really is a Swiss army knife. We use it all the time everywhere.
• 39:06
  Speaker 3: I mean, we've had this thing connected to about everything. I mean, you've had this connected to, like, renaissance power eval boards, and, I mean, just all kinds of stuff. Right?
• 39:16
  Speaker 1: For sure. Right now, at home, it's connected to chunky boy, which is the that a 100 amp, 100 amp demo board from LTC. So yeah. No. These things are odd.
• 39:26
  Yeah. We use these for for everything. I see it again. Well, let and in fact, actually, go to the next image. This is another Yeah.
• 39:33
  Sorry.
• 39:33
  Speaker 5: We'll leave that. Like, on the p mod thing, like, it is a standard. It's not great as any sort of standardized interface to ends up being. It's not exactly what you want, like Cliff said. But one of the really versatile things here is it brings out a lot of very primitive buses, and we brought out a lot of them.
• 39:52
  And so it's given us not only the ability to do a whole bunch of experimenting because, hey, it's easy. You know, we we don't, like, these are all point 1 inch pin headers. It's, you know, you can grab jumper cable, like, little jumper wires off the shelf and just hook up pretty much anything. But there's also quite the ecosystem of little P Mod boards here and there, to to try things out. And it's just turned out to be extraordinarily versatile, which turned out to be the thing that we actually needed.
• 40:20
  Like, that was the lesson from the original Gemini was, Okay, well, not only is it about breaking it into smaller chunks that we can evaluate individually, but making the interfaces between those chunks be very, very, very versatile and very simple, so that we can adapt to just whatever.
• 40:39
  Speaker 1: And you can have I mean, as a result, like, if if for those folks that are doing work on the service processor on on Hubert's, the operating system there, if you've got a GIM wetlet, you can do, there's a lot of work you on the operating system with just a GIM wetlet without even any necessarily any peripherals. So it's, it is a very, very useful building block. So you may actually, maybe you wanna go I don't know. I can go back the other way to talk about the the gimlet with NIC, plugs into that that NIC carrier there. And this is actually a board, I confess, I'm not actually used.
• 41:10
  This is, I think, the only board on here that I have not actually used, because these we just got, we did a Revlon recently. But this is this is an Arian board, isn't it?
• 41:19
  Speaker 3: I think this was Matt.
• 41:21
  Speaker 1: Oh, it's Matt. Oh, nice.
• 41:23
  Speaker 8: This is a Matt board. You can see my initials on that.
• 41:25
  Speaker 3: Matt board.
• 41:26
  Speaker 1: It's a great segue to mat.
• 41:28
  Speaker 8: Yeah. So this was, so it was funny when you were talking about the gimlet. The very first thing I did at Oxide, was not any engineering work. It was to order, like, 50 gimletlets from a company that would assemble them for us. And that was my project to buy goodwill at the company by making sure that everyone could get a gimlet on their desk since they didn't need to be lovingly hand assembled.
• 41:47
  Speaker 3: Also, you wanted 1?
• 41:49
  Speaker 8: Yes. I specifically wanted 1 on my desk.
• 41:51
  Speaker 3: Like, I would like to do work.
• 41:52
  Speaker 1: Was also good. And and let's just be clear, you bought yourself an I mean, I I what how I mean, you could you could've done horrific things after that, Matt, and you would've always been the one who brought us gimlet. So I it it definitely worked.
• 42:05
  Speaker 3: And it it was fun because, like, they landed before our meetup. Right? So then people could come by and take take a Gimlet at home, which was fun.
• 42:13
  Speaker 8: Yeah. And that was with a a company called CircuitHub, which is actually one of the few companies that will take CAD files and bring them directly to assembly. So you don't have to do, like, export the BOM and then send it to Excel and then go back and forth, which is very So
• 42:25
  Speaker 1: we have the makers of Pick Place Podcast. So another Yes. One of our favorite podcasts. So the always wanna give them a little bit of business when we we possibly can.
• 42:34
  Speaker 8: Yes. So the NIC is, what it looks like. It basically breaks out 2 Ethernet ports. So back when Cliff was designing the Gimlet LED, he very wisely broke out the RMII, which is the Ethernet MAC interface onto the kind of beigish gig square header that you can see on the left of those pictures. And so the NIC plugs into that and then has a 3 port switch, which you can see in the very center of the NIC.
• 42:59
  And this was actually based on an older design by Ariane, I believe, brought up to run on the gimletlet hardware. And this lets you connect it to the network, which is great because turns out that connecting stuff to the network is important for a lot of things that we're doing.
• 43:16
  Speaker 1: Yeah. And this has been and actually, I, you know, I said this is the only board I should not use, but I in part because I didn't realize that we had a bunch of them out here. We've got a ton of them. So I need to start using this one at home because this is a very it is very, very handy to be able to go from to Ethernet from a from a GIM well lit.
• 43:32
  Speaker 8: Yeah. The other fun thing about this board is that it's it mimics what we have on the actual servers. So it's not quite the same, setup, but this final switch, the ksc 8463 is the same one that we use in the servers. And so being able to configure that on the bench top and make sure that it works and make sure that all the VLANs are set up appropriately, and all that kind of good stuff is very useful.
• 43:54
  Speaker 4: Yeah. This is worth noting in terms of where the server bodies are buried. So it's important to us to have redundant, management cabling through the backplane and redundant management servers at the top of the rack and so forth. But finding a microcontroller with 2 Ethernet interfaces is really, really hard. So every end of service processor has an Ethernet switch next to it, which is the thing you see there on that red board.
• 44:17
  And, that that lets us pretend we have 2 ports. It's not the best thing, but, boy, it sure did get the job done.
• 44:25
  Speaker 1: It did get the job done. And, yeah, it it was, it tough to I mean, and this is I think, Cliff, I remember early on asking you, how are you actually doing this parameterized search parametric search of microcontrollers looking for all the functionality that we need? And I I think as I recall, your answer was just like, yeah. You just kinda I just a lot of, having done this before and kind of a lot of data sheet reading. I mean, there's there's no, odd it it sounds like a pretty painful process.
• 44:55
  It's what we could
• 44:56
  Speaker 4: not find. Painful is in the eye of the beholder. I go through the Digi Key microcontroller section every so often just for fun on a weekend morning. Yeah. Just because, you know, this is what I do.
• 45:06
  So, but, yeah, there's one of the most important things I learned, I think, the first time I worked with actual electrical engineers, is that there isn't a trick. It it really is just a giant market. You can get on vendors' mailing lists so that you find out about their new parts, which can sometimes help. And, I I worked with a guy, Ed Keyes, who for a while had a interesting new microcontrollers mailing list where he would digest this and send it out, and that was amazing.
• 45:32
  Speaker 1: Yeah. What happened to that?
• 45:34
  Speaker 4: I I think he got bored and moved to a different job. But
• 45:37
  Speaker 1: Oh, it's rad. Yeah. That would be a that'd be a great deal.
• 45:40
  Speaker 4: Oh, by the way. Yeah.
• 45:42
  Speaker 1: So, sorry. Right. Go ahead.
• 45:45
  Speaker 7: No. Part part searching in general is always a pain, and you just you start learning how to navigate the websites very quickly and trying to home in and figure out how they classify things. So there's there's no trick other than spending way too many hours on Google and TI and analog devices and all these other websites learning how to navigate their stuff.
• 46:07
  Speaker 1: Well, it is, it's deeply appreciated, and I think it's also just important that, like, if there is no trick other than doing that kind of that that hard work. So then moving forward, to the so we got the Gimlet let this is the 1st gen Gimlet let, and then we get to the the the dimlet, which I Nathaniel, again, this is another board that I've got great emotional attachment to. I am so appreciative for this board. I, so do you wanna describe kind of the genesis of of the Dimlets?
• 46:36
  Speaker 3: Yeah. Well, so Dimlet was one of these, you know, like, in chat, people had been talking about, you know, problem like, what we're what we are going to, you know, struggle with to bring things up. And, like, there are certain things about our server sled that are, like, difficult to prototype. Right? And we're certainly doing something kinda strange with, with the the SPD EPROMs on the DIMMs, such that our our service processor is proxying those so that we we basically can see on the control plane side how what what we have in there and, and then fake that back out to, the AMD.
• 47:11
  But the side effect of that whole circuit means that, like, the AMD processor isn't actually connected to the SPD EPROMs on all the DIMMs, and it it definitely needs that information to boot. And so, like, one of the, like, critical path tasks for our team was we need to have that proxy working so that we can actually boot a SLED. And, at least at that point in time, we thought booting would come a little faster than booting maybe ended up coming, for unrelated reasons. But, in chat one day, it was just like, you know, shouldn't shouldn't we make a board that connects to these PMOD things and has a couple of DIMM slots that we can go talk to the SPD EPROMs?
• 47:56
  Speaker 1: And it's So in particular, I had been looking I had and I had taken a couple swings at this because I'm like, this board has to exist. I'm just not finding it. Like, what I want is a board that that that holds a dim, and I wanna not speak DDR. I do not need to speak. I don't need to treat this thing as DRAM.
• 48:15
  I only wanna talk to the little bug on there, which is this little SPD chip, tiny, tiny chip on there, which you speak to, which is the serial process detect chip, you speak to it via I squared c. I'm like, I just want something that knows how to speak I squared c to a DIMM. And I I mean, Nathaniel, doesn't it feel like that should exist? I mean, I just spent a and I and I It's surprising. Yes.
• 48:37
  Speaker 3: I mean, I think, you know, DIMM slots, I think, end up in you know, I'm sure that people have made something like this before, but maybe not out in the public domain. And I feel like your, you know, your your hobbyist kind of engineer is not often looking at putting DDR 4 dims on a on a circuit. And and, like, I mean, I wouldn't really want to do a DDR 4 layout for this. But, like, this board is goofy because there are in that whole DIM, there are only, like, 10 pins soldered at all. And and and about half of them are for mechanical stability.
• 49:11
  So we basically just have the I squared c pins and power to the the SPD EPROMs, and that that's it. And then and then, like, we have a level translator and some other stuff in there, a little I squared c buffer, because the the you know, we have to level translate from the 3v3io on the on the, gimletlet.
• 49:30
  Speaker 1: But this was incredibly clutch because it allowed us to write all of that software, write the SPD proxy without actually how before we went out to do bring up. So we had that whole proxy could be tested completely independently with 2 gimletlets. I think it's doing a actually, a gimletlet to a Gemini with a the the the DIMMs hanging off of the gimletlet and then using the Gemini as an I squared c initiator to and so we were able to to actually simulate all of this. And so I I mean and and this board was incredibly clutch for it. But, hey.
• 50:04
  David, that is a good point about, like, this is not exactly a hobbyist kind of a it like, beta fruit is not gonna carry this one.
• 50:10
  Speaker 3: Right. Yeah. Like, oh, would you like to add d d 4 Deidra 40 or just like, no. Please don't. Like, none of us really do.
• 50:15
  So, the the other thing, like, these are also really portable. Like, I I think I remember you tweeting from the hotel room there in Minnesota that it was like, you know, here I sit at my desk working on, you know, this, like, proxy thing, and it's like, you know, you you had all the things that you needed, and they, like, fold up and put in your backpack. And our service led definitely does not do that.
• 50:36
  Speaker 1: Yeah. It was great to be able to do that. I know I felt like I was, I was channeling my inner trammel. Trammel is constantly doing this where he's kind of, like, tweeting out from a train or what have you or he's doing, but it was did did the be able to to have that portable, to have the software because the software and it like, a lot of these things, like, the the the spy mux issue we're talking earlier. This is one of these things.
• 50:57
  It's like, if this software doesn't work, you have nothing. Like, the thing cannot boot without the software working. So all of these little bits need to work, and being able to prototype this ahead of time was just incredibly clutch, and I am very, very grateful for Dimwit. I'm I'm not even sure if anyone's ever anyone else has used them, but but I
• 51:13
  Speaker 3: think so. I think you might you I mean, I think I put them you know, I I did, like, a read or something so I could make sure that it worked, but I think you're the one who's used them the most. But I know you also you spent you spent the time looking for them, and they just like, these things don't exist. The other the other fun thing about that board is that in typical I squared c fashion, the I squared c bus translator that I use there, has rules about which side is higher and which side is lower that I didn't notice. And so that part is actually installed backwards.
• 51:47
  And then a jumper a little tiny jumper wire flowing across it, and there's a trace cut because, the a side and the b side have to you know, they have a specific voltage relationship that I messed up.
• 51:58
  Speaker 4: And and that sounds like the sort of thing that, like, oh, well, Nathaniel made a mistake. But, no, I swear to god, the I squared c data sheets change after I design the board. I go back and I read the data sheet again, and which side is high has changed now.
• 52:14
  Speaker 3: I wish that I could say that this was probably not my fault. At least on the dimlet one, it was my fault. I I missed it.
• 52:22
  Speaker 1: Well, these are super easy mistakes to make, and I think the part of another reason that these boards are so important these product boards are so important is because so the I mean, the the stakes are lower. I mean, if if we if that had been unreworkable, it's not that big of a deal. It's not like Right. Versus And if we were delayed not being having the SPE proxy or these other or, you know, certainly, if we had it it would have it would have cost us a board spin, and we'd not have the SPIMAX board. That stuff is really, really expensive.
• 52:48
  Speaker 3: And one of the benefits too of, like, we hand like, I hand built this. Right? So I I didn't build 10 of them and then go test them. I built 1 and got it working and then did the mods before I actually built everything. And so that's one of the like, one of the balanced things that you get to is when you have someone else building everything, that that sometimes means if depending on your level of confidence, you you have a lot more, potential for rework, especially if the design is fairly nascent.
• 53:17
  Speaker 1: Yeah. And you've got it's very nice to be able to realize, like, oh, I've only I what this one is wrong. Now I know what I need to go do on these others, and I haven't lost all of them. Yeah.
• 53:25
  Speaker 3: Right. And, yeah, cut cutting the trace is a lot easier when you don't install the chip first, you know, like those kinds of things.
• 53:31
  Speaker 2: Yeah. I don't wanna point here that, you know, that you're the only consumer of this, Brian. That, like, this has become so easy for us or easy for the world at large that we can have these one offs that are just for one really tactical, but important piece of work. Whereas I'm sure, 5 or 10 years ago, that would just be completely untenable.
• 53:53
  Speaker 4: Oh, absolutely.
• 53:54
  Speaker 6: Yeah. I wonder if this might be an interesting time to talk about tooling. Because one of the things that strikes me about this is that this is possible because we have access to very high quality tools that can be used by anybody in the company. And I'm referring specifically to KiCad here, which has gotten better and better and better over the last 5 or however many years. I'm wondering if you guys can talk about that a little bit.
• 54:17
  Speaker 1: Yeah. I mean,
• 54:17
  Speaker 3: I think my general preference is, where So, like, all of these little designs have been done in KiCad, with maybe a couple of exceptions. I think, one of the BGA parts that we're gonna show a little bit was done in OrCAD for practical reasons because, we are we already had the footprint there. And redrawing a 4,000 ball footprint, you know, is is not something that we're, you know, looking to do right now when we're trying to get these things out quickly. You know, KiCad is is awesome. I hope that, you know, it continues to grow.
• 54:59
  We continue to plan to use it. And, I I would I would love to say, you know, in in some number of years, it has enough features so that we can, like, nail our, you know, DDR 10 design or whatever we're on, there in KiCad. So but it's I mean, KiCad is, like, above and beyond, you know, like, if you've used Eagle, I think KiCad is, like, totally a cool one there.
• 55:22
  Speaker 4: Yeah. I think that's I think that's probably the best comparison. Like, there's people who have been using Altium for years get mad at KiCad for justifiable reasons. But, like, KiCad feels a lot like where Eagle was for me a few years ago. And I don't know.
• 55:35
  Like, it's it's nice that these tools are universally available now.
• 55:39
  Speaker 1: Absolutely. And I think it's also nice that they know that it to give folks who may be coming from the Altium side an opportunity to do a small board to get to know the tool a bit too. That we Oh, sure. Feel like you're having to learn the tool under and adding KeyCAD to everyone's arsenal such that we can, accept you. I mean, I think in the limit, certainly, we hope in the limit, we are using KeyCAD for all boards.
• 56:01
  But as you say, maybe DDR 10.
• 56:03
  Speaker 3: It's I mean, yeah. Board. You know, there there are things we need. Right? And and there are, there are, like, workflow and layout efficiency things that we just, like, need to get supported there.
• 56:14
  But, you know, like, the Spymux was my 1st KeyCAD design. So, you know, I've done board designs in a couple of different packages before, and, you know, KeyCAD was pretty easy to, like, hop right in, and, you know, there's decent, documentation online. So when you can't figure out what you're trying to do, you can get out there and do it.
• 56:32
  Speaker 1: Yeah. That's right. So then then the the next image, this is the the the modern gimballet. So this is the I I think that the silk is the same, but we use the the the black solder mask. And, this is the one that that, Matt, you did this when you first came to the company, and I just in terms of of getting all the details of getting this thing assembled, and and having SurCap do that.
• 56:56
  And this is this is what we just have in place everywhere. And, again, this is an action shot. You can see that I squared c 4 is actually plugged in in this in this one because this one's actually doing work, and, like like, all of these.
• 57:09
  Speaker 8: Yeah. And I'm pretty sure this picture was taken during the great, semiconductor shortage where we couldn't find a power supply, which is why it's wired in, with the red and black wires you can see at the top of the picture.
• 57:20
  Speaker 1: Yeah. So this is like I think that this is my gimlet led or foam. I took this photo today, but, yeah, this is an early, early gimlet led, where we didn't have the regulators. Right? Because there's no regulator on this.
• 57:30
  I think I think still, I've I've got a
• 57:32
  Speaker 3: I went and installed them on all the boards I could find, but I didn't probably find the one at your house.
• 57:36
  Speaker 1: You do not come into my house, which is for the best, really. I just I know that I, it it there are otherwise, there'd be commentary on. Rick has insisted that I not leave the room with certain of my components plugged in. And I Rick, I'd like you to know that I have adhered to that. I've been I was using the the DC electronic load a lot this weekend, and every time I left the room, I unplugged it.
• 57:57
  Rick is convinced I'm gonna burn my house out of this thing, and I'm not arguing with him. So the and I get these so these are a little bit out of order. So the the the BGA here is actually gonna go with a later board, so we're gonna come back to this. Sorry, Eric. I got that one a bit out of order.
• 58:15
  The, okay. The next one, this is a another action shot. That that that again, with that that does have the regulator. And then do you wanna talk about that breakout board, Nathaniel?
• 58:24
  Speaker 3: Oh, sure. Yeah. So we use we use these, Samtech connectors, because they're they're not 10th inch headers. They're a little more they're a little smaller, and they're keyed nicely, which is has been really nice for, you know, use. But, like, when you need to break those things out, then you need special connectors and special pins.
• 58:42
  And so, at us, I we had these in place for sidecar bring up. Sidecar uses these. That's the picture that they're attached to the board that they're attached to there. But we had various reasons to need to get in there, and then all of our I squared C buses and stuff, come to these headers. And so it's, like, useful to be able to put a or something on these buses.
• 59:06
  And, so, you know, I was digging around trying to figure out what what kind of thing we could make as an adapter that would get this out to, like, things that mere mortals can use, like 10 inch headers. And, I think, Robert Keith got, recognized. I I did not find the right angle Samtech connector, and I was like, boy, it sure would be nice if I had a right angle Samtech connector for this. And of course, Robert, you know, church of Samtech, was like, hey, it's this other it's this other part number right here. And so, you know, I went to Samtech and, you know, you can sample them and you can buy them and they were in stock.
• 59:41
  And so, you know, we we bought a a bunch of those, and I think I built about a 100 of those at this point.
• 59:47
  Speaker 1: And these are so the and then the next image is actually zooming in on that. So these are actually 2 of these things together. The great thing is you can actually put these little dongles. You can kinda connect them. And, boy, the ability to break out immediately I squared c, or to break out some of these signals and get them on a scope, or get them on a logic analyzer, or take them somewhere else, or I think in the case of Laplace image, I think we're actually using the gimballet to drive I squared c, on the board.
• 1:00:14
  Oh, you
• 1:00:14
  Speaker 3: it looks like a Saleae A too.
• 1:00:16
  Speaker 1: Yeah. And then a Saleae A too. Yeah. So we actually were using that's why we were using 2 of them there. Incredibly useful.
• 1:00:23
  And it it it then it's also been a nice little standard to have that, you know, that any one of our buses is gonna have this header on it, so you can immediately get a an analyzer on there. Really Right.
• 1:00:34
  Speaker 3: And and the pinout's written on the back, which is handy, for that stuff. So and and, you know, it's like kind of a happy accident that you can kind of, like, do dongle inception with these. Like, I had put the the the, Samtech connector on to allow them to be a pass through, thinking that, like, you might want a programmer and a, and a connected. But, like, then because you can, you know, because we have the other genders, Samtech connector, you can just start stacking them like we did there in that picture. And it's like, you can, you can kind of build the ladder as as many as you can mechanically support at that point.
• 1:01:11
  And I don't know what you'd do with that, but it that was not like, a design criteria going in, but it's kind of a happy accident.
• 1:01:17
  Speaker 1: Well and this is, I mean, a good example of what we do with it, where you you're using one of these headers to to drive it and another to This next board, we're gonna come back to. Another zone we're gonna come back to. This is the the the VPD board. I normally, it'll be attached to my dongle, but my dongle was my car was broken into. In addition to my laptops, whoever took my laptops also got a dongleit.
• 1:01:44
  So we'll talk about dongleit in a second. The and then this is the the this is the ignition. The next one is the, the ignition lid.
• 1:01:53
  Speaker 3: Yeah. So this one yeah. This one's RENs. And so we have, we have an ICE 40, FPGA kinda out at the very front end of our system that that is essentially a power switch with a, like, low speed network connect to it. And, yeah.
• 1:02:09
  The, Matt's saying this one is a rare I I think there are possibly 3 of these in existence. It might just be the it might be just the 2, one that Ariane has and one that I have. But this was this was super useful. This has got the FPGA, and you can connect it up to a GIML, of course. And, this has kind of the whole ignition circuit, which Arion has used to prototype all of the communications protocols.
• 1:02:33
  But I also use this to write the FPGA spy interface and get that all running before we ever had, Gimlet hardware. And we use this Little more context. So,
• 1:02:44
  Speaker 2: you know, the we've got 3 different physical networks, probably more, but at least 3. We've got the very high speed network. We've got the management network that that maps, you know, Ethernet ports we're plugging into, and then the ignition network is as Nathaniel says for very low level power control. So, if we wanna turn the whole thing off, including, you know, the thing running Hubris, we use ignition.
• 1:03:12
  Speaker 3: Right.
• 1:03:12
  Speaker 1: Right. And very important to actually get the be able to have a development vehicle for this. And then this thing plugs into the PMODs. Right?
• 1:03:19
  Speaker 3: This thing does plug into the PMODs, so you can get a spy interface out there. And then, on this one, you can see all of the, we have all the SMA connectors, those big gold connectors down at the bottom connected and installed. And so you can that's where the, you know, the the low speed, 8 b 10 b stuff, can go. And so the way Arian prototype this is he connected that to, an ICE or a ECP 5 Lattice dev board, and then he can kind of prototype the whole sidecar subsystem there. Where I use this, I just put the SPY core that I wrote into that FPGA and then got it talking across that, pmod interface to the SP, and so we were able the the Hubris already had good spy support, so it was pretty easy to, like, prototype all of that and get that running.
• 1:04:08
  Yeah. So we had that we had that ready to go back in back for bring up, which we needed because that was the sequencer is an ICE 40 on the gimlet, and it does all the power control the local power control. And you need to be able to have software tell it to turn the power on.
• 1:04:22
  Speaker 1: Yeah. Again, another one of these details that if you don't have this, you don't have you don't have power on. So it's very, very important very important detail. Matt, I think these are the actual photos of yours. This is this is your cursed gimletlet config.
• 1:04:35
  It looks like was this a this a Spymux? It looks like you've got an ROT carrier carrier and and a gimletletletnet on there?
• 1:04:42
  Speaker 8: Oh, yes. This was me just, digging through my drawers and finding every board that I could plug into a gimballet, lit for a staged photo. But I think that the the interesting board here is the red one on the upper right, which has a green board plugged into it. So if you remember all the way at the beginning of the podcast, we talked about how the root of trust was built on its own standalone board, and then that would plug into another bit of hardware, so that we could test some of the security features and risk lock pricking it and locking it up. And so that's the green board.
• 1:05:11
  And we had a bunch of those, but with the dawn of the gimletlet rev 2, which you see in this picture, those no longer plugged into it. So I spent, you know, a day or so and built the red board on the top right, which is the gimletlet carrier carrier, which takes the gimletlet carrier, plugs that in, and then adapts it to the p mod connector so that it can plug into the gimletlet. And so that is, you know, 2 layers of of daughter boards to get back to the root of trust connecting to the gimletlet.
• 1:05:41
  Speaker 1: Yeah. So, Adam, if you're running with the gimlet, let carrier carrier was. There you go. That's the, it and this is but another been another very clutch board that, has enabled us to to use because we have got a ton of these. As you said, we got a a bunch of these these little LPC 50 5 boards.
• 1:05:57
  So very nice to be able to extend to be able to continue to use those with the GIM wetlet. And then, Matt, this these next two photos are, I I'd seen these so this is where you've taken, looks like a, a a Gimlet lit clearly. And then what do you is that speaking I swear to see what's that or is that spy?
• 1:06:16
  Speaker 8: That's what we're talking about. Action shots of using begin lit lit in its role as kind of the universal Swiss army knife for plugging into things. So the thing that it's connected to is the VSC 7448, network switch dev kit, which is obviously not something we made. You buy it from TI. It's sandwiched in layers of, like, half inch thick acrylic.
• 1:06:37
  It's very impressive. It weighs, like, £5. But one of the things you can do with the chip is flip a couple of DIP switches and switch it from booting from its internal Linux score to just coming up in a quiescent state where you can then configure it over spy. So we have, I made this terrible terrible breakout cable which goes from the ribbon cable into one of the t mod connectors. And so with the DIP switches in the right configuration, this, the management switch comes up, and then we configure it entirely from the gimballet.
• 1:07:08
  And then it brings up the 54 some Ethernet ports that are on the front of the switch, which you can kind of see in the other picture.
• 1:07:15
  Speaker 1: Okay. Matt, this is another thing, by the way, that I have searched the universe for that I have not found is ribbon cable breakouts. Why? Because it I mean, I'm gonna guess that you made that yourself.
• 1:07:27
  Speaker 8: I don't know why you would say that, Brian.
• 1:07:30
  Speaker 2: Child made it.
• 1:07:32
  Speaker 1: That's right.
• 1:07:33
  Speaker 3: You so, normally, to break out a ribbon cable, you'd get, you get, like, socket connectors, and you could crimp those mids mid, like, mid connector and then use, cables coming out of there. But, like, why
• 1:07:47
  Speaker 1: is it not a product that takes a ribbon cable and breaks it out? I mean, it just feels like I needed this. I this much was sued, and we got we've got one, Adam, you and I got one that we we we took this hardware hacking course ages ago that had this that was given out as basically a a kind of part of the course that and I think the thing is so useful. I anyway, I've not been able to buy them and clearly nor has Matt.
• 1:08:13
  Speaker 8: Yeah. I mean, if you wanna do this cleanly, then we'd probably make, like, yet another gimlet lit daughter board, which just connected to the PMOD header and broke out a ribbon connector compatible plug. But this was a quick and dirty hack job that worked for probably it's been up for a year now, so it works great. Right. This is,
• 1:08:31
  Speaker 6: Those things probably aren't commercially available because there's somewhere around what is there? Like, 40 conductors in that ribbon cable. I mean, that'd be a pretty impressive breakout board.
• 1:08:40
  Speaker 1: We I just wanted to bring out cable. I I want what Matt has made here. Actually and I I would even I would even buy mats.
• 1:08:47
  Speaker 3: Adafruit does make some for smaller pin count at least, IDC cables.
• 1:08:52
  Speaker 1: Well, I those are
• 1:08:53
  Speaker 3: those are installation displacement, IDC.
• 1:08:56
  Speaker 1: The I use it for squid.
• 1:09:01
  Speaker 4: I also feel like you this by just taking patient using cable and chewing on it.
• 1:09:07
  Speaker 1: Yeah. Well, that which clearly I am not sure how to superglue. Yeah. Exactly how our superglue was, was sacrificed here, but, then we get to the the the next one is definitely, a crowd favorite, kludge.2. Nathaniel, do you wanna talk about the origin of Kludge.2?
• 1:09:24
  Speaker 3: Yeah. So this was this was, you know, we're we're talking about how to get our SLED network booted and programmed. And, like, it turns out, like, the only kinds of PC interfaces we actually have on the SLED, up until this point were that are, like, useful PC interfaces are, all of our, like, ST Link and MCULINK dongles and some UARTs. And so for a large portion of time, many of us spent a lot of time putting RAM disks over a UART with Xmodem. But, like, as we started thinking about what was gonna be required on the manufacturing line where we're gonna try to make a bunch of these, That like, doing an entire I mean, we we had a few different things we looked at.
• 1:10:10
  We were trying to figure out how to get our SSDs programmed. And so, like, you can you can buy some adapters that you can plug m. Twos into, and we tried those first. That was kind of our primary path. But our m dot twos are, you know, industrial grade m dot twos, and, it's hard to find inexpensive dongles that will do programming.
• 1:10:31
  You can get big, like, fleet programming stations that will do, you know, many of them at a time, but we were
• 1:10:37
  Speaker 1: because so this is the idea here is that we would program the m dot twos outside of the server, and then we would load the now programmed m dot twos into the server, which felt like in in this you know, this is another one of these things where it's like, this must be like, how hard it can be to find an m.2 programmer that works. It turns out
• 1:10:54
  Speaker 3: It also you can.
• 1:10:55
  Speaker 8: They just
• 1:10:56
  Speaker 7: that our m.2's take a lot of power.
• 1:10:59
  Speaker 3: Yeah. I mean, that was so so Joshua tried that actually. Like, our primary path was gonna be, let's just set a computer up and put m run m.twos through there. And, like, you know, there were some things we had to go figure out, like, you know, is it safe is it safe to hot plug those things? How are we gonna deal with that?
• 1:11:15
  But at at the first point, it was like, let's just go get, an M. 2 to USB dongle or something and and try to program these. And for the ones that we could, locate that were reasonably priced and, sourceable, our m.2s, tipped them over, and and the m dot twos wouldn't run-in there. So
• 1:11:35
  Speaker 1: Yeah. It really did not go well. And, Adam, I don't know, were you involved at at at I mean, this is a very frustrating problem because this feels like it should be a non problem. It's like, I just, I've got my bits over here. I've got this m.2 over here and I need to get the bits onto the m.2.
• 1:11:50
  Surely there's been a product that does this. And
• 1:11:54
  Speaker 2: Yeah. No. I I I went down, some weird rabbit holes with Josh just looking for random solutions, and and this it should be noted, wasn't you know, he did cobble together a bunch of more or less off the shelf parts to kinda get to where the klugetot 2 was, but man, was it horrible.
• 1:12:15
  Speaker 3: Yeah. Well, and our our drive bays are deep. So, like, that cause because we kinda we're expecting to be inside a a drive carrier. And so anything you find off the shelf is really painful to get aligned and and pull out. And so, like, you can get it in there carefully, but, like, then trying to get the thing out.
• 1:12:35
  And so then thinking about our manufacturing line again and, like, you know, having to have operators sit there, you know, the poor operator having to, like, fish the stupid thing out every time after it's like, that's that's not a great solution. And and, you know, as you start thinking about this, you're like, well, you know, on a lot of our bench setups, we've basically replaced a u.2 since they're all PCIe. We we pulled our little shark fin card out, and we've been dropping in, you know, an Intel NIC and using that as a, like, useful computer interface since none of us have, you know, 400 gigabit Ethernet at our house or 200 gigabit Ethernet at our house. And so, you know, it was like, well, what if we just made an extender that just extended PCIe out the front, and we can stick a NIC in? And so that's that was kind of the genesis for this.
• 1:13:24
  And so that's
• 1:13:24
  Speaker 1: why Unfortunate but reasonable.
• 1:13:27
  Speaker 3: I I I Just
• 1:13:27
  Speaker 2: to emphasize that, Nathaniel, it's like we, we scrapped so much of the stuff that a standard server would have. Like people will say, why not just do this over USB? We don't have USB.
• 1:13:37
  Speaker 1: Right. Like, deliberately. Like, the Eversmart is not a leaving port.
• 1:13:41
  Speaker 3: Right. And we don't have an RJ 45 jack. Right? We have an exomax connector.
• 1:13:46
  Speaker 2: We don't have any of these things. We have exactly what we need and nothing more. And
• 1:13:51
  Speaker 3: And maybe not quite. That's right. Maybe not quite what we needed. And so in in this case so this was kind of the first version of this. And, like, I you know, you can tell I'm not a mechanical engineer by the design of this thing.
• 1:14:05
  I kinda, like, stuck it in my sled, but, like, I had an earlier sled. And, you know, the the later sleds had a slight mechanical change that wasn't accounted for in my in my sled, and it it may have been a little cozy in my sled anyway. Part of the problem was when I did this, I didn't actually have the whole sled put together and, neglected a top piece that actually impinges on the slot a little bit. And so, we got these in, and they didn't actually fit. And so, I took a nibbler and basically nibbled the, top edge of this and then Kapton taped it, and it's just a ground plane there.
• 1:14:39
  Speaker 1: That is a nibbler. Is it nibbler? Is that a technical term, or is that It is. It's it's actual tool to view.
• 1:14:44
  Speaker 3: No. It's a tool. You can look it up, but it it basically takes a little bite out of metal. And so, my my nibbler is no longer sharp because it it ate a lot of fiberglass, but it it takes a
• 1:14:56
  Speaker 1: little bit. Yeah.
• 1:14:59
  Speaker 3: So, anyway, we did this and, like and got it working. Like, this this was provable. It it like, it worked. And so that, you know, we'll get to the you know, our refinement on this because now, you know, this has turned out to be useful for, maybe a lot of other places that weren't just manufacturing, as it turns out. Like, in Iraq, the front is kinda the only thing.
• 1:15:19
  If you ever need to get something into a sled, the front is really the only way to do it in a rack.
• 1:15:24
  Speaker 1: Well, it yeah. In in a rack without actually g racking it. This is a hot pluggable interface, so we can we can safely hot plug things in, which is great. And then I also love the red color of the solder mask, just making it clear, like, Kloochot 2. This is, like, removed before shipping.
• 1:15:38
  Speaker 3: Like, yeah, don't ship.
• 1:15:39
  Speaker 1: Yes. You're not shippy. I am not done. This should not be, but the it's in this photo, we've got we've got Klugeot 2 a k a k dot 2, and then we've got a little Intel NIC plugged into the PCI slot. So this is just taking you from u dot 2 to, back to PCI, which is, ridiculous.
• 1:15:56
  Well, it is I mean, it really it's it's just what it says. It's unfortunate but reasonable, which I think is. And I just I also love, obviously, computer goes this way, genk goes that way. It's just terrific. Alright.
• 1:16:08
  So we'll get and we'll get to the, k.2 reimagined or or, refined a little bit in a second, but this is super important. Another one of these, like, completely load bearing boards. So, on the right, you see the VPD board, which the board we saw earlier, that's plugging into via pmod, the like we saw with Gimletlet, but it's plugging into this new board, Dongwet. And so Dongwet has got a a g zero on there, g 0 31. And, Cliff, do you wanna talk a little bit about our work on the the g 0?
• 1:16:41
  Because this is one this is another one of those. I feel you've done this a bunch where you have this idea, like, I this this could be useful in the future. And then not too far in the future, it becomes very, very useful, and this was definitely the case with the the hubris work that we did for the g zero.
• 1:17:00
  Speaker 4: You know, that makes it sound a lot more low key than it was in practice. I think I I would describe it as, I was sitting there doing scrolling Digi Key as I want to do. And, in between my interference screaming at all of the products going out stock, I realized that there was a real possibility that we weren't gonna have any microcontrollers available that we could actually run our software on by the time we needed to make hardware. So, in the background for a couple of months, I just kind of randomly ported to things. And the Cortex M0 is the tiny arm controller in the STM32 G0 series and the G0 series is interesting for a couple of reasons.
• 1:17:43
  One, it's real cheap. 2, it was in stock. So, that was the case for me. And that wound up we had to set the whole Hugo system into something like What do we have? 8 k of RAM.
• 1:17:55
  Speaker 1: 8 k of RAM and importantly 64 k of RAM, not 32 k. That's second 32 k is important.
• 1:18:02
  Speaker 4: Yeah. 64 k is important. Yeah. I mean, you can buy 32 k parts. They're a little bit cheaper, and we can fit Hubris.
• 1:18:09
  We just can't fit Hubris in anything useful. So, we we buy the 64 k instead. But that's, what? One to 2 orders of magnitude smaller than our original target processors for Hubris. So that that was fun.
• 1:18:25
  I keep getting to write a blog post about
• 1:18:26
  Speaker 1: the process I used the book for reading. And so it is very small. In particular, this is smaller than any computer that I have ever had. So this is my my my first computer in the in the eighties had and even the Commodore Pets that I was using in school still had, you know, 32 k of RAM or what have you. So and, I mean, this is a this is really tight, but we can actually get a useful hubris in there, and we can get in particular, we can actually speak some I squared c, and we can actually do some things.
• 1:19:01
  And what and and do you wanna Nathaniel, you wanna talk about so we we kinda done that work. We had that in the back pocket. And then what problem were we actually trying to solve, with Donglet here?
• 1:19:12
  Speaker 3: Well, so, you know, I think one of the common things, like, at Oxide is, you know, how long ago did CLIF port the the stuff to the little microcontrollers? I mean, it was, you know, a month or 3 months or so. I mean, you know, it was some time. It was kinda like, hey. I did this thing.
• 1:19:25
  He he demoed it at one point in time, and, you know, it was kinda like, I don't know if we're gonna need it, but, like, I did it. And then, you know, fast forward 2 months, and it's like, well, now that work is critical path for something. So, in in this case, so, like, as Matt had said, we built so we could use this is, the the right side board there, the bigger one. The VPD board is or programmer is built so that it will interface with PMOD. And so it was built to potentially connect up to a gimletlet.
• 1:19:55
  But as we realized, we were actually running short on our gimletlet processors because the 50 units that Matt built, have kind of, you know, been distributed out to team members and are in load bearing lab things. And, we we couldn't actually find more of that microcontroller. So just building more, gimletlets wasn't really an option. And so then once you're you know, now we're back into, like, part search time, and it's like, what what do we have that, like, we can use our code with, like, the least amount of modification possible, and still get, you know, some programming fixtures at our, at our vendor. And so one of the options here was, like, these, you know, g zero thirty ones that we could find in stock.
• 1:20:39
  And so it was like, well, rather than try to claw back a bunch of development hardware that people are using in load bearing applications, let's just build more of these things. And they're not gonna be super useful in a lot of cases, but they're going to be able to do the thing that we need, which is, like, do some I squared z. And so so I built this this little board. It's got, you know, a regulator. It's got a USB, 3 connector that's only wired up for power, so that's how you get your 5 volts.
• 1:21:07
  And, you connect up an ST link to it, and it'll talk I squared c. And I did put the SPI connector on there in case we need to use that for something in a different targeted application. We're not currently using that. And there is, I think the other 3 spare IO or whatever are pushed up to that header up on the top. And so we could potentially use this for, like, toggling power supplies or something, should we need to.
• 1:21:31
  But
• 1:21:32
  Speaker 1: Great to pull that out.
• 1:21:34
  Speaker 3: But this was all just to we need we had to program we have to program a bunch of EPROMs in a bunch of different boards. So we have we we call our vital product data, our VPD boards. We have, EPROMs on our all of our shark fins, so we use 10 of those per sled. The sidecar fan assemblies have a VPD on them. The gimlet fan tray has a VPD on it.
• 1:21:55
  And so all of these things are used, you know, 1 or more times in a given sled or a system, and we need to get them programmed. And we have all this cool, you know, hubris and humility software that knows how to do all this already. So we're just we built some targeted hardware to sit on the line and and do that. And so one of the things, you know, we did there is, like, these two things go together. They have a little power switch so you can turn them on and off.
• 1:22:19
  And, it turns out that power switch, required us to make a couple of little changes because I squared c really doesn't like it when you go away. So we ended up having to reset the microcontroller each time, but it was kind of a fun debug session. And then this thing has just a little crises ago and forgotten about it. Right. This thing has just a little I squared c mux on it, and so you can pick of you know, you can pick 1 of the 3 different 4 different devices and do your VPD programming.
• 1:22:44
  And so, you know, we've done, you know, many thousands of boards through some of these now. So
• 1:22:49
  Speaker 1: Yeah. And this is an example where we we did our own manufacturing software as well as our own manufacturing hardware. So we control all the software and hardware that's on the line.
• 1:22:59
  Speaker 3: Well, we'll have to do Oxide and Friends on that with Joshua.
• 1:23:02
  Speaker 1: Yeah. Absolutely. And I I think, you know, it it one of the when we talked to to the our our, podcast for podcast lovers, last year, the end of last year, and I think it was Jeremy who I don't I can't remember who recommended the Playdate, podcast. And and then I understand
• 1:23:20
  Speaker 2: it. Yeah.
• 1:23:21
  Speaker 1: Yeah. And then you followed up with me being like, you've gotta go listen to this. Like, I thought it was really good. I'm really enjoying it, but they also have discovered a lot of the same things we discovered. And in particular, the need to write that there you need to write your own manufacturing software effectively.
• 1:23:34
  So, there's a lot of software and hardware involved in manufacturing, and this is a great example of having, the the I I feel like with with Donglet and the VPD board, we had, like, a bunch of fortuitous things that we had done began to add up, and we did this. I mean, Nathaniel, you did this very quickly. Josh, they did the software very quickly. I was able to do the Cliff and and I were able to do the Hubris side of this very quick this whole thing just came together really, really fast, and we went from, god, how are we gonna solve this problem to we've actually got a pretty viable solution in hand Right. In a matter of weeks.
• 1:24:09
  Speaker 3: Oh, yeah. And we had we did this in stages. So, like, I did the I did the right side board, which we knew would connect to a gimlet lit first, because, you know, like, it's no fun to sign up for more critical path work if you don't have to. And so donglet was gonna be kind of a if we get to it. And then even, like, the 3 d
• 1:24:26
  Speaker 1: printed frame and everything came even after that because it was just a
• 1:24:26
  Speaker 3: matter of, like, we need a came even after that because it was just a matter of, like, we need a thing that works, and then we can, like, incrementally add to it to get it better. Yeah. I
• 1:24:38
  Speaker 1: don't know. Cliff, are you hop trying to hop in there?
• 1:24:40
  Speaker 4: Oh, I was just gonna say, like, one of the things that culturally I really enjoyed about coming to work at Oxide is that we're all pretty aligned around the idea that sometimes you have to go slow to move fast later. And, like, in this case, this is an example of we've been building reusable parts. We've been using building, general purpose firmware, you know, operating systems and reusable board interfaces and things like that so that when a problem comes up now, now that we've got all the stuff on the shelf, we can move really fast. And Yeah. That's been really nice to see that come together.
• 1:25:16
  Speaker 1: It has been really nice. And it's been and I think they're very vindicating again. I mean, the oh, your belief that, like, hey. I think the m zero and the m zero is gonna play an outsized role for us, and it's a great little part. And, the and we're and we love the g zero is a great little instantiation of it.
• 1:25:32
  I love this thing because from a peripheral interface, this looks exactly like the h seven, which is a much bigger microcontroller. So it's great to have the same, what is clearly the same I squared C block anyway, down on this much, much, much smaller micro. It's great. The next slide so the the next image, this is a, a close-up of the the the Gimlet let carrier carrier, it, Matt mentioned, preserving the all IO 3.3 volt from, an earlier instantiations of this, the warning there, and the new to trust.
• 1:26:06
  Speaker 3: And the
• 1:26:06
  Speaker 8: new to trust.
• 1:26:07
  Speaker 1: Yes. Exactly. The new of, which is another another Gemini, throwback. And then, Eric, what's it, with the what's the these next boards, what are these?
• 1:26:19
  Speaker 7: Yeah. So these aren't something that we made, but this is something that we had to customize ourselves because, so we're testing our our sidecar Ethernet switch, to be able to handle 10 watt transceivers, q s, QSFP 10 watt transceivers. Turns out there's no, passive load that'll do 10 watts. So this is a picture of the modifications I did to an off the shelf, QSFP loopback module from, you know, sfpcables.com or whatever. No.
• 1:26:54
  10 g tech. And they had a little board that had SMT pads for the TX and RX, power supplies. So I had to modify those to be 10 watt and then add in this little aluminum spacer you can see on the top so that it would, not have to be a, you know, 6 millimeter thick thermal pad. And that's how all I made all the 10 watt loopback transceivers for regulatory testing.
• 1:27:19
  Speaker 1: I was gonna ask you how you did that because I just actually, just today, I was doing the LTC 4282 support and add your screenshot of the the you had 40 amps going through there. And I'm like, how did he do that actually?
• 1:27:33
  Speaker 7: This is how.
• 1:27:34
  Speaker 1: This is how. That's great. The Well, Eric,
• 1:27:36
  Speaker 8: you you haven't said the funniest part about these loopbacks.
• 1:27:40
  Speaker 7: Oh, yeah. So they, you know, they're loopbacks. They have this little I squared, and there are actually 2 chips on here, like an I squared d problem and maybe some sort of tiny micro or something. And through the I squared interface on these things, you're supposed to be able to read the module temperatures. But these don't actually report temperature because
• 1:27:59
  Speaker 1: why would
• 1:27:59
  Speaker 7: you ever need to on a passive loopback that dumps 3 and a half watts? Well Well, when we dump 10 watts, it turns out that's really important. So I had to put some thermocouples on there and just manually tweak the fans and be like, yeah. That's good enough. When, you know, the real ones have temperature sensors, but, unfortunately, these did not, and I did not find that out until way too late.
• 1:28:20
  Speaker 1: There you go. Do yeah. Do we get god's own temperature sensor? Did we get, did any of these, do we get the smoke out of any of these? Or No.
• 1:28:28
  Speaker 7: No. I I knew these are gonna be hot, and then that's why I lit a fire under Matt. Be like, dude, we need, I we need to incorporate the temperature sensing and the QSFPs into the, thermal loop. And so Matt goes and implements that, and then he's like, hey. These aren't reporting a temperature.
• 1:28:43
  I'm like, oh, crap.
• 1:28:46
  Speaker 1: Yeah. These get but they must get toasty because you've got I mean, you've got 40 amps disappearing in all these. So this is definitely getting getting warm.
• 1:28:53
  Speaker 7: 320 watts of sidecar. Yeah. They get toasty. But we can keep them under, keep them under 60 c easy.
• 1:29:02
  Speaker 1: That's awesome. Alright. So then this, and this image goes with the b j image earlier. So, Eric, do you wanna explain what this is?
• 1:29:11
  Speaker 7: Yeah. So this is part of a test adapter. So this is a our network switch is a 4,000 some number BGA, ball BGA, and it takes, the order of 500 amps at 0.8 volts. And so you ideally wanna check that out before you dedicate one of those really expensive things to testing. And so this adapter is made to go in place of that really expensive network chip so that you can test it with external loads.
• 1:29:44
  So the first one we got we got from, you know, Load Slammer, the company, which, you know, they did they did good work, and it was fine. But it's one of these things where, you know, we certainly have the capability of doing it ourselves. So this is my first try at doing it ourselves, with the 4,000 pin BGA and all that. So I'm going to compare the performance of my design versus the other design and, you know, see if there's any gotchas there. But, this is another as we go through here, kinda back to the podcast where we talked about building tools, this is part of the tool development for oxide and the fact that we need to be able to make these kinds of adapters ourselves without paying some external company a lot of money for something that we should ostensibly be doing ourselves.
• 1:30:28
  Speaker 1: Absolutely. And the ability to generate loads synthetically on these things and then study it has been I mean, that was essential for both the the gamut for the for the compute side bring up and the switch bring up. I mean, we Eric and I know that was another one of the very first things you did when you arrived at Oxide is, like, we have got to get a SDLE equivalent for sidecar, because we're gonna need it to be able to hit this load step and be able to to debug this without actually having a chip in place. We do not wanna debug this in terms of dead chips.
• 1:30:56
  Speaker 7: No. That gets expensive real quick. Turns out that really expensive load adapters aren't that expensive when you consider how much it costs to replace one of those network switches.
• 1:31:06
  Speaker 1: Yeah. Amen. And then, we're talking about the we get we the the cat makes an appearance here. The, from the from Gemini as well.
• 1:31:15
  Speaker 7: Yeah. Grumpy cat. If you, if you insert a coin to the slot, it'll, it'll tell your future.
• 1:31:22
  Speaker 1: Yeah. Especially if it's hot.
• 1:31:24
  Speaker 7: It's, I predict melted zinc.
• 1:31:26
  Speaker 1: No. Exactly. And then this is the the next one is your updated that's 3 d printed. Right? That, the power adapter?
• 1:31:33
  Speaker 7: Yeah. So this is an updated version of that huge board with the enormous, Phoenix terminal block on it. And this is a much more stealth version that is meant for more ready ready use. So this is a 3 d printed enclosure made out of ESD, PLA, and it has the right offset so that it'll plug into our sled and be, you know, on the bore on the, surface so that it's not, like, hanging out in space, which is nice. And it's a much nice smaller connector.
• 1:32:03
  Speaker 1: It it it it looks it definitely looks, pro grade for sure. And then what is the, what's this next board? I think I think I've seen this for the first time.
• 1:32:11
  Speaker 7: Yeah. So this is one that, again, goes back to tools development. So this is the synthetic load development board. So this is basically 3 MOSFETs in parallel, generating a a current a current load based on analog input signal. So this is looking at the control the control loop from the, little op amp that's up at the top, And then it'll drive these MOSFETs to be kind of like an SDLE or a, you know, load slammer or whatever.
• 1:32:39
  So this goes towards developing the, synthetic load boards that we'll have to use for DDR DIMM testing, because DDR 5 uses 12 volts. So we don't have any more DIMMs that we can buy that have that kind of synthetic load on them, so we're gonna make our own. So this is the dev board I'm using I'm going to use to develop that before I make something more fancy with a micro on it and a to d and all that stuff.
• 1:33:09
  Speaker 1: That's awesome. I do love how quickly can the MOSFET desolder itself. I
• 1:33:14
  Speaker 7: It Yep. It happens. I've seen it. I've done it. It's it's it's it's fascinating when it happens.
• 1:33:21
  You're like, did did I seriously just desolder that thing? Yes. I did. Oh, crap.
• 1:33:26
  Speaker 1: Right. And then, finally, we get to this is the the the Kluge shot 2, is going out to the big city, Nathaniel. This is this thing is getting all pro now.
• 1:33:37
  Speaker 3: Yeah. So this this is what happens when you're at in the bring up lab, and Eric sees your really janky kludescot 2, the first one. And, we we were in kind of a board rework cycle, so there was a a few down cycles. And he was like, you know, I can just pull, CAD up with the mechanical CAD for our sled and actually, like, get all the parts aligned and, like, do everything fancy. And so, like, you know, he does that.
• 1:34:02
  And then he's like, well, you know, it's kind of annoying that the PCIe connector and, like, you'll notice the PCIe connector on the original Clujot 2 is the same one that I had sourced for the VPD programmer. So I just had those in stock, and that's why I use those. But Eric was like, you know, we could probably put the NIC in line with a different u.2 connector or with a different PCIe connector. And so, you know, like like refinement on designs, Eric made it, you know, a 1000 times better than, it was originally. And, and then, you know, he did, like, controlled impedance and everything, which I had basically just skipped on because we were we built the, original Clueshot 2 kind of, you know, in a crunch trying to get it done.
• 1:34:45
  And, we went with a fab house that didn't do controlled impedance. And, like, the NICs are only by 1, so it's not really that big a deal. And they're, like, you know, gen 2 or something, gen 1 maybe. So it's not that big a deal, and they work. But the ARIX improvement is just like an improvement across the board, and we have a 3 d printed enclosure for this and everything.
• 1:35:05
  So he he can probably tell you more about that even. Well
• 1:35:09
  Speaker 1: and this is where, I mean, we are now have now learned how important clue shot 2 is that, you know, we kinda did clue shot 2 to kinda solve one thing, and now we're using it for a bunch of other things. And it's actually helpful to get this thing Yeah. Kloojot 2.
• 1:35:23
  Speaker 7: Convenient to be able to to get an Ethernet port out the front of a rack.
• 1:35:28
  Speaker 1: It is really convenient. And then it's also I guess, it's convenient too to be able to to, you know, you get the necessary kind of IO that you need on the manufacturing line. But then it's also great to be able to pull that surface area out, and not have to ship with this, you know, not have to ship with regrettable IO interfaces that are never used in production.
• 1:35:45
  Speaker 7: And setting up this attack surfaces and all kinds of other horrible things.
• 1:35:49
  Speaker 1: And then everything else. So, yeah. This is, the Kloochart 2 is growing up, but I you know, it's I I don't know. Bittersweet, I think. I don't know.
• 1:35:57
  I'd, yeah. The the the red Kloochart 2 is always gonna gonna play a gonna play a special role.
• 1:36:03
  Speaker 3: Yeah. Well and this one is like a grown up design because I like, Eric and Ian, dropped the thing into our, like, simulation software too, and so, like, the VRAs have been tuned, and, like, things are actually nice here. So
• 1:36:17
  Speaker 1: there you go.
• 1:36:18
  Speaker 3: Well, it's not a real board.
• 1:36:20
  Speaker 1: It's a real board, but it it's also based on the feedback that, like, hey. We're gonna be using this thing in a this is not going away anytime soon. We're gonna be using this thing for a while, and getting this thing just totally nailed is is, a good use of of time and energy. Well, that's a that's a great whirlwind tour of a lot of boards that we have done. I mean, the thing I don't know what you thought, but when I was actually, like, beginning to catalog all these, I'm like, I'm I already know that I'm forgetting things like admission led, for example.
• 1:36:49
  Yeah. We've just done a lot of these. And, you know, Cliff, as in so many things, I think you were very prescient about the the the Swiss army knife effect of Gimme Whitlet, and this incredible superpower, Nathaniel, that you've now honed of being able I mean, having our own in house pick and place, it just allows us to move that much faster, and I feel there are a lot more of these boards in our our future. I don't think we've regretted any of them, honestly. I think that they'd be they have all been and being able to explore small ideas, big ideas, and being able to use it to distribute the team, to have allow everyone to have, again, what let, for example, has just been enormous.
• 1:37:39
  Speaker 3: So it's
• 1:37:39
  Speaker 1: did not
• 1:37:40
  Speaker 3: yeah. Drives driving that driving that culture, you know, other people who maybe aren't going to build the board have the opportunity of coming up with ideas for, like, boards that should exist. And so, like, I I always feel like that's kind of a fun conversation, like, should this board exist? And, you know, we can talk through that as a team. But, like, as people find needs, like, those are things where, you know, we've we've lowered the barrier of entry to getting a board.
• 1:38:03
  So you're you're not gonna go it's it's not a 3 month design cycle with, you know, a bunch of reviews and, you know, tens of 1,000 of dollars. It's something that we can, like, you know, turn the crank on pretty easy.
• 1:38:13
  Speaker 1: It turned the crank on really easily, and, boy, has it been useful from a software and hardware perspective to just to explore different aspects of the system. I'm just are just using them absolutely everywhere. And it's been it's been tremendous. And I I think I think also your point, it's like it they really being able to do these boards quickly and lightly and turn around quickly, really fosters a a sense of teamwork and collaboration where we've got I mean, just like the dimlets where, I mean, I the dimlets were so meaningful to to me to develop that particular software that we really needed that was gonna be really painful otherwise. And I think it's been true for a bunch of these other boards where they may not have a huge audience, but that audience is really, really appreciative, and really allows us to move much more quickly as a broader team.
• 1:39:03
  Well, I think that, I I know that that was more than an hour. So, my,
• 1:39:10
  Speaker 2: this was great. I mean, I I, you know, this should be a talk that we pitch to a conference because I think that this was just such a great tour through the product at large and all of these really innovative solutions to problems that I don't know, would have been so much more expensive and time consuming and laborious, without the kind of tight integration between hardware and software. This is so fun to go through.
• 1:39:34
  Speaker 1: It is fun. And I have said this whole, that all of these proto boards have just, I feel gives us a lot of agency. It just feels like with any problem, we've got all these different tools that we can bring to bear as a team to go solve them, and it really feels exciting. It feels like, you know, we're we're we're ready ready for the crisis, which is a good thing because
• 1:39:53
  Speaker 2: And and the fearlessness to build the next one. Right? What when's it I mean, once you've built 10 or 15, you know, building 16th or 17th, it doesn't feel like a big deal.
• 1:40:02
  Speaker 1: Yeah. And I think that, also and then, you know, I know we've talked about this, but have we opened up any of the certainly, there's no reason for us not to open up these designs. Have we opened any of these up? I'm not sure if we actually have.
• 1:40:10
  Speaker 3: I don't know that we have actually opened them. There's really no reason not to. I think, probably some of them need a cleanup pass for properly documenting the mods in the actual designs as opposed to over in our, like, you know, rework tracker. So that that's prob and and, you know, that's probably mostly the reason that we haven't done it yet because it we have you know, we're also trying to ship a product.
• 1:40:34
  Speaker 1: So we are trying to ship a product, but I does feel like some of these, like, I the Gimletlet and the Donglet, maybe Kloojot 2. I mean, a couple of these are gonna be useful to other folks as well. So, I'd love to believe the Dimlet's gotta be useful to someone else too. I know that we're not gonna have a problem.
• 1:40:48
  Speaker 3: I feel like DDR 4 is, like, ship is already sailing, though. You know?
• 1:40:52
  Speaker 1: It's true. Yeah. I know. Well, thank you so much, Nathaniel and and Matt, Cliff, and and Rick and Eric. Thank you for all of your hard work on all this, but also thank you for for walking us through all of these.
• 1:41:09
  This has been really, really fun to go through all of this. So thank you very much for the extended time tonight. And, Adam, thanks for, I I this is something I know was driven by the demo that that Nathaniel gave on Friday, but this has been a lot of fun to go through all these.
• 1:41:24
  Speaker 3: Yeah. Super fun.
• 1:41:26
  Speaker 1: Alright. Thanks, everyone, and we will see you next time.