Latest / Oxide and Friends / Open Source Firmware

Transcript

Open Source Firmware · Oxide and Friends

• 0:00
  Speaker 1: How are you doing this week?
• 0:00
  Speaker 2: I am doing well. The yeah. And, I'm glad that Antranig is here because I've we there's been so many times, but he's been up at at 2 in the morning. I'm like, if anyone has earned a 5 PM local time Twitter space, it's intradix
• 0:18
  Speaker 1: though. Absolutely. I don't know why my mind turned here, but I was thinking of, bloopers recently. And do you remember the Twitter space we had where, like, I leaned on the keyboard and the recording had, like, 3 minutes of of beep booping. Anyway
• 0:33
  Speaker 3: It's fair. This is fair. This is actual, justice. What time is this there?
• 0:40
  Speaker 2: There's Christmas.
• 0:41
  Speaker 1: 9. It's it's very civil here. Yes. No. It's a it's a reasonable time for us to be.
• 0:45
  Speaker 2: It is a and not on the wait a minute. I know I've got I know exactly where you're going right now and it's not what I okay. Fine. We need to do 2 Twitter spaces on time zones. 1 on what you wanna do, which is the the the the time zone the time zone database.
• 1:12
  Speaker 1: No. No. I think we literally did a Twitter space on time zones.
• 1:18
  Speaker 2: Oh, that's why you were interrupting me because you felt that, like, that we've already done this.
• 1:22
  Speaker 4: Like
• 1:23
  Speaker 2: yeah. I'll be on time.
• 1:25
  Speaker 5: And then
• 1:25
  Speaker 2: we and we and we talked to the you know, so I don't mean, like, the nerdy parts of time zones, although all that too. I mean, actually, more like the human side of working with people across these big offsets? Because the they're like, that's not going away. Right? I mean, that
• 1:41
  Speaker 1: okay. Okay. Okay. Okay. That totally makes sense.
• 1:44
  I I was like, is this deja vu? Did I dream this whole episode? Because we we did, you know, the train stations in 1911, and I think Right.
• 1:52
  Speaker 2: We we did. I no. Yeah. So we we we what I mean more is the just the challenges of dealing because I mean, for example, a 12 hour offset, I think is unworkable. I've never seen it work.
• 2:08
  Speaker 1: Yeah. I mean, which makes some of this, like, you know, teams that are split between North America and India, I think, extremely challenging.
• 2:14
  Speaker 2: Really challenging. And it's actually something that makes it you know, something that when, Ed Yordon had this book, The Decline and Fall of the American Programmer, which I think we've sure I've mentioned once or twice around here, but absolutely terrible book. But they, written in nineties about how all this this software engineering was gonna be in India. Yeah. And it doesn't deal with time zone at all.
• 2:35
  And it was my first time dealing with just at any 12 hour offset, between the US and India. But a 12 hour offset is just absolutely brutal because it means that I mean, somebody is absolutely off hours in that conversation. Like, you're not both on there is no 12 hour offset in which you're both kind of, you know, it's like what? 8 PM and 8 AM maybe? I don't know.
• 2:57
  But it's it's brutal.
• 3:00
  Speaker 1: Yeah. My my brother's company where he is the VP of Enjush, they are, spinning up a bunch more folks in India, and he says the way they do it is pretty inhumane. They asked the folks in in India to stay up until 11 for the all hands at
• 3:16
  Speaker 2: So I would actually probably go one step further. I would actually ask them to work nights. I mean, which is also brutal, but it's, like, actually some if you work people do work nights. Right? That is something that people do.
• 3:27
  People do work 3rd shifts and so on. So
• 3:31
  Speaker 4: Yeah. But but
• 3:32
  Speaker 1: as you say, we're really tough for
• 3:34
  Speaker 2: So, Christian, I'm trying to add you as I keep trying to add you as a speaker, and I'm I'm not sure what, oh, is Christian not on the mobile app? I thought Yeah. Christian You gotta be on the phone. And sorry. We, if you're not on the phone, it won't let you speak, and I don't think it will tell you anything.
• 3:53
  I just think we can't turn you into a speaker. It won't tell us and woah. So, but so I was in the so, first of all, it's great to have, our European friends with us. I I definitely got a full taste of this when I was out, and and the that's just a good kind of entree into this topic on the at the open source firmware conference, just this a couple weeks ago. And, yeah, working out of Europe, being in Sweden and trying to work with folks on US Pacific is it's it's hard.
• 4:25
  You end up I ended up getting a lot less sleep, honestly.
• 4:29
  Speaker 1: Oh, that's what the so I spent a month in Prague when I was in the Solaris kernel group
• 4:33
  Speaker 3: Right. To
• 4:34
  Speaker 1: bring up that office. And and it was great because during the day, I could get all this work done. No one was sort of asking for my attention. Then I would spend I would stay up until 3 in the morning watching baseball on MLB TV and, like, responding to contemporaneous emails.
• 4:50
  Speaker 2: Yeah. And I think it, like, it it is you do have when you got a that kind either it's, like, just enough overlap where you can kind of almost plan your day and get some time. Alright. Christian's back. Let's see if, oh, and Christian, I think he is a speaker.
• 5:07
  Hey. You're here. Very good.
• 5:10
  Speaker 3: Alright. Yeah. I was indeed
• 5:11
  Speaker 2: I'm so sorry.
• 5:12
  Speaker 3: So I thought that just works.
• 5:14
  Speaker 2: No. That's a very reasonable I know. I did like, if Musk announces that he is gonna allow Twitter Spaces to operate from the desktop and also that he is no longer gonna allow HPE GreenLake to run ads, my feelings about this whole thing are gonna become a lot more complicated. But I I well, I'm, I'm glad to hear. So, Christian, I I wanted to maybe pick up with the open source firmware conference.
• 5:42
  Because, Adam, I don't think you you've not gotten a chance to go to one of these yet.
• 5:47
  Speaker 1: No. Never. I'm I I this the legends that you tell sound amazing, but, no, I haven't been.
• 5:52
  Speaker 2: So, actually, where are we getting to the origin of that little bit because I think the first OSFC was in 2018. Is that right, Christian? I was not there. But is that is that the first OSFC?
• 6:02
  Speaker 3: I think Philip can actually add more because he's the original founder of that conference. And, he can probably give some more insights on that.
• 6:12
  Speaker 2: Yeah. We can hear you, John.
• 6:13
  Speaker 5: By the way. Oh, okay. Great. That's good. Yeah.
• 6:18
  So yeah. When when did it start? So I think it was, like, 2,000 Chris, correct me if I'm wrong. It was 2017 or 16. I'm not sure.
• 6:27
  That was the first OCC we did. That was an airline. And, so the OCC basically came out of the idea from the Cobble conference. So I like the Cobo project, and, I was I was, like, a long time, part of that and also also Cobo as a Cobo developer and so on. And so we had to, like, European and American Cobalt, conferences.
• 6:49
  But the problem was, like, we could never meet with UFI guys and with other people, in this firmware world. And so we decided to to basically, start, yeah, something which is more broad for broader, let's say, for the public more, and you have more different topics you cover instead of just talking about. And so that's where the the open source came open source came came from. And yeah, that's that's how it went like with the conference. And then later on we started after the conference, we started to think more about how we can can set up a foundation as well, and that's where we ended up resolving some
• 7:27
  Speaker 2: That's correct.
• 7:27
  Speaker 5: Foundation. Yeah. So the first
• 7:29
  Speaker 1: one Yeah.
• 7:30
  Speaker 2: That that's great. So in you said the first one was in 20 17? The 20 18,
• 7:37
  Speaker 3: I think. 2018.
• 7:38
  Speaker 2: You know, it's
• 7:39
  Speaker 5: Yeah. Somewhere there.
• 7:40
  Speaker 3: That was Erlang, basically, in Germany. And then you
• 7:42
  Speaker 2: moved to the US. And then were you, because I mean and and OSSC 2019, which is the last time we met in person until OSSC 2022. Mhmm. Certainly, I was delighted. It was a great conference.
• 7:57
  As Adam says, I I I have apparently become insufferable talking about the legend of of OSFC 2019. But because it it was pulling in so many different folks, it felt like a really old school conference, very, not just collegial, but but very, it was, you know, one of these old school conferences in that you have this kind of niche interest that you think no one else shares. And then all of a sudden you're in a room with 200 or 300 people that care about this really obscure thing, which felt really neat. We're used to prized. Because, I mean, also, I've seen 2018 by I mean, looks like it was another great agenda when I would I sorry.
• 8:34
  I had not heard of it when it was when it was originally launched. Were were you surprised about the about the level of interest, the scope of interest?
• 8:45
  Speaker 3: For 2019, we were a bit surprised that that so many people are coming. I mean, we did it, together with Google and Facebook and their offices. They had interest in firmware, and just came by. So that that was pretty huge. And in 2019, I think there were 3300, 350 people around, which is for firmware, that that's quite a lot.
• 9:17
  Right? That's kind of
• 9:18
  Speaker 2: It is Woodstock, and and it was also I mean, that's I think that's a good analogy even though, again, Adam's throwing his eyes even more being like, look, I didn't go to this thing and I wish I had. Okay. So just, like, knock it off everybody.
• 9:30
  Speaker 1: That's right. I've seen the pictures. Everyone was, like,
• 9:31
  Speaker 2: new. I paint everywhere. Yeah. But the but the density was also really high. I mean, this is not like a, you know, this is not something you're going to because, you know, it's not it's not like KubeCon or whatever.
• 9:44
  This is not a Linux Foundation conference. Some, you know, massive kind of undertaking. Like, people are there because they're interested into the topic, and so the density, I thought, was really high. And I love, like you did a hackathon, after 2019, which was tons of fun.
• 10:00
  Speaker 3: Yep. Yeah. I think, what we're trying to do is so to keep it very developer centric. So that's, like, our real focus. So we don't wanna, like, huge product announcements or whatsoever, but rather than really, really keep it to the core and keep it to what the people are interested in, I hacked on this and that.
• 10:19
  And, I mean, you see the talks. Right? Like, some of these talks really start that, yeah, I know it's hacky, and I know it's it's kind of crap what I did, but it of worked in the end. So that's nice, and you just share that. So that was great.
• 10:32
  And, the hackathon afterwards, I mean, we do that in the office, like, where we're coming like, the company that we work for, we do that as well. Right? Sometimes we just sit together and and hack on things. So we have a Friday off where where we can hack on projects, and we just love love to do that in our free time. So why not bring that into a conference where people, like minded people, are around and probably also wanna do that?
• 10:56
  And I think that that was that was a great Hey.
• 10:58
  Speaker 1: And, Christian, have have you seen industry participation over time? You know, both attendees, I know that they're not making product announcements, but folks from representing companies, tuning in?
• 11:12
  Speaker 3: I mean, obviously there are a couple of people from companies, right?
• 11:15
  Speaker 1: So Oh, pardon me, I guess I mean like, maybe offenders when it comes to non open firmware.
• 11:24
  Speaker 3: No. No.
• 11:25
  Speaker 1: Okay. Okay. Okay.
• 11:26
  Speaker 5: Not yet. So Not yet. The the funny part is also the hackathon ideas basically comes from the German hackerspace community. So I'm I was part of, 15 years of a hackerspace. So and so they use it also for the hacker, so for the biggest hacker conference worldwide, the Chaos Communication Congress.
• 11:46
  And so they have also everywhere hackathons about different topics. So I think it's it's quite famous German idea to together and hack on things. So it's also deeply rooted in the companies and, yeah, in the hacker culture as well.
• 11:59
  Speaker 3: Actually, in 2019, before the OSFC, we also did a small hackathon, in Bochum, in our hackerspace. Right? So in that local hackerspace. And, was that was really great, but that was really different, really. So that you have to imagine this, like, room with 25, 30 people from all over the globe.
• 12:20
  So I don't know. David Hendricks from the US came in. Right? A couple of people from Facebook, from Google, from the Netherlands. Like, really, a lot of people for 4 days just hacking on on firmware.
• 12:31
  Oh, that was great. And the hackerspace is right, underneath the methadone clinic. The the the the those set up really is, yeah. It it looks like you could imagine a hackerspace, right, like an old couches, you know, in that one corner and, like, LOD panels on the walls, you know, and you can play some games here and there. And, there was one guy really dressed up in a suit because he thought it's like like like a like a serious event, I guess.
• 13:00
  Right? So he he came in with a suit, right, and has, like, a small small suitcase where he has his laptop. And he came around, looked around, was like, okay. And he completely was kind of lost. They were like, okay.
• 13:10
  What what's going on here? Am I right here? Or what's going on? And that was, like, pure fun, actually. And next day, he came with shorts and t shirts and just started hacking on stuff.
• 13:20
  So that was great to see. And I think this is, like, what we do and and what we love to do and why not bring that also? Yeah.
• 13:27
  Speaker 2: I know. That that's great. When I think it's you also get this amalgamation of a lot. I mean, you obviously have a bunch of big companies there that have and you've done an excellent job of pulling in because you wanna have, you know, the the HPE is there, and you wanna have the I I don't know that we've had any AMI speakers in AMI, but you've had an Intel presence and an AMD presence and certainly in Amazon and Google and and Meta presence. But then you at the same time, you have folks that are there because that that are are individuals that are not that are either students or they are that they are hackers that are just interested in it.
• 14:04
  And you get this, I think, interesting amalgamation of of folks. Actually, it was with, at least one person that was at OSFC. An American was at OSFC in Sweden 2022. He's like, hey. I'm I'm actually here because you you guys were talking about it so much on on the metal.
• 14:21
  I really wanna to go, like, oh, wow. Alright. This is why I hope, but it he had a great conference. It was fun.
• 14:28
  Speaker 3: So so nice sales there, basically, that you're doing for us.
• 14:31
  Speaker 2: Oh, wait. Well, I mean, I I think that, honestly, I I I don't I mean, I I feel that I'm just expressing what's going on. I feel that there's a real grassroots movement, and I feel I mean, it it feels you know, for so long in my career, and I I mean, you must have felt the same. Adam, I know you felt the same for sure, where it just felt like firmware was going to just stay proprietary. It just did not feel like it was on a trajectory to be opened.
• 14:57
  There was, I I mean, I remember a particularly, low moment talking to Intel in 2020 in 2012. And, one of our our current colleagues, Keith Vasosky, I was there with the Barbara Stockey, Keith and I were talking to the Intel rep, rep, talking about and being just emphatic that we we absolutely needed, open firmware and the classic intel in that the engineers in the room didn't disagree with us. Like, no. No. I yeah.
• 15:26
  No. Those are all really good points, but it just doesn't feel like it's gonna happen. And for like, it doesn't feel like it's gonna and, like, they they were not I mean, they were supportive of it in the abstract, but I think all of us in the room felt like just this is just not gonna happen. It's not on a trajectory to happen. And it feels like, you you know, that was only 10 years ago, and it feels like there's a lot of momentum.
• 15:50
  Am I am I delusional or or have things shifted in the last 10 years?
• 15:57
  Speaker 5: So I can probably answer the question. So I'm I'm I'm not that long there As Ron Minnick, who's the founder of Coboot, project or Stefan Reinauer, which are both, like, basically founded everything and started with open source in
• 16:13
  Speaker 3: 1999.
• 16:14
  Speaker 5: But, in general, it's always an up and down. So it was always in fight, with soft vendors and also with OEMs to to succeed what we plan to do. And so it was like in I would say in around 2,000, it was, the quite easy thing because, they still started early and there was not so much out there. So, I mean, the overall growth on a chip level is, like, from the 2,000 from, I would say, 256 kilobytes or probably we already had 1 megabyte, but I'm not sure. That would probably say less than that of, of firmware on on the SPI flash.
• 16:50
  And now nowadays, it's 32 to 256 megabytes. So, the feedback was not that as a lot not a lot. And so the law of vendors, they still made it a so still made it, possible that you can launch your own firmware on the devices without any security restrictions. And so you could basically, do your own thing. Right?
• 17:12
  So as long you know what you're doing there, you probably can write your own and launch it. And so it was easy. And then in 2,000, I think, 12 or something like that, they got, like, security technologies like Gutgaard from Intel and some other technologies introduced for security reasons. But the thing is it made it it made it harder for for individuals to work with the topic. And so we had to it slowed down a bit, and then it just started to to grow again, I think, around 2016, or 17.
• 17:44
  Not sure. But around that time, it got speed speed up. And so it was also, finally, people got awareness about the topic of firmware and development in general and also firmware security. So that was quite important, and security was one of the major points for a lot of companies, transparency control as well. And so, it's somehow started up, and now we we are on a good way, I would say, to achieve the goal with open source, but still a fight.
• 18:11
  So it always goes down and up and down and up depending on which kind of is is trying to move into the right direction. But I think with even with with 5 and and other soft vendors being more open, it's, it's a clear shift that we see in the future. So yeah.
• 18:26
  Speaker 2: Yeah. Wait. And so that that that's a good point too that it was it was actually more open at one point, and then it actually became more proprietary. And that 2012 moment may have been actually that's interesting that that may I hadn't really thought of it that way, but that may have been a real low point for in terms of kinda maximally proprietary firmware or kind of entering this maximally proprietary era. And you talk about the security for sure.
• 18:51
  I think also there's a, I mean, security was both prompting these things to become more proprietary, the security through obscurity. I I I think for me, the big issue is this the pace of platform enablement. And, I mean, one of the things that we've talked about before, but, man, I'm definitely we're on just on the front lines of, is how much comp more complicated these SOCs are. And, you know, the end of Dennard scaling and in 2007 and what that meant in terms of pulling more cores on DIE. And then if you pull more cores on DIE, well, now you have to have this other software that actually manages those cores.
• 19:27
  And those cores, those management cores are are completely opaque right now. The PSP on AMD, the SMU on AMD are all completely opaque. The ME on Intel is completely opaque. And I think it's really important that we like, we the microprocessor now is this hybrid of software and hardware. And that software component, it's not just microcode, and that software component is remains proprietary.
• 19:59
  And we've really gotta open that up, because it's it's it's impeding our platform enablement. It's a it's a it's a real
• 20:08
  Speaker 3: problem. Yeah. I also think that it actually really got more momentum within the last couple of years. Right? So I'm I'm in that space since, like, 4 or 5 years now.
• 20:17
  And so, probably, don't have, like, the huge view back basically on how it was, like, 15 years ago. However, in the last 5 years, you really you really feel that momentum, basically. Like, a lot of people, a lot of companies actually, want to move to open source feedback. Exactly what you said, Brian, to enable their platforms faster, to be able to debug their own platforms, like, what's going wrong and what's not. Right?
• 20:41
  Because the reality is that if you cannot if you cannot debug it anymore for whatever reason, you just have to send the platform to Interim and say, you know, fix it for me because I cannot do it anymore or to to any other SoC because you cannot look into the code anymore. And that's, like, a real big issue that you send out your platforms for a couple of weeks. Right? This SoC vendor starts debugging the whole thing, sends it back to you. That kind of delays the whole process by weeks or months, basically.
• 21:08
  And we see that, that that a lot of companies really go into that route. Hyperscalers, but also, like, medium sized industry, companies that, that they really wanna have control over how they can enable their platform and how they can debug their platform and how they can maintain it. Yeah. And that's that's a huge thing.
• 21:29
  Speaker 2: So do you think that the Bloomberg super micro story helped accelerate this at all? Because the this is you recall the story and, Adam, you read the story, remember, in in 2018. Yeah.
• 21:40
  Speaker 4: Yes.
• 21:41
  Speaker 2: And, I I wonder if that story, which is, seems to be false, has never been verified, but I think could have been true. I think there was a plus Trammell Hudson just got a have you seen Trammell's talk on this, Adam? Oh my god. It's so good. So Trammell Hudson who is delightful and have you ever if you bet Tremble, I you spent time with
• 22:04
  Speaker 1: Oh. No. I I think I just know him through the on the metal
• 22:07
  Speaker 2: Oh, I I episode. Right? Tremble. Tremble's so great. And it actually well, highlight for me and for I was in c 2022 with it being able to hang out with travel in Sweden.
• 22:17
  But, and you you use travel's firmware, I think. Think. Right? I
• 22:20
  Speaker 1: don't That's right. The, mesh Right. Yeah. It's awesome. I mean, it I mean, firmware on the camera.
• 22:29
  Also, huge respect for the guy for because he does he not only made this amazing thing, he also let it go. And, I feel like I'm often o for 2 on that.
• 22:39
  Speaker 5: But,
• 22:39
  Speaker 1: I think both both of those are incredible skills. Right? To make something beautiful and really hand it off to the community is is extremely impressive.
• 22:46
  Speaker 2: Yeah. And it's kinda like the way he actually does his preferred mode of operating is, like, I like to, like, you know, kinda go into this kind of new territory, open things up, and then hand it off. And I like I like to find people that I can hand it off to and, you know, the magic lantern that which the can the firmware on the Canon d the DSLRs, DSLRs, which I always I mean, man, the guts, I just do I do not have the courage to put my own firmware on something that I have paid too much money for. On, like, an SLR or like a car. Like, I just
• 23:15
  Speaker 3: I mean,
• 23:15
  Speaker 2: you you know the degree there which I've injured myself just an oxide. I mean, I would I would absolutely injure myself putting my own firmware on a car. I would. And I've got total admiration for the folks that have that that are that are are doing that. But so Trammell has this great talk called mod chips of the state, where he shows, basically, he's like, could I create something that is this small and and and and could actually meaningfully distort the system?
• 23:47
  And it's pretty interesting and and shows that, like, yeah. This, like, this this could be done. I think and and and, Christian, Philip, would love your take on this. I think this is one of those wake up moments. I think there are a couple of wake up moments for the industry.
• 24:00
  I think that was a wake up moment, and I think Spectre and Meltdown were wake up moments. I mean and I think that those were, where you you had a you people talking about some of these security issues that were the potential at the lowest layer of the stack, and then all of a sudden you had Spectrum meltdown in the same year, 2018, that all of a sudden opened people's eyes. Do you think were those things what do you think about those things that serving as kind of accelerants?
• 24:28
  Speaker 3: Yeah. I totally think that that that is true. Right? So these, these moments that you described there, is really something that that that the people really see now that you can really do harm in the system. That's that's really a security issue.
• 24:43
  Right? And we always get the same argument, on on UFI rootkits, for example, as well. Right? So it's it's the same story always that you say, but there are rootkits out there. Like, yeah.
• 24:54
  But, you know, no one hardly, you know, abuses them or use them at all with these kind of things. But if if, this story happened, like the Bloomberg story with Supermicro, people are really afraid of, oh, goddamn. There is some blind spot in my in my computer that I don't see or on my servers that I don't see that that can really, kind of, yeah, attack or or do harm to the host system. It it And, that
• 25:23
  Speaker 2: And do you do you
• 25:24
  Speaker 5: do you think but do
• 25:25
  Speaker 2: you think that that kind of broader awareness accelerated the desire for open firmware?
• 25:30
  Speaker 3: Totally. Yeah. I mean, firmware per se is like a, I mean, how many people know what firmware is, right, and what's firmware about, and what's actually happening down there in the stack somewhere. Right? You you just don't care.
• 25:43
  I I mean, Philip knows these stories, but, even, like, data center, operators or companies that run data centers do really don't know what what what's going on in their in their lower stacks of of of the platform. Right? What does the firmware do? Do I really care about it? Not really.
• 26:02
  Right? And and these kind of stories that get a lot of attention, I think they they wake people up a little bit and say, okay. Probably probably I should take care of this and probably I should, like, at least check what's going on on my systems, and and how can I improve here there maybe?
• 26:17
  Speaker 2: Well, the fact that it that is effectively impossible to even enumerate all of the cores on a system. It it certainly we the the the number of days that we have had an oxide since discovering our last hidden core, I think now stands at, like, you know, maybe it's, like, 12 or something like that. But it it feels like we are are constantly and I don't know that we could actually in fact, I know that we couldn't, without actually decapping every single component that we've got. We there are absolutely compute elements and therefore firmware that we do not see because there are implementation details of components that we use.
• 26:55
  Speaker 3: Yeah. Totally. I mean,
• 26:57
  Speaker 4: Go ahead, Phil.
• 26:58
  Speaker 5: Yeah. Sorry. So I mean, even the CPU, so the central processing unit nowadays has its own firmware for power management and other stuff. So it contains, like, cryptographic keys. So I would say, overall, it's like, you're having components and components and competence, and so it's getting even more complicated.
• 27:19
  And so as I say, like, 256 megabytes of, you have normally on a modern server board. So that includes BMC, BIOS, Southbridge, additional probably option option on super, PCI express and, also all other things like, like super IO controller and whatever. So even take the m team, therefore, security chip. So it's it's a real, it's growing more and more. And so the bus is getting faster and faster, and that's the main reason why we have this this explosion.
• 27:51
  Right? So, and this future will bring even more.
• 27:56
  Speaker 4: Yes.
• 27:56
  Speaker 5: I'm trying to Yep. Get more more integrated stuff and everything moving in system on chip. So SOC basically system on chip means move everything into 1 chip, and, we're getting more and more there. And I think, think, you probably not only have 1, but you have, like, one which has which is really powerful with law of firmware components in it and soup components. And so, yeah, that's that will be in the future will be a big problem.
• 28:19
  Speaker 2: Yeah. Absolutely. And I think that we are and especially if you you look at these these cores are so small. You can now synthesize a core, be it a you know, we've got a z 80 derivative in the I don't know. Adam, we just saw this t v 80.
• 28:34
  See if it They no. No.
• 28:36
  Speaker 4: Which is
• 28:36
  Speaker 2: this this, Zilog derivative, but it's it is an open source core that is easily synthesizable. So it actually I it it shows up in more places than people might realize. And, of course, the 8051 is is is practically ubiquitous. But the, actually, I think I'm very excited about the rise of the the Cortex M 0 Plus line, which is a super small microprocessor from a number of gates perspective and from a physical perspective. But it's actually like a real 32 bit microprocessor as opposed to this kind of 8 bit nonsense.
• 29:08
  So you can actually run a real operating system on there. So I'm actually excited about this trend. That so to your to your point about, like, we're gonna see more of it. Like, I think we are gonna see more of it, and I think that's a good thing, but it needs to be open. And that's where, like, that's, I think, kinda where we are in terms of, like, getting cracking this stuff open.
• 29:27
  So OSFC has been great, and I, again, thank you both for and and everyone that's been involved in that effort because it's such a it's been such a great conference. At what point did you start thinking about, foundation? Because this is kind of why we're we're doing this now is because you're getting the the open source firmware foundation off the ground. When did that start occurring to you?
• 29:54
  Speaker 3: I think that started somewhere in 2020. Philip and I kinda started talking about it. We that idea was kind of floating around already that that we need some, some umbrella, some neutral ground where we can bind the community together. Because that that's also what ossivec is for. Right?
• 30:14
  Getting the community together at one place, let them talk to each other, present your work and these kind of things. And that that idea, that we have to have these like like a playground where where we can bring everyone together, that came up around 2020. But, you know, sounded kind of crazy at first. So, okay, let's start a foundation. How do we do that?
• 30:39
  I don't know. Who should to who should we talk to? I really don't know. And, so it took a little bit of time until we really really got started on that.
• 30:48
  Speaker 2: And open source, foundations are I mean, it's oh, it's like almost like open source licensing, and then it's like feels like any, conversation about open source is destined to go into kind of the foundation vessel. And, Adam, have you dealt with I I know I definitely have suffered have you dealt with a you've dealt with a lot of foundation.
• 31:10
  Speaker 1: Only a teeny bit. Actually, one of their sort of like umbrella, smaller vessel organizations that was trying to shake me down for some exorbitant some for some very nominal gain on our part.
• 31:25
  Speaker 2: That's it, that feels like the story of Boernecks Foundation.
• 31:30
  Speaker 4: That's it.
• 31:30
  Speaker 2: That's it. That's the summary. That's I mean, I literally you could be talking about any one of their I mean, I I I you're gonna have to be way more specific to have any idea of, what is it so do you wanna describe that a little bit? Because I think that that is and I Krishna, I imagine you and Philip were kind of, like, looking as, like, boy, do we go to the Linux Foundation? That's something that has been, I mean, obviously, I've got strong feelings about, but, I mean, what were they trying to shake you down for exactly?
• 32:00
  Speaker 1: Yeah. That's a great question. That was never really clear. So, so I was I was, talking to, like, you know, again, one of these smaller, organizations that was under the umbrella. They, they they had sort of divided that organization into a technical steering committee and sort of a marketing organization or whatever.
• 32:24
  And, the the the admission fee was for this marketing organization that basically did nothing, allowed us to use the logo that we had no interest in using. And then all of the technical stuff was off to the side and divorced from it. And I think the actually, the breaking point for me is when I got fed up and I looked up in their, you know, 501c6 disclosures, how much the guy the project manager I was talking to made. Yeah. And when he was, you know, making $750,000 a year, I I was just like, you know what?
• 32:55
  That's we don't need to talk anymore. It's fine. I'll I'll just go participate in the open and not use your Yeah.
• 33:01
  Speaker 2: And they got very upset with me. I tweeted out their 990, which is the the the the the federal form that you need to file for when you're a five zero one c 6, which is not a 501c3. 501c6 is a, is it is an industry consortium effectively, not a a charitable organization.
• 33:21
  Speaker 1: That's right. You can't write off your, donations to the Linux.
• 33:24
  Speaker 2: You can't write off your donations to the Linux. And what and so what and once again, actually, I I found really disingenuous with the Linux Foundation is that they had this donations page and you would have these, you know, but I mean, like like like your son who's like, you know, oh, I, you know, I'm enjoying using Linux and I I wanna support it. So, you know, I'm gonna take some of the money I made from, you know, mowing lawns, and I'm gonna give it to the Linux Foundation. You're like, no. No.
• 33:46
  No. Like like, give it to literally any other corporation. I mean, if
• 33:51
  Speaker 1: Or or you know what? Keep it. Like, no joke. That's that is a more no
• 33:55
  Speaker 2: joke. Or blow it.
• 33:56
  Speaker 1: Things to do.
• 33:57
  Speaker 5: Blow it
• 33:57
  Speaker 1: on something. Yeah. Just just just
• 34:00
  Speaker 4: What is
• 34:00
  Speaker 2: the dumbest way you can think of? You 15 year olds can think of
• 34:03
  Speaker 4: The fact
• 34:03
  Speaker 1: you're right. The fact that they have that donation page, you know, to suggest that it's, you know, Bernie Sanders getting, you know, $13.90 per donor as opposed to, you know, shaking down corporations for 100 of 1,000 of dollars is so distinct.
• 34:17
  Speaker 2: It is very disingenuous. And I think I mean, I I I mean, there are aspects of Linux Foundation that I definitely admire that and and there are things that they do that are helpful, but there are they have created this extremely heavyweight organization that I feel is is looking after itself first, just bluntly. It's just like the actual artifacts are not something that so I don't know. It's helpful, Christian. Did you consider I I mean, with all that preamble, did you I mean, I I did you even look at the Linux Foundation, or did you think about that?
• 34:48
  I mean, I'm glad that, obviously, was relieved to hear that you'd that you'd are, by the time you and I spoke, you'd already rejected that route. Is that something you looked at?
• 35:01
  Speaker 5: So probably, I would say we we we we've also thought about that. I think we discussed it once, but the thing is, like, the Linux Foundation, in my opinion, is for for Linux or operating system related stuff. And so it was already out of question to just go with them, because I already got also experiences at Launch Foundation. So I think it's too big and too diverse in my opinion, and, we wanted a foundation who concentrates more on the needs of the FEMBA projects. And so that's why we basically set up the foundation with other reasons as well like, like failure safety because we're currently running the open source swimmer conference.
• 35:40
  And but we want to to have more, companies join that and, make that independent from us. So that was one main point. And I think the Linux Foundation is, as you say, quite big. It's it's hard to work with them. They have probably also their benefits.
• 35:54
  I completely agree. But I also think you need to be like some kind of superhero project that they really take a lot of work, into you because, all other smaller projects, they they basically left out. So that's that's my experiences with Linux Foundation.
• 36:10
  Speaker 2: But Yeah. Well, I can tell you that the large projects are also very frustrated with the way they they they just do not run the the and, I mean, there is and, boy, the number of times I was in the room when Kubernetes is like, we're done. We're leaving the Linux Foundation. We're leaving the CNCF. And which, you know, needless to say, they they they're, that would be not, not in the Linux Foundation's best interest.
• 36:33
  So they definitely were trying to talk them out of that. But they were the actual people that were running the Kubernetes as a project very, very, very frustrated. Because they're not getting I mean, like, the things that I mean, for a software project, like, the things that you need are not flashy and they don't involve, like, gigantic events. I I mean, that, you know, OSFC is a terrific old school conference, but the you know, these these huge events that are are really, vendor oriented are actually not what you need when you're an open source project. Like, you actually, like actually, I could really use some help on documentation or someone to, you know, help triage incoming issues.
• 37:15
  And those are the kind of gritty issues that you really and, you know, as when I was on the TOC, the the technical oversight committee, oversight in air quotes, for the CNCF, that was something that that really trying to get them to be better about giving grassroots support for open source because that is why I think the bit that that is really, really missing. And, boy, I don't I mean, I think that no open source community is more grassroots, I feel, than open source firmware. So I think it's really, really important that you have a foundation that is that consists of the real practitioners.
• 37:53
  Speaker 3: Yeah. I think so as well. And, as Filip said, like, one of the reasoning, really was that I mean, we are planning to, to move the conference basically over into the foundation, And that's why we wanted to keep it separate, basically, and that we have control over everything, maybe, on that part, because it's it's a pretty, yeah, I mean, in firmware, it's a pretty big conference. And that's something that that we own right now, in in terms of because we're running it, but we wanna wanna move that to to the community and and and, you know, let let go of it, basically, so that that if we, 9 elements, are not there anymore, the the the the conference still continues. So we have to have, like, an yeah.
• 38:36
  Something separated that that kind of all the system, the whole thing.
• 38:39
  Speaker 2: We do which is which is great. And I think that, you know, certainly, speaking for we at Oxide, we are not just enthusiastic about about open source firmware. Obviously, open source firmware is very, very important to us, but very enthusiastic to be among the initial members of the Ultra Stronger Foundation. We're the we we're really excited that you're you're doing this, and we're certainly looking forward to to being a part of it. And I think in part because we love the way you're running it, which is, very oriented on the artifacts, same way the conference the conference and the actual, like, supporting these these teams.
• 39:16
  Speaker 3: Yeah. I mean, we're really glad to have you on board. And I think, like, Oxide is one of these companies, that have, so I, have kind of the same mindset, at least in terms of open source and how we should do things that 9 elements, for example, also has. Right? So we we share we share the same spirit there, and I think it's just, it's just good that we, like both companies, basically, are there within the within the founding members to really start that and to to keep that core, that core spirit, basically, in in the foundation.
• 39:48
  Right? And that we that we can move forward.
• 39:49
  Speaker 2: And the thing that I kind of love about open source firmware from an economic construction is that it is basically in you don't have this kind of problem that, Adam, you might have had with this this vassal state that you were dealing with in the CNCF where you got kind of startups that are trying to directly monetize software in the ecosystem. So they have their motives for being involved can become somewhat questionable. We don't really for in firmware, it's like what people are selling is the thing, and the, like, the physical thing that the software runs on. I do feel like we didn't actually talk what the definition of firmware is, and I know, Adam, I'm sure you're rolling your eyes right now. Like, I feel like this is gonna turn into another, like, what is software conversation.
• 40:32
  I got, like, I I I like, no way. But I do feel that, like, firmware necessitates a physical artifact. Can we can we agree on that? Yeah. Okay.
• 40:44
  That sounds great. Alright. Yeah.
• 40:45
  Speaker 1: Yeah. And they're sort of tied together.
• 40:47
  Speaker 2: I I know that I'm kind of, like, inching around, like, the chasm here, so I don't wanna, like, ask to follow in. But I think we can agree that there's a physical artifact. And that physical artifact is normally what is like, that's that's not free. People are paying for that. And that means that there's a very clear business model for companies that are like, we make physical stuff.
• 41:06
  And as a result, like, we can actually, I I don't sell the software. The software is very important. It software helps enable the physical thing. But we are really incentivized. And certainly at Oxide, like, we've got, I think, God's own open source business model.
• 41:20
  Namely, like, we're incentivized to open it all because we're not really worried about someone creating a rival oxide rack. I mean, good luck with all that. But, and we would love to see the stuff that we've done, low level platform enablement with the stuff we've done certainly with Hubris and Humility and the stuff we've done elsewhere in the stack be be used by other folks would be terrific.
• 41:43
  Speaker 1: Yeah. Let me try this out again, Brian, and and Christian and Philip too. You know, as we're talking about this, it's so great that this conference and foundation are so grassroots. And Brian, as you say, things like KubeCon or whatever have gone off the rails in terms of that commercialization in every booth trying to sell you something. As much as it would be great to resist that, I think that would also be a harbinger of success.
• 42:04
  That is to say, there is, I think, in the future, this kind of damn breaking moment where it goes from default closed to default open. And as that happens, then these corporate interests will want to participate and will want to show off their wares. And we'll see it as a venue, not just for the creation of these artifacts, but to evangelize them. You know, it'd be sort of bad, but also, maybe a maybe a good good
• 42:30
  Speaker 2: Well, actually, the the nice thing is, like, those corporations are doing that. It's just they're not trying to sell it to the the conference goers. You you know what I mean? Where it's like HPE talking about what they've done on their BMC and getting open BMC to work on their their ILO effectively, but not actually trying just really trying to show that off, and that's why I try to sell it to you. So, yeah.
• 42:51
  No. I think but I think you're right. I think that that's, like, gonna be an indicator of well, actually, let me ask that you'd post that kind of question to to felt to you in the question. I mean, what do you view as some of the of the the hallmarks of budding success here?
• 43:10
  Speaker 5: Yeah. Probably I can go first. So from my perspective, I think, I mean, probably that sounds strange, but, I think, it's, it shows, yeah, the success if the one of the most proprietary companies working on open so on closed source firmware or general starts doing open source firmware stuff. They're probably not completely committing to it in the right way, but they're already starting to to be more open. So what I'm talking about is Mi, you probably noticed.
• 43:40
  So they're trying to get into the open source VMware space. They're still learning the special way of that. Right? And the let's say
• 43:47
  Speaker 4: say special. Yes.
• 43:48
  Speaker 3: Yeah. Very special.
• 43:49
  Speaker 5: Let's call it special way, but it's already a success that they think about it. I mean before they never thought about it and I think this is like one of the biggest signs that things moving aside from other companies joining. And also that we got a lot of community growth in the open source communities. So I mean, nowadays, I think this is like I I'm not sure how many, contributors we have to the Cobo community now, but I think like around 5 or 700, 800. I'm not sure.
• 44:19
  So it's this is also another, let's say, big growth in that area. Yeah. What do you think, Chris?
• 44:28
  Speaker 3: Yeah. I totally agree on that. That it's good that these companies, try to be more open and and try to kind of embrace open source. Yeah, I totally disagree with the way how they do it or if they do it in general, or if it's just marketing right now. Right?
• 44:46
  So, but that but that that's a different topic, basically.
• 44:50
  Speaker 2: Well, and and AMI is in a very different AMI is very different from, basically, everyone else in the industry, and that AMI actually sells proprietary firmware, whereas everyone else actually sells the artifact, the physical thing.
• 45:05
  Speaker 3: Yeah. Yeah. Yeah. Right. I mean, it's I mean, you have to think about, the traditional IVVs.
• 45:11
  How do they sustain their business model? Right? If the private property, artifact and selling it to you, we are, like, licensing it to you. Right? So, like, how how do they how do they keep that, their business model up and running, with open source?
• 45:28
  And that that's that's a pretty big thing. And I think this is something that they're wondering about. And I think it's good that they're trying to get into it. But, you know, it's not really I wouldn't say that they really, really bought into this right now and that they're really, you know, they're just, you know, dipping into the water a little bit and see what's going on and I think, and and and look around there.
• 45:49
  Speaker 2: Well and I I I mean, on the one hand, I'm very sympathetic and that it is very hard to be disruptive disrupted by open source. But on the other hand, I hope that that and it's it's great to hear, Filip, that you picked got some kind of budding participation from AMI. I I do think it's important that no company is gonna prevent this from happening. And so it's important that they realize this is gonna happen to I mean, a big kind of flip is when companies go from, I'm gonna prevent this from happening to and if this if this happens, it's an existential threat to, this is gonna happen no matter what, and how do I reinvent myself in this world? Because it doesn't necessitate reinvention.
• 46:30
  It's not, which is really hard. And and it may be that the new business model looks really, really different from your business model, but that's something that has to happen. That's not something that that that is you you can only hold back the dam on this for for so long.
• 46:48
  Speaker 1: Yeah. Sounds, Brian, like you're drawing strong analogies with in the open source revolution we saw, maybe in the odds or the 20 tens
• 46:56
  Speaker 2: or whatever. Yeah. And and and and companies, you know, were able to and, you know, for years, Microsoft spent all of its kinda energy trying to prevent open source from happening, And Microsoft could not prevent open source from happening. And they had to reinvent themselves in light of open source. And they've done actually you know, honestly, I I didn't think did you think we're ever gonna see a Microsoft that's as open as the one we've got right now, Adam?
• 47:24
  Speaker 1: No. No. Infantically not. In fact, I was at a conference in maybe 2,004, 2005, and I think it was one of the early open source conferences, maybe the first European OSCON or something like that, where Microsoft announced open source licenses, and everyone lost their mind. And I just thought these are licenses without actually software attached.
• 47:46
  What are we doing here? Like, what why are we clapping? But, yeah, to see to see how they have shifted.
• 47:51
  Speaker 2: Well, actually, that's why that's actually that that dovetails actually with Philip's point in an interesting way that you're saying that, like so there was this these kind of things that felt very performative, but actually did presage some really fundamental changes at the company. So, you know, when we see some similar things for similar things that feel like only gestures from companies like AMI, we should be encouraging of that, which is not necessarily my first instinct. So
• 48:20
  Speaker 1: Yeah.
• 48:21
  Speaker 2: So let me ask you. How the question is, how do you feel about, because what what are the challenges that you have, not when you're dealing with the IPVs, which have their own biosenders have their own kind of, interests, but when you're dealing with, component vendors who have firmware components, or have firmware in there, there's the their their components. And they've got bits that need to remain proprietary. You got these binary blobs. And because I feel like it it it's hard to be a a total purist about this because if you're a complete purist, we won't get the open source firmware.
• 48:53
  So what's your your thinking on on binary blobs? Oh, boy. Sorry.
• 49:03
  Speaker 3: Yeah. Yeah. No. Because it's a big topic. Yeah.
• 49:06
  The thing is this. Right? If you're, if you're trying to I mean, this is what we're trying to we're trying to change the industry on how we how we handle firmware right now. I mean, this is this is what we're trying to do. We're trying to be disruptive to, in the long haul, really change that as as Adam, I think, said pretty good that open is the default standard, basically, and not not close.
• 49:30
  Right? So you have to be very political. Right? And you have to understand their needs and then their their their issues and their fear on, when they open, like, what happens if they open blobs up and these kind of things. And I do understand that they that that this is like an iterative process that goes over a couple of years, until, we have maybe, maybe not, maybe have a have a full open source firmware running running on a on a x86 system.
• 50:01
  Right? Like, with all with all the bits have to sometimes it's feel that you really have to, convince each and every person 1 by 1, right, to to really go that route. But I think at some point, it kind of falls, and then it all breaks together, and then it goes faster and faster, And we will really be there, and have an open system there. So I think it's a long run. I don't understand why we have so many binaries, actually, you know, of VMware, I never understood that.
• 50:42
  Speaker 2: But Well, I think that's yeah. I mean, I I I think part of the reason you got binaries is and I think in many cases, it's not because people are don't believe in in the kind of the inherent goodness of of opening these things up. It's because the interfaces they represent are in some cases, it's not even not documented at all. And, like, there's literally there are no there's no documentation describing these interfaces. And there's a sense of, like, well, we don't wanna open this up because we we don't wanna be committing to this interface, and we kinda need to get people past that where it's like, no.
• 51:17
  No. You can open this like, opening making the source code to I mean, forget the p s I mean, PSP 2 would be great. But but things like the system management unit on on on AMD Milan, it's like you opening that up into the intel management engine. Opening that up, actually allows people to get greater confidence in the thing that you make and allows them to help you understand it, help you, yes, you know, find some vulnerabilities maybe or find some reliability problems. But that's gonna be the you know, that that community will ultimately be very net positive.
• 51:52
  But I think it's hard to get people there when you got these things like, yeah. But this thing isn't documented at all. You've got no way for people to actually meaningfully participate in this or what have you. So as you say, it is a it's a journey for sure. And are you are you finding that people are, that that it is gotten easier to to to make that case to to companies?
• 52:17
  Speaker 3: To SoC vendors or to to, in general, companies that that, build components that that have
• 52:23
  Speaker 2: actually, are you seeing a difference? That that that's kind of interesting if you're seeing a difference.
• 52:31
  Speaker 3: I'm not sure if I I'm here. But I I don't think that that has really got got easier in that term. I think, the the view on open source within within, for example, SoC vendors is a totally different one than from from from the community side. So what what open source means. And and I know if you have the discussions, what about if you if we talk about open source firmware and, we have this blobs where they're do we need them all and and, and can't we get rid of them and and, you know, take a journey basically or do that journey and open them up?
• 53:09
  You always get the same argument that we cannot do it. We already have an open source solution, which is this and that. It kind of consumes the binary, but it works. We can maintain it and these kind of things. And I really don't think that everyone really got yet the, the upside of opening up these systems.
• 53:29
  And everything that you just said, that that really that you get all these, all these these benefits, basically, from the community. Right? I don't think that that it really that it really, got into the minds of of the the people that make the decisions that that the associate vendors there.
• 53:46
  Speaker 5: This is what I would agree on. So, I mean, it's not about, like, a SOC vendor as a company. It's about also the development culture of the firmware development in this company. So Yeah. That's a good point.
• 53:56
  Most of the the the people in there, so not all, but a lot of them, they kept for the last 30 years. And then, let's say, they put it in into, like, they close it and don't get let them out. Right? So don't let any kind of new software development go through to them. So it's it's really, like, the entire VMware development is still based on assembly and c.
• 54:18
  I mean, so probably some projects now starting to use Rust, but I mean, this is like just early stuff. Right? So it's like starting to use Rust, but I mean, this is like just early stuff. Right? So it's like still, like, in the in the old ages of software development.
• 54:28
  And so these people these people have really no background for open source because they're coming from a completely different background, like and that's why it's for them hard to understand why this is a benefit and it's also a change of like you know some people like keeping status quo and so I personally don't like. But, they are not so so open for change. So it's a really, really hard thing to do. Yeah. Sorry.
• 54:54
  Speaker 4: So may I I think we have also as a as customers, to to voice and to value the work you are all doing on this side. I can share the story we did with HPE. When we are buying service and we are buying for 1,000,000 of dollars of of service, we are putting a bonus on the one which are playing fair with open source. Yes. I think it's a way also to share with them that also it's not only a technical point of view to have open source firmware.
• 55:26
  Customers and users wants to get this open, this openness, and we value it. And I can tell that one of the ODM, have lost because of open source. They were not able to do it on the platform we have been selecting, and HVAC did it well. There was, they opened the gen 11 systems to have open BMC as a default. And for us, we said, okay.
• 55:52
  If you go on this way, you will have a bonus on our evolution method. And I think we have a greater responsibility as end customers to value that and to be explicit with our provider that we want open source firmware.
• 56:06
  Speaker 2: Yes. I think that's really Yeah. I think yeah. Exactly. Sorry.
• 56:09
  Go ahead.
• 56:09
  Speaker 3: Because what what you're what you're doing is, you're really putting system firmware or or or doesn't matter if it's a BMC or or host firmware. Right? You make it part of your business. Right? And then and you make it part of your business decision.
• 56:22
  And I think that's a great way to go. And that's actually really, I mean, everyone knows how it works. Right? They have like, the SoC vendors have to have a business case at hand, basically, why they should drive into that direction. And, Erwan, as you said, it's it's great that you say, okay, we have to kind of put our business basically or make decisions, business decisions, based on on on the technology stack.
• 56:45
  And that also includes firmware. And that's, I think, totally the right way.
• 56:50
  Speaker 2: I'll just put it the right way.
• 56:51
  Speaker 1: Yeah. In in a lot of in a lot of these domains, you have not that many players and kind of a prisoner's dilemma, where they're all staying proprietary, and they all have sort of their mutual shared incentive to stay proprietary. But all it takes is one of them to flip. And then, you know, the consumers of those technologies to write that in, Ville, as you say, as a requirement for selecting something or or a bonus for selecting
• 57:17
  Speaker 2: something. Exactly. And I you know, I think that the and we all heard it. You know, no customer is asking for this. And I'm sure you heard it too, Val, when you your people were at were it's like, well, you know, you want this to be open, but you're you're the only customers asking for this.
• 57:31
  And I I used to believe that. I think it it many of us believe that, but then you begin to no. No. No. No.
• 57:39
  I may be I you may not be hearing this from everybody, but I I am definitely not the only customer that wants this. I may be one of the few that's telling you about it. So I do think, you know, you got a very, very good point in that if those that it's the consumers of infrastructure when they begin to use the openness of a platform as a component. And it's not the only component, but it's a component of a purchasing decision. And for the for those that didn't see it, you've got a great talk of at at o s f c 2022 on your journey, to open BMC away from the proprietary BMCs with some terrific anecdotal.
• 58:19
  Some terrific slash horrifying stories of of firmware bugs that you've been that you've encountered.
• 58:25
  Speaker 4: Yeah. And I can also share that, now I'm using this to, enforce open source firmware to also vendors because, I can share that during the OCP summit, we have been visiting many vendors because we have a project incoming, a great one. And we were speaking about all these topics, and I said to them, okay. I know you know the rules. They did it.
• 58:52
  HPE did it. If you don't do it if you don't do it, you have no chance to to pass through because, up to now, the open source firmware was a bonus. Now it becomes a requirement because one did it well. And now we we are in a position where we can increase the pressure to them because one did it. And once more and more I mean, by the time that more and more companies will get right on this topic, we'll be, as a customer, in a position to enforce open source email because one already did did it.
• 59:26
  Speaker 2: Yeah. That's great.
• 59:27
  Speaker 4: Yeah.
• 59:27
  Speaker 1: Yeah. You know, I I think I may be a victim of my most recent experiences here on this one, but I was just talking to I was at a conference for a SSD vendor, and the interactions with the SSD vendor really made me feel like SSD firmware may be a long lagging indicator. You know, they they had a big session on security that didn't mention open source at all. In fact, intimated that open source firmware would lead to a less secure product, which, you know, I sort of made enemies in the room talking about, and, you know, insisting that a a more secure product was a more open one. And, yeah, I get this typical vendor feedback of mumble mumble lawyers something something.
• 1:00:09
  Right? And not a lot of other customers in the room beating that same drum. Interestingly, though, at, you know, talking to other other, you know, customers who not only consume SSDs, but also CPUs, they were really interested in what oxide was doing, and what we were building and how we were building this firmware in the open and booting systems with this holistic insights about its capability. So it's so interesting that they were fascinated by open source firmware on one component and sort of dismissive on the other. And so I think for a lot of these kinds of vendors, it really or or or part of me, customers of these vendors, it really just takes some examples.
• 1:00:47
  You know, show me what you can do. Show me that it exists. Show me that I can can have this, that this exists and isn't, you know, just a fantasy. And then it won't just be one customer saying it. It will and it won't be just to one customer thinking it, it, but it takes that demonstration, you know, of those cases.
• 1:01:03
  Speaker 2: Well, and, Adam, we got you back to the SSDs because I think that there's another I mean, Philip, you had mentioned this earlier in terms of seeing more firmware in more places. We now as you have more and more powerful compute elements and also, like, more that they need to go do, You've got very sophisticated firmware in the SSDs that right now is completely proprietary. We've got very sophisticated firmware in the next, especially with the rise of smart next That's the degree that's gonna you know, you can argue about what degree that's gonna happen or not, but we've got more compute elements out by the next. That's all proprietary. And then we got the I mean, I feel like the the the the one that's really lurking is the GPGPU, which is becoming increasingly, obviously, very important.
• 1:01:42
  And we've got there a single vendor with a complete hammerlock over the entire ecosystem, and that is all proprietary. So we've got a lot of work we we need to go do, to get to, and and I'm glad you were fighting the good fight, and and representing Oxide's good name. I'm sure they're like Yeah.
• 1:02:02
  Speaker 1: Yeah. On. I don't
• 1:02:03
  Speaker 2: Oh, did you like did they like, oh, god. The Oxide people. Oh, jeez. You it's always open for where this open for where the other thing with you people.
• 1:02:11
  Speaker 1: You're the only ones asking for it.
• 1:02:13
  Speaker 2: Exactly. But that that and so, I mean, Christian, do you and actually, a couple of other point I wanna go back to that I also love that you mentioned is the cultural transformation that you're trying to affect. And I did I was talking to a vendor that would definitely remain the aim of us who's like, no. No. Like, we're making good progress because we're using Git.
• 1:02:32
  I'm like, you're using what? Using, like, oh, do you have a repo on GitHub? And they're like, no. No. But we're using Git.
• 1:02:38
  I'm like, okay. You did that sorry. I mean, it's great. I mean, congratulations, I guess. I mean but I I I think
• 1:02:45
  Speaker 1: no. We're living indoors. I know it's terrific. Yeah. You should try.
• 1:02:48
  I was like, no. I I have been trying it.
• 1:02:50
  Speaker 5: Right? Yeah. But this is like is it like using Git as, I would say, the top of the iceberg. So, I mean, someone from our team, David Henricks, from Facebook basically moved once to to the OEMs, to China, and they told him, so the most advanced version control system he could find was subversion. That was only one vendor, and the rest is, like, sending emails with zip files
• 1:03:12
  Speaker 3: of code.
• 1:03:13
  Speaker 2: Oh, that's dark.
• 1:03:15
  Speaker 3: Yeah. Yeah. But that's still that's still the status quo for for a lot of things. Right? I mean, we are also working together on on on other, there are more micro, and so more on MCUs.
• 1:03:27
  And that's still the status quo that you get zip files with this, typical, you know, underscore final underscore something something dot zip. And, that's
• 1:03:36
  Speaker 2: Final final draft 4. Yeah. Right. Oh, god.
• 1:03:40
  Speaker 3: That that's just the still the status quo, basically. Basically. And it feels like that that this whole the the sole software development industry kind of got stuck somewhere, And and I don't know why they don't move forward anymore. And, and I think that's also one of the problem that we need to need to check on that we actually need to, need to push forward there, because that's really something that is lacking behind. Because this is not possible if you do, I don't know, web development, page with a zip file.
• 1:04:15
  Well That that's just not And
• 1:04:17
  Speaker 2: so this I
• 1:04:18
  Speaker 3: I don't know what it means.
• 1:04:20
  Speaker 2: Well and I think it's a very good point that we don't do this in other domains. And in part, we don't do it in other domains because it's a lot slower. If you were to do I mean, did you ever use cold fusion?
• 1:04:32
  Speaker 1: Oh my god.
• 1:04:33
  Speaker 2: I mean, so be there there were proprietary alternatives to the web. Flash, I guess, would be the kind of the proprietary alternative to the web. And those proprietary alternatives, due to the hard work of lots and lots of people, those proprietary alter and all and
• 1:04:58
  Speaker 1: Did we lose Brian or am I lost?
• 1:05:01
  Speaker 3: No. I think we lost him.
• 1:05:03
  Speaker 1: Oh, dear. Cold fusion. That's a tough one to pick up on.
• 1:05:10
  Speaker 2: I I I think Musk must have pulled the plug on that. I didn't like where I was going with that one.
• 1:05:15
  Speaker 3: He he Oh, dude.
• 1:05:16
  Speaker 1: I'm, like, proprietary.
• 1:05:18
  Speaker 2: Somebody has already leaked the email that I'm make I'm forcing everyone to go to cold fusion, or they'll be fired by Friday. Sorry. I I guess you lost me at cold view. But there was a proprietary web. You don't do it that way because it's slower.
• 1:05:32
  It's faster to much faster. Much, much faster. Like, it is and, Krishna, to your point, like, we can't even contemplate doing using proprietary tools to the web because it's so much faster. We all know here in the room that this will be true for that firmware layer as well, that they will go faster if they if things are open, but they don't view it that way. And that's, I think, part of the the argument that we're gonna need to show people.
• 1:05:58
  Lucas, you got your your hand up.
• 1:06:01
  Speaker 6: Hey. Yeah. Thanks for having this, and happy to see it happen during the day as well. Coming back to the firmware blobs, the binary blobs discussion, I wanted to ask a question about that. I'm I'm not an expert in this by any means, but the thing I've always heard is that if they open sourced their their binary blobs, that somehow an expert could infer design decisions about the actual underlying hardware, and that that would somehow reveal their their secret sauce of hardware design.
• 1:06:39
  Is there any truth to that?
• 1:06:42
  Speaker 2: Or For
• 1:06:43
  Speaker 6: Is that hogwash?
• 1:06:43
  Speaker 2: Yeah. Yeah. No. No. No.
• 1:06:44
  That's not hogwash. I mean, for sure, if you were to open I mean, open sourcing the Intel management engine would absolutely reveal design decisions that have been made. Open sourcing the PSP. Open sourcing the the the open is any binary any of those binary blobs to make it open would absolutely reveal design decisions. The question is, are those do those design decisions represent real secret sauce or not?
• 1:07:08
  And I personally would say, yeah. They're design decisions. But it's like by the way, that's not the hard part of doing this stuff. I mean, that's like the the like, me knowing your design decisions does not allow me to get to a design any faster, really, and it's also really hard to execute. So I don't know.
• 1:07:25
  Filip, what do you think? I
• 1:07:27
  Speaker 5: I so so, basically, I call it hardware API. So I see the hardware as API. It has API interfaces, and you somehow need to configure it to run with properly. And so these blobs, interfacing these sock burned into the chip itself. That means, it's it's not real confidential information or not so much confidential information anymore.
• 1:07:57
  So it's probably 1% left or whatever. From 19 from 100%, that means 99% are completely just calling an API. And, I mean, yeah, you can figure out design decisions, but, I mean, this is with all APIs you have, like, REST APIs outside of the rep and everywhere else. Right? So, I mean, how do you develop against something you don't understand?
• 1:08:18
  Right? So it's just, like, completely useless. So in my opinion, it's not hardware API. It should be documented, and it should be, a way it should be, possible to write open source for
• 1:08:29
  Speaker 2: it. But but, Lucas, I can tell you that your that I mean, that that perception is a very valid one in that it is something that even the executives at these companies don't necessarily know. They they they become kind of they get this nonspecific angst that if I open this up, a a competitor will be able to copy this. And, you know, if you're dealing with a, you know, something that's, like, you know, a GB GPU or an SSD or an SoC, it's like these things are not really, like these are much more complicated than
• 1:08:57
  Speaker 1: So especially because the your competitor, your extensible competitor who's gonna follow this already has their own mountain of foundation where where they diverged in terms of their decision process literally decades ago. So you think they're gonna unwind that whole thing to go left instead of right to follow you down this path? I mean, just lunacy.
• 1:09:17
  Speaker 2: Yeah. It is absolute lunacy. And, you know, I did a, I, a talk on cope corporate open source anti patterns. And, what you've I then one of the antipatterns that that I described is this idea that your competitor is going to is gonna copy you. It's like your competitor, not only as you say, Adam, they got their own mountain of stuff.
• 1:09:38
  Also, your competitor thinks you're a jackass.
• 1:09:41
  Speaker 1: That's right. Irrespective of their relative market positions right they they they go to all their customers and tell their customers what idiots you are right and and then they're gonna turn around and copy your technology. Give me a break.
• 1:09:54
  Speaker 2: Right. And I think you and I saw that very visually when we opened to DTrace. The very first thing to be open sourced in in open Solaris in 2005 was DTrace. And we're like, well, it's like, oh, they're they're immediately they're gonna take us and put us into Linux. And it's like, no.
• 1:10:08
  They think we're idiots. It's like, sorry.
• 1:10:10
  Speaker 1: If anything, right, it increased the NIH.
• 1:10:13
  Speaker 2: I think it I think it did increase the NIH, and it forced it to make an even, like, make weirder arguments, basically.
• 1:10:20
  Speaker 1: You know, also harkening back to those data days, though, you know, as we open sourced, Solaris, there was a binary blob. Wasn't there, Brian? Like, there was some pieces of the process.
• 1:10:29
  Speaker 2: Yeah. Yeah. Yeah.
• 1:10:30
  Speaker 1: That, like, we didn't have IP 2 and or or something like that. So it it it was a little complicated even back then.
• 1:10:37
  Speaker 2: Okay. So that's actually a really important point, Adam, because this is, I think, even more true, like, way more true with this layer of the stack, that the reason you can have a proprietary blob is not just because, like, we don't wanna open it up or it's not documented or what have you. It's because no. No. We can't.
• 1:10:52
  We license this IP from someone else. Our agreement does not allow us to open source. Like, that is actually, I think
• 1:10:58
  Speaker 4: Yeah. Well, that makes
• 1:10:59
  Speaker 1: And it's and it's like we've changed it over time. We've carved it up. That company in the meantime has been bought 3 times and chopped into pieces and distributed to the wind. And so even unencumbering ourselves to feel that we were on, you know, stable legal ground. Setting aside, you know, the lawyers, you know, fighting open source more generally, but specifically around this, it's really tough.
• 1:11:22
  Speaker 2: It's really tough. And so and that's part of the reason that, you know, and and Chris' point you're making earlier that, like, when I mean, you can't be too absolutist about this or you, like, you won't get anything. You you actually need to to be we no one likes proprietary blobs, but no one likes binary blobs. But at some level, we want them to be as small as possible and as few as possible. But if if we don't accommodate them at all, we won't end up with any open source firmware.
• 1:11:49
  Speaker 3: Yeah. That's right. I mean, I think Stefan, Reinhard, once made, said that, he rather has, a couple of binary blobs in the femur rather than, having no or no open source femur at all on on the Infinix 86 systems. Right? So you have to you have to do compromises over and over again.
• 1:12:09
  But I mean, it's really not about demanding the the the the maximum kind of thing, but rather than really going the the journey together and trying to explain and see, okay, there's also upsides on open source streaming. Right? And this is what you can gain of it. Right? And and this is how we can move forward together faster.
• 1:12:30
  And I had a talk on on at the OCP Global Summit now, on how the platform enablement changes with open source. Right? Because right now, you have the SoCs and you got, like, a handful of IBVs who have a very strong connection. And it basically all scales at the IBVs. Right?
• 1:12:52
  And the IBVs have to enable, like, all the OEMs and ODMs then in the end. And and and there, it scales. But if you have open source firmware, it scales directly at the SoC, and you get more partners, which are I call them enablement partners, that that you have a group of people which can scale much, much better and and can move much, much faster forward, and do much faster basically, because it's open. Right? And we can we can kind of collaborate on things.
• 1:13:21
  So, that's how these these kind of tiny bits that you have to explain always 1 by 1 so that also the companies really see see the upside and wanna go that way together with you rather than, you know, just demanding things, that that will not end up, very well in the end.
• 1:13:39
  Speaker 2: Yeah. Totally. Sorry, Erankaj.
• 1:13:41
  Speaker 3: I want to
• 1:13:42
  Speaker 4: I can also share Yeah. Yeah. Sorry. I can also share the the way I'm pushing back the circuit salt circuit salt story when my vendors are trying to go in this field. Usually, what I'm telling them is, I'm buying you a hardware on all time ad issue with software, and you are not software vendors.
• 1:14:00
  And you are causing me trouble because of you are not doing well as a software. You should use the communities as you're saying because every time you are trying to do software, you fail. It's usually pretty efficient. I don't know I don't know if it works and when, but usually, it works. I mean, they understand this point, and I'm asking them to offload the software to someone who knows how to do it and not trying and tell them not to do it themselves because they always fail.
• 1:14:26
  It's a it's a drama.
• 1:14:29
  Speaker 1: You know, it's such an interesting one where, in my experience with the SSD vendor recently was that that of argument resulted them saying, no, we are software vendors that they, you know, they think that if they're just NAND vendors with a controller, then they're in this commodity market. And every hardware vendor wants to resist commoditization to some degree because then they just see their margins slipping away. Yeah. So they wanna be software vendors adding software features, you know, mostly that we don't want. But, you know, some folks want them.
• 1:15:05
  Speaker 4: Yeah. But when we're reporting issues Yeah. He's and the the the funny the funny story starts. I mean, all time we are reporting we are reporting issues, and it's always about software, never about hardware. So this is how I'm showing them that on this on this device, you have this on this, I've reported you, and you took this amount of weeks or months to fix it.
• 1:15:28
  If if it have been open, I would have been in a position probably to help you to reproduce, to add traces, to debug, and then to help you finding the the issue. But as you are trying to close source everything, I'm just consuming your stuff, it's broken, and okay. I'm stuck.
• 1:15:46
  Speaker 2: Yeah. But, like, hey. No. We're we're a software company. Like, hey.
• 1:15:48
  Yeah. No. I know you're a software company. Your software's broken, so that's why I need
• 1:15:53
  Speaker 3: you to fix that. Actually, with companies that we work together, so on, like, my day to day job, I've we are running that that IVV or that open source IVV. And, the companies that we work with, they actually already do fix their own bugs. Right? So we provide them, like, open source firmware, have everything up upstream integrated so they can pull the stuff on their own and can build everything on their own.
• 1:16:15
  And what they do nowadays is they say, okay. This is not working. I'm just debugging it on my own, and, I'm just sending you the link to the patch, basically, saying or they are just writing us and informing us that they fixed it on their own. Right? And that we should take a look at it and that it goes upstream in the end.
• 1:16:34
  But they're already eliminating kind of, the need for us. Right? And that's it in that sense. But it's great to see that they are able to debug some of the issues on their own and move forward on their own and not relying completely on us anymore. Right?
• 1:16:49
  And that that's, like, the way to go there.
• 1:16:51
  Speaker 2: Right. That's the power of open source. We've seen that. Hamza Zalal, you've been trying to get in here. Oh, you still there?
• 1:17:05
  Oh, maybe not. So I, one peril of, doing this early in the morning is we do have we end up with hard stops here.
• 1:17:15
  Speaker 1: I
• 1:17:15
  Speaker 2: mean, you know, we got we got hard stops, madam's toddler, normally, when we do. You think? And hard to know, actually, toddler I'm actually no longer a toddler. He's still called the toddler.
• 1:17:23
  Speaker 1: No. No. He's he's he's like a a real boy now.
• 1:17:27
  Speaker 2: I so I thought, like, I've like, Pinocchio. I feel like I mean, he he has always been man sized. So I feel like that
• 1:17:34
  Speaker 1: Yes. Yes. Yes. No. We do get a bunch of you know, your 8 year old is kind of immature.
• 1:17:38
  It's like, no. My 5 year old is kind of large. Yeah.
• 1:17:40
  Speaker 2: That's right. But I'm I I assume that you don't have a meeting with him at 10:30 in the Pacific time. No.
• 1:17:48
  Speaker 1: No. No. No. No. No.
• 1:17:49
  I hope not. I hope not. Right. Exactly.
• 1:17:53
  Speaker 2: Your one on one with him is the same thing, but, I I am gonna have a hard stop here in a couple of minutes. So, so, you know, let's see if we can get, him back in here. Hamza, you are you there? Is the caller there? No.
• 1:18:12
  Maybe not. So, Christian, if folks wanna get involved Christian, if folks wanna get involved in the Office of Firmware Foundation or if they want to to kinda point their vendors, their partners in this direction saying, hey. This is something that we're gonna because, you know, I want another thing you could be looking at is, like, hey. Another thing on our scorecard is gonna be, are you participating in the open source firmware foundation? I know yep.
• 1:18:35
  You told me that you're it's gonna be a long journey to open source this stuff, and you're only going to announce the licenses at the next conference or what have you, but can you be a part of the foundation? How should they get involved?
• 1:18:47
  Speaker 3: So we have a home page. It has a catchy name, open source firmware dot foundation. So it's pretty pretty long name, but I think you can also use osfw. Foundation, to get there. And there you can basically get all the information that you need.
• 1:19:01
  So you can sign up to the newsletter if you want to, or you can join the Slack channel. So you can just drop us an email, or, I think that's that's how you can reach us there. And then you can, you can get involved. Right? So if you wanna be a member, if you wanna start one of the the work streams that we have, so we have a couple of work streams starting right now, which is kind of a focus group on a specific topic.
• 1:19:29
  So we have one about silicon interfaces. Like, how do I interface a blob? Should we kind of write a standard around that so not that not every SOC kind of stuff does their own thing. We have some some payload hand off right now going on. So there there's a lot of things, that we have.
• 1:19:48
  I think Fidip is running a firmware security work stream. So if you wanna get involved there, feel free to drop us an email or join on Slack, basically, and just, yeah, just message us. What we also always looking for is people to help, like, running the whole thing. Right? It it's kind of a beast.
• 1:20:07
  There's a a lot of, like, just organizational stuff to do, every now and then. So if people want to wanna get involved in there, that's also just fine. Right? So it's not only technical, but also a little bit organizational stuff around it, where we can always need a hand, basically.
• 1:20:24
  Speaker 2: That is great and a a lot of good starting points for people. You know, I've gotten I I think I mentioned this last time, but I've had a lot of young people, younger technologists, technologists that are in their careers, who I know have been either joining us on the space or catching the recordings. And if you're a technologist, Royer, in your career and you're looking back, it's like, boy, I I wish I had, you know, been around in the seventies. I wish I've been around in the 2000 to be a part of, you know, the open source movement or be part of the well, you the good news is, like, you're not too late. You there's, but the the that's the good news.
• 1:20:57
  The bad news is it's hard work. But the I I think open source firmware is gonna be a huge trend, for all of us in the next 10 20 years. And if you are a younger technologist, this is a great thing. There's a lot of work to be done. And I think, you know, Chris, you and Philip have done a terrific job of setting up a foundation that's really welcoming to those new folks.
• 1:21:22
  And, there's a lot of ways people get involved. So they and I thought I love the fact that there were a lot of younger folks, in addition to a lot of, you know again, I would say gray beards, but I can't actually grow a facial hair apparently. But the, we had a lot of older technologists, certainly, more seasoned technologists, but a lot of younger ones too, which I think is is great. And so there's a lot of ways to get involved. This is a very welcoming community.
• 1:21:48
  I think I actually like the fact that that this is I I think part of the reason it's welcoming is that, there's no single dominant technology here. There is people are working on, you know, on c, on go, on rust, on, like, you know, there's Python kinda dripping around. You got, like, a lot of thing different languages, different systems. There's not a single dominant operating system here, and I just think that makes it way more open, I feel. I feel that it's not, there there's not a lot of cargo culting here.
• 1:22:19
  People are really very pragmatic and very and very welcoming. So a lot to get involved with. And, Philip, Christian, thank you for your hard work. I know this has probably been thankless has felt thankless at various times, but, you and 9 elements have really served a, an essential and catalytic role here. So, thank you.
• 1:22:40
  Speaker 3: Yeah. Thanks. Thanks for having us. Right? And thanks for
• 1:22:43
  Speaker 5: Thanks for having us.
• 1:22:44
  Speaker 3: Spreading spreading the word, you know, and supporting us. And, yeah, especially also Oxide that, that you, you know, try to do, what you can do there. So that it's, it's great to, to work with people who share the same mindset and have the same goals. Right? And when then we can all put it off in the end, I think.
• 1:23:01
  Speaker 5: It
• 1:23:01
  Speaker 3: will be nice.
• 1:23:03
  Speaker 2: I think we can too. And I think we've we've already made great progress. So, thank you very much. And and then, Adam, thank you for you for fighting the good fight at the
• 1:23:10
  Speaker 3: with the yes.
• 1:23:10
  Speaker 2: At Steve Andrews.
• 1:23:11
  Speaker 3: Oh, yeah.
• 1:23:12
  Speaker 1: Making friends.
• 1:23:12
  Speaker 2: Yeah. Absolutely. Exactly. They're very good. Very good.
• 1:23:16
  And, Erwin, obviously, thank you for for helping to to to put I mean, everyone's got a role to play, in in pushing the industry forward here. And we all know this is the right thing to go do. So, I think we're working together. I think we can get there. And and, Christian, love the optimism and love the foundation, and thank you very much for joining us.
• 1:23:34
  And thank you everyone for joining us for what is for those of you European joining us in the the evening there, in the the morning here at the Pacific. Sorry, Australia. I know it's the middle of the night for you, but, you know, it's gotta be the middle of the night somewhere. So, the Perth is not here, I don't think. But so
• 1:23:52
  Speaker 4: It's good you are making it time compatible with Europe. Thank you.
• 1:23:55
  Speaker 2: Well, we did at this point. Yeah. So I I I I'm gonna make no promise. I think we we would like to make it, I think we're probably gonna and, Adam, you and I can put our heads together and and, we'll probably move it around a little bit. But we wanna be at least have some of these that are talking about with Europe because I know it's been heard that a lot.
• 1:24:13
  People wanted to join, but, obviously, it's an awesome night. So, it was good to good to to be able to do this this time, and I think we'll wanna do this at least somewhat regularly. Musk growing.
• 1:24:28
  Speaker 1: The elephant in the room.
• 1:24:29
  Speaker 2: The elephant in the room. Yeah. We'll be next time on Mastodon Spaces.
• 1:24:32
  Speaker 4: That's right.
• 1:24:34
  Speaker 2: Alright. Thanks everyone. Thanks, Richard. Thanks, Paul.
• 1:24:37
  Speaker 3: Thank you.
• 1:24:38
  Speaker 4: Bye. Bye.