Latest / The MonkCast / Evidence Over Certificates: John Ellis on the Eclipse Trustable Software Framework

Transcript

Evidence Over Certificates: John Ellis on the Eclipse Trustable Software Framework · The MonkCast

• speaker-0: Hello and welcome to this Red Monk Conversation. My name is Kate Holterhoff. I'm a senior analyst at Red Monk. And with me today I have John Ellis. He's the president of CodeThink and the contributor of the Eclipse Trustable Software Framework at the Eclipse Foundation. John, thanks so much for joining me here on the MonkCast. speaker-1: Good morning and welcome. Thank you. Appreciate the time. speaker-0: So let's begin with some definitions here because I'm sure not everybody is familiar with some of these things. So what is the Eclipse Trustable Software Framework and how did the project begin? speaker-1: ⁓ first code think is the contributor, not me. I know an I Aaron Joe sort of wet, just make sure. But so the Eclipse Trustable Software Framework actually started as the trustable software framework. ⁓ and it started ⁓ CodeThink started that journey ⁓ a good number of years ago, about 2016. Bosch asked the question, could open source be used to deliver safety critical contacts? ⁓ we ⁓ being a Bosch partner, we looked to in look into that question and ⁓ we came quickly to the determination that it could. But it wouldn't work commercially because of the just the ⁓ opaque just the contribu Open source is open, clo ⁓ the safety critical is closed. There's contracts, there's NDAs, there's confidentiality. So there's a a mismatch in the business model. But in doing that work, we actually came upon a deeper question, which is well, why do we even trust software? And question actually prompted us to look for getting the safety certificate, why do we trust software? ⁓ and as professionals, we had a vast amount of examples. was suggested maybe we shouldn't. ⁓ and thus was born the project and a whole bunch of other sort of steps behind that. ⁓ but it then resulted in the the trustable software framework, which we got to a point where like, this is this is gonna something that's gonna work. And then we realized it needed to be open and so then we contributed it to the CLUPS group, ⁓ specifically the SDV working group. speaker-0: Okay. That's a that's a good pithy history here. I like that. ⁓ so you've been doing a lot of talking about this. ⁓ I I've ⁓ been fortunate enough to watch a a couple of your presentations. So I I feel like I have my bearings ⁓ at least a little bit here. And so I'm hoping that we can dig into some of the the things that you you have said already about this ⁓ framework. So ⁓ just to begin with, I was really struck by your ⁓ metaphor about Bridge building and and how that relates to trust here. Because I feel like trust is kind of one of those ambiguous terms, like, yeah, we want trust. Okay. But like what does it mean specifically ⁓ to have trust ⁓ in software? speaker-1: yeah, so it it is it's a great it's a great metaphor and it does often you know poke the audience in a in a very specific way. Right. The metaphor very specifically is is it in ancient times, right, the anecdote is is that architects and bridge builders were forced to sleep under the bridge, right, to prove to society that what they built was was in essence safer usable, right? ⁓ and if they were if they refused to do it, then you know society should take a step back and question like what actually happened. ⁓ So the metaphor says exactly the same for software today, ⁓ especially as we're talking about the software in such embedded and regulated markets, right? Aviation being one, but autonomous vehicles, we are we are asking society to allow for, you know, basically in essence, technology companies and engineers sitting at a desk to make decisions that impact society at large. ⁓ and so, you know, we've actually provoked the quick engineer would you be willing to put your loved ones on the road, laying horizontally in front of an oncoming vehicle? ⁓ and again, it it's it's it's intended to provoke deeper thinking about like what does it really mean to trust software? Are we s are we comfortable that a safety cert is sufficient? Are we are we comfortable that commercial contracts are sufficient? Like what does it really mean to to have trust? ⁓ and what part of the trustable software framework and ⁓ sort of the journey, right, in terms of getting there is those at those those individual points. Are not sufficient. Like I can't have just you saying, here's a cert, trust me on this date. Like I it's a continued relationship between me, me society specifically, and and how this works. So we hope that in framing that metaphor, right, we get people to really pause and say, like, ⁓ wow, like, yeah, this is a really big deal. And a contract or a license or a cert or whatever else we currently use today is is really insufficient. ⁓ in allowing for us to actually have that conversation. speaker-0: Okay. That helps. And I love thinking about software as a bridge because I feel like you could take that image anywhere. ⁓ but one of the places that I I I f find interesting that that you take it i is to this conversation around integrations. So ⁓ you have described integrations as this moment of truth. ⁓ so it's like this button push where everyone they're just hoping that it works, right? And these unknown dependencies and non reproducible binaries, they suddenly surface and and everything is kumbaya. So Talk to me about integration and how it factors into your thinking around ⁓ trustable software. speaker-1: yeah. I I just as with everything I like to do, there's ⁓ there's analogs and metaphors and and the reason for integration, in in part one of the conversations that I had in at least at least one of the talks that I think you've seen is we talk about sort of like the you know the idea of a factory or the idea of where integration occurs. We can kind of a factory metaphor. and at least for most of of the audiences working in equipment like if it's automotive or medical or others, having very similar models, right? They They have an OEM that has a whole concept of a factory and building of a physical product. And it is incredibly interesting that when you talk about the factory, down to the nth detail, they have exactly what they're expecting, when they're expecting, how to expect it, how to integrate it. Right? The factory is where there are no surprises. Like there are no surprises. You do not, in fact, everything is about making sure the factory works. Unfortunately, that mindset has hasn't actually come over to software. And so because it's not in software, that's what we mean by the integration. And so the one talk that you saw where we had the picture is like all these things, very similar to a physical factory. I've got a package coming from a vendor or a silicon package with software coming. I I have deliverables in essence coming to me, but they're not governed as they would be governed in the physical world. And so because of that, this integration process is where surprises come. And again, that's where software becomes the reality of the integration, becomes the moment in truth when like, ⁓ for the first time ever, I'm now looking at all the dependencies and I'm seeing all the different things. And that's where things and surprises happen to come from. So again, we start from the basis of that physical world model and we're like, there should be no reason why it's not in the software world. We call that and that's where the trustable software factor ⁓ framework basically embeds and embodies that in some of the tenets. speaker-0: Interesting. And can you give any history to that? Like why is it that it's this moment of ⁓ integration that we have tolerated that this ⁓ please work moment is is is gonna be so important. Where does this come from? speaker-1: You know, it's it's ⁓ I think it's it's a couple different things. Number one is is we have historically f certainly from purchasing, we have brought people over from the physical world who would you would think would have that, but when they came over to the digital world, they they didn't bring the the model with them. So there's not the same level of of diligence, I would argue. ⁓ number two is is is we have some you know, oftentimes it just worked. Or or there's been a disconnect to work, meaning, ⁓ it it it was able to be shipped, and then sort of afterwards the recalls or the other points and the and the sort of you know failure thing. We just haven't applied to the software engineering discipline ⁓ the same level of rigor that we apply to the physical world of of factory and and and physical engineering. many of my colleagues at CodeThink would argue software engineering is y not yet really a discipline. We we haven't gotten there yet because we we don't have all of the other things that you would expect to see in some of that engineering discipline in terms of the rigor, in terms of the post analysis, in terms of the society understanding something failed, we understand it, we make it, we remediate it, we fix it, and we go back. and so i I think it's a multipart problem, right? There's a lot of different stake for the reasons to the why. ⁓ and I think what we're finally seeing is is is at least in some of our our example products, the rigor is now drawing into question because we can see the recalls or the failures. I mean, I think you saw the one slide I talked about, there's trillions of dollars being accounted for economic loss. Like like we have the data point. And so I think it's and and again, the Eclipse Foundation and the SDV working group specifically is trying to address that. Like they're acknowledging that. And the Trustful Software is sort of a step down that path to try and Be the place where integration is contemplated and understood, right? One of the tenants, understand what you're getting, understanding what's coming to you, right? Is tenant number one. Understand what you're building. ⁓ and that is kind of that first step. speaker-0: Okay. And as a former QA engineer, one of the things that I was really attracted to in in this argu bit is that the the beginning of the SDLC, the beginning of this production or the factory metaphor, it gets more attention than the end. ⁓ and that's where things kind of fall off the rails. ⁓ do you feel like the SDLC is is equipped to to deal with these challenges or like how is h is your how are your ideas about this framework really challenging ⁓ the way that ⁓ software engineers have functioned up until now? speaker-1: so so the trustful software framework is well, Bosch asked the first question, and then there was a whole generation of ⁓ there was a period of time where there was an open like in good software and in good open source form, ⁓ Code think provoked the question publicly and invited a public discussion about like how can we trust software? How do you get to trust software? Which those those six tenets kind of come as the sort of summary of the 12 to 14 months, whatever the duration of the time was behind it. And it and and so doing this, right, there was a lot of questioning about like what does it mean to actually engineer software and how do we do it? And and and what was there an what there's an acknowledgement of like software is accessible more so than so much in the hardware world. Like in the hardware world, you need certain things that are not necessarily easily combiable. And yet we can do in software. Like I mean, you could write a macro in Excel and suddenly you're doing software, right? How many businesses have built themselves around macros built by people sitting at a desk and they don't even understand where that comes from, right? I mean, you're smiling, but like that's that's that's how easy this is. Yeah. So as we as we talk about what does it really mean, I think there's we're differentiating, I mean, more broadly, Cutotink would argue, and I think the Eclipse Foundation would argue, software engineering writ large, right, needs needs to be thought of. But when we talk about the the the concept of safety critical or regulated industries, ⁓ this is where we're really focused on and and the idea that software under contract isn't you know, putting it under a confidentiality agreement, putting it under a contract, put making it hidden, making it less accessible, right? It that's not the right approach. Like exactly a bridge metaphor. Like if somebody built the bridge and said, you can't study it, no third-party structural engineer can come in and look at this and do weight measures and analysis, like you can't do that. ⁓ I think we'd all as a society balk at that. So why do we accept the alternative, like the opposite side of that on this on the software side? So when you say, like, is the software development lifecycle or the way engineers are used to building software ready for this or whatnot, there are changes, right? There are there are understandings, ⁓ but it comes down to being really engineering and and we really do want people to think deeply about what they're building because again, society is impacted by anything and everything that these product companies release. speaker-0: Yeah, that's huge. ⁓ okay. And so just thinking about this in terms of brass tax, organizations they're already using, like S Bombs, CI Pipelines, testing requirements, management. and so my the question that I have then is what is the ETSF connect or ad that these practices aren't already providing? speaker-1: I well, I mean it in and that's a great it's a great observation, right? We do have a lot of the different things and and we do have ⁓ right? And they do work to whatever limit they are, like the S bombs and C S C D testing and others. And I think what what I was highlighting when I talk about this publicly is they all exist, but in some cases they all exist in either silos or fragmentation. Like we look for an S bomb by the S bomb point. We don't think of S bomb as an every everyday evidence that's coming in the course of a pipeline, right? Again, going back to that mainstream idea of a physical factory and like walking through a factory and what gets delivered and managed and how do you manage a factory and looking at like throughput and you know failure rates and others, right? ⁓ there are things to be learned in that way. And so what the TSF, you know, the Eclipse Trustable Software Framework brings is is a way by which to contemplate that in its entirety, reason about it, and then reason about it in its entirety, not just an S bomb and the S bomb is SPDX 2.x or 3.what Or even cyclone DX, right? Not just SPDX, it could be cyclone. We allow for the reasoning. Because what we're trying to get to at the end of the day is we don't think you can make a blanket statement, this is X. It's X for a period of time T against some understanding Y of the world. So at so at the end of the day, there has to be a level of confidence. Like we're confident that this does what we say it does. against these credentials or against these things and here's why and here's the evidence. And so ETSF, as you're referring to it, right? The Eclipse Trustful Software Framework is intended to be that holistic way by which you can come to make the confidence statement, provide the evidence, and then allow a third party to reason independently and be able to say, yeah, I'm I'm in agreement or no, I have a different view of confidence. Or yeah, I'm good. I'm good and I'm let's go release this. speaker-0: Okay. Yeah. And I yeah, I'm I'm interested in in how that ties into the way that developers often have this fear of touching software, right? So w in most regulated industries, they're treating changing chip software as this terrifying thing, right? It adds this this level of uncertainty to something that they already know works. But my sense is that that ⁓ the ETSF allows you to be able to make changes. ⁓ and reason about whether or not they're actually gonna work in a better way. And so that that can help organizations feel better about ⁓ making these changes. I is that a fair way of framing it? And how does that even tie into like where the regulators sit into this, like how how they're ⁓ changing their posture in in relation to the these ⁓ deployments. speaker-1: so I think I mean there's a couple different maybe threads to pull on on there. I think the and probably the biggest one is the idea that this product, whatever it is, has to be has to ha has to continue to operate in some level of of your statements for fifteen to twenty years. one of the things that I like to when we talk to customers and and clients when I talk publicly is software. at the base level is just a ⁓ a mathematical or mental model of how I see the world at a point in time. ⁓ and as I as an engineer or I as a company, you know, spend time, what I see the world today may or may not be the way I see the world in a year or two years or three years. I will have learned more about the world. I will have better understood the world. And so that mental model has something called decay. I know we don't talk about it, but just again like the physical world. We've spent so much time trying to figure you know figure out exactly where the tensile strength of steel fail fails when it's bent a certain way. We understand the incredible, you know, physics and chemistry of everything that we build so that we have an understanding of tolerances and and we can think about them and we have an understanding of that world. Very similarly, we have the same thing in software. We have created a model about what the world looks like or what we think the world looks like at that point in time. The model has to change because our understanding the world has to change. So we there's a construct that we we have to contemplate changing our model over time. With that, against the 15 to 20 year viewpoint, somehow, some way we have to accommodate the idea that we're gonna have to make changes. So when you ask, like, well, how does that work or how do we get to it? We would just offer and argue if you fully understand what you've built, if you fully understand where it comes from, if you fully appreciate it, then you should be comfortable. Making changes. You may choose not to make changes, but that's a that's a that's a decision as an engineer that you're making with all the facts. Not because, ⁓ I can't change it because I I don't know what's gonna happen or I'm afraid of what's gonna happen, right? ⁓ so our tenant three is the ability to make changes and understand, right? And be comfortable to make those changes. Again, when I say that, please don't take from we were talking making changes willy-nilly. It's not like some guy today is just gonna wake up and tomorrow make this is I'm understanding that I can make changes. I do the deep engineering, I understand it, and I've come to an agreement as a community, whatever that community definition is, that yes, this is an appropriate change, it will work, we've studied it, we understand it, and we've made the change. And we're not afraid to make the change. Moreover, we're f we're willing to study the world's impact as the change propagates and again understand, etcetera. ⁓ I always like to point out to people, right, if you look at any of the Voyagers or any of the stuff that NASA and JPL have done, like where they have stuff built from the seventies or eighties out, and they're making full software changes years later, it just doesn't happen. Like the engineers come and they deeply think about it, that's the kind of stuff we're talking about, and that's the mindset that we're trying to get into. Do not be afraid to make changes. That doesn't mean you do make changes, just do not be afraid to make changes because of X. That's what we're trying to get across to people. speaker-0: Okay. And my sense is that versioning and git are extremely important to how this will look in practice. Can you talk more about that? speaker-1: I it's it's it's versioning and git, but I mean it's gonna be down to business practices as a company, writ large, right? The ability to actually manage ⁓ again, this comes down to if I go back to physical factories, right? It is incredible the amount of documentation and detail we have about anything that we received, how we manage it, how we manufacture it, how it comes to like we we have all that. That same level of engineering understanding is what we're looking for as our sort of North Star goal, as we talked about software. So yes, it's Git, it's versioning control. It's not just versioning control, it's like understanding edits. Did I allow, you know, Kate to make edits independent to did Kate's edits get reviewed? Were Kate's edits, did they touch upon anything safety critical? If they did, what do we do about that? What does that look like? I mean, it's not just versioning, it's the entire of how do I manage this as a thing and how do I understand the, you know, the the management of its changes, its evolution, its life? And then again, am I still able to stay that with X confidence I'm good that this thing will will do what it's supposed to do and is is gonna behave in the way it's supposed to behave and I've and I have a level of confidence that it won't harm people or you know or goods. speaker-0: So I I f I feel like to some folks might listen to this and be worried that this is gonna result in higher headcount, less lean, it'll ⁓ decrease velocity. They they might be worried about the bottlenecks that this might create. ⁓ how do you address that fear? speaker-1: ⁓ I mean I I it could could it? I mean maybe, maybe. ⁓ but i I think that's a bubble. Like it's at the beginning part. If you if you know in our experience with customers, once they've gotten a factory model, once they've adopted the idea that this factory is no different than the physical factory that they run to build the product, it has the same governance, the same discipline, the same things. They get that and once they have the necessary governance in place. Then you can start optimizing throughput for builds, others, et cetera. ⁓ it's only once you have that in place can you then have the ability to start figuring out how to optimize where necessary for you, right? ⁓ at the end of the day, the Eclipse Trustable Software Framework leaves a lot of latitude to to what does it mean to be trustable? Right? It it's really up to the company in question to start saying, like, here's my evidence for the claim I've made, and here's why I think this is good. Like now others can look at it. But once you have the instrumentation and the machinery in place, now you can start thinking about like, well, how do I optimize? Do I have development in a specific region? Now how do I start changing things? How do I start optimizing my the contracts that I'm entering into? How do I optimize again? One of the things that we we we look deeply on is how do I optimize, you know, again software sovereignty? Do I own my own software? Or am I or am I just a massive integrator of scale and I'm just getting all the software from third parties and shipping it and putting a label on it, right? These are the questions that come up naturally, commercially. Once you've answered them and you have the instrumentation, our our our experiences, you can get speed, but you have to start from the concept of this structure. You just can't you just can't try for speed for speed's sake. Mm-hmm. Yeah. speaker-0: That's reasonable. I I want to return to the bridge metaphor because I'm thinking through the sort of positive things that this enables versus I guess the negative in the absence. So it's very clear with a bridge. You either sleep under the bridge or you don't. Right. With software, it's how are you testing this negative? Like the how are you determining an absence of bad behavior here? it it just seems like ⁓ it's it's hard to check Or the absence of a happy path, right? So, like, how do you actually generate continuous evidence for this negative, this untrustworthiness? speaker-1: Yeah, so it's it's it's a great question when we we get often and and for for us specifically we we'll talk to you a little bit about so we I wanna be very clear, TSF does have as as part of like some of the expectations. So, you know, tennis four and five, the expectations and the evidence. ⁓ We actually do employ a risk analysis. Like it's not like just a bunch of people sitting in a room. Like there's an actual formal risk analysis. And in the case of of the trustable software framework, what we actually came through was it's predicated on STPA, systemic theoretic process analysis, which came out of MIT, Nancy Levinson and John Thomas. ⁓ and was originally designed for or originally envisioned for ⁓ aviation and the idea that you don't get safety by plugging two safe things together. You have to look at things in a in a in a very in a in a much higher level. is what the the premise of the of the approach was. ⁓ through working with MIT and and and this, you know, code thing came to understand that it wasn't necessarily tuned exactly for software. And so we created something called Rafia, also open source, but the risk analysis fault ⁓ induction automation. So the risk analysis is all the STPA. You do your loss scenarios, you understand and try and for the context that you're talking about, you try and identify what can be done or what can't be done, i.e. the loss scenarios. Our insight in terms of the fault induction is that well with software. So we can actually either write, change, modify to induce the fault that we think shouldn't happen in the world and see how the system performs and validate that the system holds true in the case that that negative actually showed up. So we actually do that. And in so doing it, we've uncovered problems. For any package that we've looked at, where again there are corner cases or others where we've made changes and upstreamed them. Once we understand it, we can then automate the test to validate did di is does that does that still hold true over time? And so again, the SDPA is where we do all the analysis and come up with the ideas. The fault induction is where we actually take advantage of the fact that it is software and we can we can force the changes that we think shouldn't happen to happen, and then we can see how the system performs. Then we automate it and then we just run that every time. And as ⁓ as we like to say, if it squeals, then we know something's wrong and we have to stop, we have to go look at it and understand. But again, if it doesn't squeal, we're we're able to continually keep that. So yes, that's where the testing and the body of testing increases over time. Every single time we find where there might be a loss and we've forced it to happen and proved that it shouldn't, or we've proved how the system behaves. speaker-0: So I and what interests me about that is there's there's such a I guess a binary between this passfail mindset that you think of with like regulators and assessors versus this statistically driven like, you know, we are confident to X percent that this complex software ⁓ i is gonna be trustworthy. ⁓ how do you get regulators and assessors on board? with this understanding that's a little bit more nuanced and a little more ⁓ effective in the real world. speaker-1: ⁓ similarly to how like so you're right, historically their view of the world has been in the physical world, it's all about testing. It's all about statistical analysis and failures and understanding failure rates, which is why you get the hours of use or the exceptional sort of testing rigor that takes you to the negative exponent of of of time, right? And so part of our journey has been to raise the awareness that software should be treated no differently. Like the st Physical confidence is something that's there. And in fact, code think's not ⁓ you know unique in this. There's a lot of embedded developers who are putting forward that the idea of of claim should be Bayesian based, not just analog true, false based, because it be true for if it's only true for eighty five percent, but you're in an analog, then it's false. And and that and then the engineer's like, well, but wait a minute, like eighty five percent like so so this is the exact same world that we live in in the physical world. You're right, it's a change. It that that is a mindset change. And it's a mindset change that, especially in the embedded world, we think is is overdue simply for the fact that the complexity of the software we're talking about. Even if the software was written by hand, the complexity of what we're talking about in terms of the world is far greater than I think the idea of just binary true-false allows for. And so we know how to do it, statistical process controls in in the real world. We understand how to handle it and accommodate it. And it turns out even some of the standards bodies are beginning to acknowledge in software that there might be a need for some statistical understanding. Not there yet, but there's at least a growing conversation broadly that's just not just about the TSF. And not even because we necessarily said it. We're just amplifying that message. speaker-0: Well, I mean, that certainly seems to resonate with what we're hearing in our AI era, where there's this whole new breed of chained vulnerabilities and we're sort of in this new era about what the C V E landscape looks like. So sort of applying that to what you're talking about here, ⁓ we're seeing the need for changing our priors, right? That we we are all adapting to make sure that we're we're creating the best software possible in this new era. And a lot of it's attributable to AI, but a lot of it's been here forever, right? This is this is ⁓ a historical situation that is perhaps shifting, but isn't going anywhere. and as ⁓ maybe part of that, I I wanna talk about the black box nature of certifications historically. So traditional certifications, they're kind of opaque for folks on the outside. So I'm interested in how you're grappling With this sense that teams are unsure what's actually happening behind the scenes with the NDAs, the the fact that they're just not entirely sure what what these assessors are looking at. ⁓ can you speak to that at all about like trying to ⁓ break through that veil? speaker-1: ⁓ yeah, sure. So as just background, right. So Code Think has gone through a number of different certification processes. We've actually achieved an ACLD against two sixty two six two for some of our building construction. ⁓ we've actually gotten, you know, the TSF itself certified against sixty one five ⁓ eight and we've gotten our version of c of control, which is Linux, right, to a baseline assessment through sixty one five. So we're we're familiar with the process. And I think context to to talk about then sort of like why and how do we take our position in in terms of trying to, as you said, break through. I think the first the first thing to acknowledge is the the idea of a safety certificate, I think itself is a misnomer. There is a certificate from an organization, AK Exeter or T U V or UL, against some standard, but if you actually read the certificate, at no point is the word safe present. What is present are words like, you know, we have seen, you know, ample evidence of the company in question following process for this product in question, right? We we've seen and therefore they have received this certificate, right? We don't see the idea of the third party intermediary saying, we the company hereby attest that this product is safe based on XYZ. So and this is all with respect to software, right? I'm talking about ⁓ and so we find that super interesting. ⁓ and when we've pushed on it, it's like, yeah, because at the end of the day, right, they're not making a claim of safety. What they're making a claim of is a process was followed, and then it's up to the industry to recognize whether or not they accept that process as ergo being something that's safe. Why that's interesting is because that ties back to your earlier question of like statistical process control. The idea of it's if it's if you build it by hand and it's binary and all these things are controlled and it's contained and it never changes. We can make some statement about something and so, yeah, if you follow the process, we'll equate that to save. In the complex world we live in, we know that that is not really the case. And I'm offer as evidence is, you know, as I said to everyone if if if it were the case that it was right, then in theory we shouldn't see what we're seeing in the marketplace, which is the idea of a safety recall, a recall from the United States government, is a safety issue. And most and increasingly these recalls are software related. So if in fact it were working, we shouldn't see an increase in recalls. We should in fact see a decrease in recalls because we're actually doing the right things. So what we offer as evidence is there is something about the complexity nature of software. There's something about the integration. There's something about everything that we're talking about that demands us as engineers to really deeply study this and understand how it is that we construct, build, and deploy this software. ergo why we go back to ⁓ traditional methods I think are gonna they're they're short in terms of being able to actually have somebody sleep on that road while you know a multi thousand pound vehicle at eighty miles an hour is traveling down the lane and we expect it to stop or avoid the person on the road. So ⁓ that's where we would put our our our our our thought process at. speaker-0: Do you suspect that this will have some upstream ⁓ ramifications, ⁓ this being the the framework and trustable software? do do you see that the assessors are then going to need to shift their way of ⁓ addressing the industry? speaker-1: ⁓ I I don't think it's just assessors. I think it's assessors, regulators, society. I mean, yeah, yeah. I think all of us all of us need to shift how how we manage, how we deal with, ⁓ and how we understand and deploy, right, this this level of software. ⁓ and again, the data points in the marketplace, whether it's the increase in s in safety recalls or if it's just the losses economically, right, suggest that what we're doing isn't sufficient. Uh-huh. And again, we would offer the ⁓ Eclipse Trustful Software Framework in in in support of a change. We're not suggesting that it is the only change, but we are suggesting it is it is a piece of a change. ⁓ and again, offering to stand by it as part of like how do you come to be able to say the word? I'm I'm confident or I can trust that what I'm doing, you know, won't harm. speaker-0: And with so many stakeholders involved, I suspect many of our listeners are gonna be ⁓ you know, are gonna have their skeptics hat on of like, is this a foxes guarding the hen house situation? Like who's really making sure that this is ⁓ something that we need to ⁓ all be following here? So I just wanna make sure that we're addressing those those questions. ⁓ speaker-1: But okay, so to just add the plus one color to what you just said, like our suggesting for a moment that you know that that the ETSF by itself is the is the tool, like there is still somebody or some entity that has to say I'm confident. But what we what we are suggesting is this framework then allows for you, the recipient of my statement of confidence, to actually pendently look at that and look and understand it and study it and be able to come to your own conclusions. Was the evidence provided sufficient or is it sufficient? Is it sufficient for the statement of confidence that they're giving us? And if it's not, to challenge them. Like, I'm sorry, but that evidence is insufficient for your strong confidence level. You're missing the following. It just it allows finally for the ability, right, for that structural engineer on the outside looking at the bridge to say, independent of you sleeping underneath it, I have the issues with how it was built and how it's supposed to work and how society is supposed to accept this into the lexicon of things that society depend on. speaker-0: All very reasonable expectations here. I I I like that. As someone who could potentially drive over a bridge with my kids in the backseat, I do want that. ⁓ yeah, so this is this is all for the good here. so thinking of regulators and folks who who might be looking into the the trust issue, ⁓ let's talk about the CRA. ⁓ so we at Red Monk have spoken with the Eclipse folks ⁓ at some length about the importance of this. I feel like ⁓ you folks have really taken a leadership ⁓ position in ⁓ trying to express to to ⁓ the industry and why this is so important and and so the CRA being the EU Cyber Resilience Act. and so the consistent signal that we are hearing is that businesses are unprepared. And I I'm interested in your read on the CRA readiness issue ⁓ across Europe today. What what is driving that awareness gap? speaker-1: well, I mean so first off, as you can tell by my accent, I'm not European and I don't I don't live in Europe. ⁓ anything I'm gonna share with you is really a ⁓ a recital of what comes from my time with Eclipse. so ⁓ I think broadly, regardless of Europe and all those disclaimers I just made, I it is my ⁓ as I say in most of my conversations when I talk about the fact that so much of our world is governed by software. It still to this day shocks me at to just the gross misunderstanding or lack of understanding that sit at board levels, at executive teams for what software really is. You know, as as it's super interesting when I when I get asked questions and I always point to this, right? If you want to be a board member or if you want to be someone CEO or an executive or a COO, you better understand the factory. You better understand supply chain and finances. Nowhere is it, you better understand software. Or you better understand how software is built, even if their product is so dependent on it. So when we talk about the CRA gap, in part it's because these executives don't fully appreciate just how much software is in or part of what they do, number one. I think it it it's also the fact that I don't know how much in the US, and I'll just being through the lens of the US, did we talk about the CRA? Like I can remember EDPR was coming up, it was a brand new novel concept. But guess what? Lots of law departments were all over our our stuff raising this awareness. When I was back, you know, working in a in a in a a for-profit publicly traded company, the lawyers were all over us like this is a thing and we're gonna have to comply with it. And I don't know that that's actually happening at the same level because I don't know that there's an awareness of that necessary requirement. And again, unfortunately, unless the attorneys are telling you you have to do it. Sometimes that gap exists. So I think there's an awareness at the level of the executives. I think there's also awareness broadly. That said, once you go through it and you start pointing these things out, I've been in meetings or I've been in sessions where, you know, we'll go through the the key points. So when we first start, how many believe it covers them? You know, no hands up. By the end of the session, how many believe it covers you? And all the hands are up. And you're like, wow, okay. So now what? It's ⁓ it's a journey. And I think it's gonna be a journey just like the GDPR. I mean, to this day we're still kind of fighting through key points of the GDPR. Like what's really the rights in all this? It's more settled than it was, but that's almost fifteen years, right? Roughly ten, ten, twelve, thirteen years. I mean, it's been a long time. I suspect the same journey will happen for the CRS. speaker-0: Man. So my takeaway from what you just said is that yeah, the awareness is still lacking, but also that this is not gonna happen anytime soon. I mean, what I keep hearing is like, it's coming, it's coming, it's coming. But even when it's here, we're talking years in the future, we're still gonna be grappling with what that actually means. ⁓ my goodness. speaker-1: I think so because at the end of the day, there's no as as week point out, right? There is the d I think the what is really well defined. We can argue how it's defined, like who's a yeah who's a who's a who's a a vendor and who's a who's materially responsible. We can argue the definitions, right? That's the what. And it's it's obviously codified and it's been made law and we can debate that, but it's it's codified. So I think we could argue that the what is defined well. What's not defined, or at least not defined well, is how do you actually show when up when asked compliance or that you are in in in in in in line with, right? Whatever the word right is. I don't know if it's a compli well w how do you do that? Like and and sir, there are things now that are that's where I think gaps are. And that's kind of why we are positioning the ETSF as the way to show the how. Like we are arguing that the evidence required to support the know what you're building tenant one, being able to build reproducibly tenant two, being able to make changes and and understand them, right? Tenants one two, the evidence just behind those would be strongly sufficient, we argue, for what the commission's looking for when they've laid out the CRA what. Again, that's still to be debated and it still has to be tested. ⁓ but that's what we think is missing in the CRA is the how. And therefore, until there's a real definition of how I think it's gonna be time until people understand and settle on, okay, we've all agreed, this is a how, they've agreed and and we're now showing you the how on a regular basis or whatever the requirement is. speaker-0: Mm-hmm. Okay. I'm glad we did this check-in here because I think it was last fall that we we published our episode on ⁓ the CRA ⁓ with ⁓ Mike Malikovich. So this is this is all good where we're checking in. ⁓ okay. ⁓ Yeah. ⁓ I wanna return to a subject I alluded to briefly, which is AI. ⁓ so a growing share of code is written by AI now, unquestionably. and that means that the provenance issue is becoming more murky, right? I am interested in how you factor into that. Like you've got this evidence based model. ⁓ it seems like it's becoming more important now. And I guess I'm just I I would like to have your sense of like can a framework accommodate the code whose origins are so difficult to trace? speaker-1: ⁓ so to start, I would argue that like the framework just gives you a framework for making sure that you've answered the question on providence. Like the definite acceptability of providence is ultimately the company in question or the entity in question who's making the confidence claim. So, you know, and that's where again the ability to reason about this. So if I show up to you, Kate, and say, Here's my confidence, you look at Providence like you didn't tell me where the you didn't tell me where that stuff came from. Like you make that claim. That's sure. Only do that. In light of the framework giving you the tools and the in the quote unquote framework by which to analyze, study, and challenge me, right? But we don't actually make a claim for like what is pro like provenance is a framework. You still have to do the thing necessary for a company to but we give you all the tools, we give you guidelines, and we show you this. We what's interesting for us at CodeThink is we are using this to take our Linux distribution through the actual same process that we're expecting anyone else to take an open source. Or even a closed source project through, right? We're not we're not sitting here pontificating on high. We're actually trying to do the actual work necessary. And so as I mentioned earlier, like we actually have a baseline safety assessment that says, yes, yeah, you you can actually use this Linux in an engineered system up to SIL 3 ACL D. Like it said that. It's a properly engineered system, right? It's not just generic. So when you ask me about the provenance, basically our answer for AI is. absent any ethical concerns that we might have or anyone might have, it's very easy to say, is did did you check to that with did you do you have a claim or a requirement that who gave you this, who is delivering this to you, attested or made a claim for is there AI? That's like an early step. And if there is, you make the decision, are you comfortable with that? Like you have to make the at the end of the day, the company still has an obligation as engineers. Are you comfortable that AI was used? ⁓ and if you are and you're confident, then you give me your evidence for that confidence level. And and we can now talk about that. ⁓ in terms of us personally, if we find AI involved, like we what we guide our clients and customers to is like, listen, the framework allows for you to provide a lower score or a lower trust or a lower factor. So if we say like we don't know where this provenance is, it's AI, or for example, a a vendor only gives me a binary, like they will not. Ever give me the actual source, only give me a binary. We would also offer and argue that that's lower trust. You you can't again, I can't inspect it. I can only make a trust claim that commercial rules allow me to make. AKA, I asked for this, they gave me this, they say it's a SHA, I can verify the SHA, but that's it. Is that enough trust to put the vehicle on the road? I I don't know. That's up to you as a company, right? But the framework gives you finally, we believe, the ability to ask the questions. challenge the responses and then come to independent conclusions and then debate it societally. ⁓ so we don't make a claim right or wrong about whether AI should or shouldn't be used. We just say that the framework allows for you to determine if it's used, how do you assess its ability from a confidence standpoint in what you're shipping. speaker-0: Fair enough. ⁓ okay. Well, as we think about winding down our conversation here, I do want to ask you about the roadmap because I suspect that the ETSF is not a static thing that, you know, this is this is shifting, this is dynamic. So talk to me about what the project is is contemplating next. speaker-1: well, I mean there's a there's a couple of different elements. one is we b you know, CodeThink as as one of the contributors and one of the one of the users of of the framework demonstrated the ability to take a third party, right, or to take a standard, a K A in this case sixty one five ⁓ eight and map it TSF and then show how we could take a thing against the TSF and be able to satisfy sixty one five ⁓ eight. We would want and look to see any number of those kinds of mappings happening. We believe CRA is one that's that's rich and we're working through. we're also working, you know, internally with a number of the different stakeholders from different companies, right, to take the Eclipse development process and to also make it part of like a TSF. So generally speaking, anyone can say, I want to be assessed against TSF, and they can ask for what the assessment should be focused on. 61508, Eclipse Trustable Framework, 62 26262, ⁓ 31 name your favorite standard, right? And the mappings exist and suddenly you're able to prove this. And so what that does is it in essence simplifies for development engineers the ability to say, like, what am I needing to worry about? Because unfortunately, one of the other outcomes of our current approach is all the standards are proprietary. And you need a license. And the licenses, generally speaking, are to a single person. So UK can get a 62 or 26, 262 license, but it's only to you. You can't share it with your colleagues Or your friends or anyone else, regardless of its expensiveness, that doesn't that doesn't foster the kinds of engineering conversation that we think needs to happen in a world of such criticality of the software in question. So being able to say, yes, you're reasoning at the TSF level, you're debating at the TSF level, you're debating in open, you're debating at a societal level, right? Is where we're that's one of the big roadmap items to try and get to. ⁓ and we started, you know. Code Think was able to contribute. We did under an early release, our sixty one five ⁓ eight mapping that got us to where we were with our assessments. one of the next ones up is gonna be the CRA. The Clips group is trying to figure out how best to do that. ⁓ and then we're open to anyone else you know coming along and saying, ⁓ I've got another standard that I wanna be able to look at. And okay, great, how do we do that? speaker-0: is a full plate. Sounds like you got your work cut out for ya. I I look forward to seeing where this goes. ⁓ okay, so ⁓ before I sign us off here, John, ⁓ where are you directing folks who want more information? Where should where should they ⁓ look? speaker-1: In short, it's the Eclipse Trustable Software Framework project, which exists inside of the Eclipse organization. speaker-0: I will include a link in the show notes here to that so that folks can follow up. ⁓ I have really enjoyed speaking with you today, John. This has been an absolute pleasure. And again, my name is Kate Holterhoff. I'm a senior analyst at Red Monk. If you enjoyed this conversation, please like, subscribe, and review the monk cast on your podcast platform of choice. If you're watching us on Red Monk's YouTube channel, please like, subscribe, and engage with us in the comments.