Latest / Autonomy Insiders / From AV Pilot to Public Service: What It Really Takes to Operate Autonomous Vehicles in Europe
Transcript
- Daniel Abreu Marques: Welcome back to Autonomy Insiders. I'm your host, Daniel, and most conversations about self-driving vehicles stop at the vehicle. The sensors, the AI the next demo. But today we go behind that. Because once autonomous services stops being a pilot and starts carrying real passengers in public transport, the hard part is not the car. It's everything around it, like safety governance, who is accountable when something goes wrong, who holds the data, and How a dozen companies coordinate to keep a service running every day. My guests run exactly that. So Holo operates autonomous vehicles in Oslo for Ruter and enables operators to run autonomous mobility services across Europe. And Stefanie Berliner leads their DACH business and Anders Toft runs operations. So Steffi and Anders, welcome to the show and so happy to have you on today. Anders Toft: Thank you for having us. Stefanie Berliner: Thank you. Yeah, thanks for having us. Great to be here. Daniel Abreu Marques: Yeah. So let's start simple, because of l a lot of people listening know the technology names, but not the operators behind the service. So Steffi, if someone has never heard of Holo, how do you explain what you do and where you sit in the autonomous mobility picture? Stefanie Berliner: So Holo is basically an operator in in in the Nordics. So we operate the autonomous driving vehicles for Ruter. We are their PTO, so to say, in on demand service in Oslo But behind that, based on this knowledge, we consult and advise other operators to to to run autonomous ⁓ vehicles safely and compliantly on the roads. So what we do actually is enable them to do the same in other European countries based on the experience and knowledge we gained in Oslo so far. Daniel Abreu Marques: And looking across your current portfolio, what kind of A V deployments are you supporting today and where do you see the industry shifting from pilots to actual passenger ⁓ operations? Stefanie Berliner: So we support every kind of operator, so to say, be it in the public transport sector, be it in private mobility services, enterprises, but also airports, so to say, and so and actually what we see is a focus in shifting from proving technology to proving operational scalability. So coming to your question is where do we see the industry shifting from pilots actually? Where I would say all the companies really have a long term vision. Right now all the projects are stopping, so to say, when they only have planned for pilot with five to twenty vehicles. So everything comes with scalability might be the long run. And we see that actually where we heard already about a vision, for example, in Hamburg, for example in St. Gallen with with postauto, and also especially for the public transport who have built their vision over the last years. Also with TPG in France, there's some project in London specifically, because they're also Everyone is gaining big, big fleets. So here I see the shift from pilots to actual passenger operations. Daniel Abreu Marques: And you also both work from from a clear point of view, so that the public conversation is looking at the wrong part of the problem. So let's get into that. So the public still fixates on vehicles, sensors and AI, like I mentioned in the intro, and you argue the hard part is operational. So make the case where does the real difficulty actually live? Anders Toft: Well, I think I can go a little bit into that because we've done operation in Oslo for seven, eight years or so. And we've seen many places in the world, the technology does work. You can put vehicles into difficult traffic scenarios and they can handle it. But there's so many things that are working behind the scenes that no one ever sees. the, let's say magic behind the scenes, so to say. ⁓ Sorry, can we redo the question? Actually. Yeah. Okay. Can you repeat the question? And then I can, I need to get into this podcast mode here. Daniel Abreu Marques: Yeah, sure. Just start again. Okay. Okay. So the the public st ⁓ I I think ⁓ I also had some some introductory words, but I just repeat the question. So the public still fixates on vehicles, the sensors and the AI. ⁓ but you argue that the hard part is operational. So can you make the case where does the real difficulty actually live? Anders Toft: Yeah. So we've seen many places that the technology does work, as you say. We have self-driving vehicles in the Middle East, in the States, in China, and we're getting very close to fully driverless in Europe, right? But actually making it work, especially also in the European context, takes so much work because right now we have a lot of PTAs who have some sort of experience with buses, trains and sort of various mobility services. But what do you do when you remove almost all the people from that operation and the technology has to work itself independently from all the people around it, right? You have to build up so many new procedures. You have to think in a very, very new way. If you are a bus operator and you want to operate self-driving vehicles, how do you handle software updates? How do you take care of passengers when there's no one in the vehicle to sort of assist them immediately, right? They have a stop button. If someone in the bus has a medical emergency, the bus driver can attend to them or call the police or the authorities. But if there's no one in the vehicle, how do you do all of these things? So basically every standard procedure has to be developed from ⁓ a new and all of these organizations that are working with self-driving vehicles, they need to transition their entire organization into a new way of working. And we've experienced that that's quite a task. on them, some are very willing and some maybe needs a little bit more nudging to get there. And at the same time for the operator, we operate, for example, mobileye vehicles with MOIA now, but previously directly with MOIA and Oslo. And we need to prove to the authorities that these vehicles are at the very least as safe as private vehicles and as the buses that are already driving. So how do we do that? Right? We need to get out of data. We need to get very deep into the systems to prove that just because you've done it in the States, does that also mean you can do it in Oslo? Not necessarily because the driving, the regulations are so different and the requirements and the public, let's say understanding or perception of this technology is so different. So as an operator, it's very good to have functional systems. That's a perfect place to start. But all the other steps behind it, yeah, there are so many there. Daniel Abreu Marques: And you also sorry, go ahead. Stefanie Berliner: Or maybe ⁓ sorry, Tanya, maybe can I add one one one sentence which makes it a good comparison or ⁓ make it clear to make a case here. As a customer, do you choose the aircraft model when when you want to fly from A to B? Or you or do you choose the airline? I think that makes it very clear why operational should not be underestimated in that sense of autonomous driving. Daniel Abreu Marques: But it's that's really making it clear. And you also see organizations underestimate how demanding A V deployment becomes and what specifically do they get wrong before a project even starts? Anders Toft: I think, let's say with Ruter they've been incredibly accommodating and then be very, very open in Oslo, right? I think they're one of the pioneers in European AV deployments from a PTA perspective. ⁓ And they've been very willing to dedicate resources immediately and to build up a structure or a new organizational structure on their side that can support all the people from the operators and the software providers. So there's always a person leading it and managing day to day fully dedicated. I think maybe some PTAs in Europe, they can underestimate the amount of new knowledge you have to absorb and the, let's say the organizational transition you have to make. ⁓ I think maybe that there is a, maybe they think that what they do with buses or the existing technology or services can be more or less migrated one-to-one in some way to the self-driving vehicles. ⁓ And it makes sense. It's another mobility solution, right? But I think that can also be a component of that they underestimate. What do do if the passenger is sick and there is no safety driver? How do you verify that the vehicle was okay to operate on May 3, 2004? 2024. How do you make sure that all the data that you need, that you can at any time produce it to the authorities if they ask it to? How do you make sure that the vehicle actually does the minimum risk maneuver when it loses connection to the control tower? There are so many things and I'm not sure they understand the full scope of what it takes to deploy AVs. So that's a little bit of something that we've seen. Daniel Abreu Marques: And I think probably also there they underestimate a lot of things when they start kind of a a pilot project, but ⁓ there's a huge gap between a pilot and also the continuous public service. So when a project then crosses that line, what is what they underestimate first in in the really ⁓ public service? Anders Toft: Yeah, ⁓ definitely. We have a, let's say a pilot project in Oslo now, and it's very forgiving in the sense that it's an R &D project. We don't have an uptime requirement. have, no, it's six vehicles that we're going to start with. ⁓ But the purpose for this is to test and validate. And we want to just know more about the platform and its capabilities in the ODD. So in the Oslo suburbs. If we were to, let's say, go to, I don't know, 500 vehicles or so, it would also be with a financial uptime requirement because we have to deliver a service. And that also means you have to supply the correct amount of staffing. You have to have procedures that can support getting a vehicle back into operation as quickly as possible. You also can't really compromise safety, right? So if there is an issue and it grounds the fleet who then becomes accountable and how do you manage all of that liability between the operator, which is the permit holder and the software provider who provides the vehicle platform. ⁓ So I would say basically everything changes from a pilot project to a full scale up scenario. The investment, the amount of ⁓ new team members you need on various sides of the consortium. ⁓ Yeah, it's a huge change. Daniel Abreu Marques: Yeah, and if a city or transport authority calls you today and say, I want A Vs in service next year, so what do you tell them and maybe also something they're not ready for? Stefanie Berliner: Maybe let's start with the timeline, right? If they want to be ready or see the service already in one year from now, I expect they are in the middle of the front loading, already extended, have already negotiated with the technology provider they want to use. So I would ask them a bunch of questions. Where exactly do they expect having the service? What kind of technology they want to use, or what kind of vehicle size, right? Is that already clear? Do they have Do they have already made their front work? Who will be the operator? What does it mean? Do they need a tender for it? Because it's like you say, may ⁓ it might be public if a city is is asking for that. ⁓ and in Europe there are clear, clear guidelines, especially when when it should be yeah, a big service, so to say. And and then I would say if they just for example, what I hear often from cities or from mayors at the moment is still we need just a vehicle as a showcase. We just want to showcase this technology is working, but it's actually not the case. And right now the OEMs and technology providers are not giving any vehicle away for that because this time to prove technology is over. Now you have to prove scalability, operational readiness, so to say, right? They want to have bigger fleets. So they're not interested in all the support work the support work they have to give, for example, with a service of one to two vehicles. They're not interested in. So I don't know where to start. I had a lot of questions and then I would probably Give them a lot of advice where to start if they don't have the made their homework yet. Let's put it this way. Daniel Abreu Marques: Makes sense. And ⁓ I think one reason this is so hard is that also that it's not one single company that does all of it. So maybe let's map who actually does what and can you walk me through the players in a real deployment, like permit holder, operator, OEM, ⁓ software provider, et cetera? And where do people assume one party is responsible when it is really another? Anders Toft: Yeah, so a good example is how we do it in Oslo. There is of course the permit holder. So the authorities will receive an application for these vehicles to operate in a given city or an area and someone needs to be responsible for all of that, basically the day-to-day operation. That means that the permit holder is also, let's say, the operator. So in Oslo, we have the permit. And we are also then liable for anything that happens, which is one of the things that I think people or the public maybe understand differently. ⁓ to take it from the beginning, is the operator. We have the safety drivers. We have the remote staff. We have the back office, so project management, head of operations. In this case, we also have some maintenance activities. Ruter they're the PTA. So they're the service provider. Publicly. Those, they are the ones who are presenting the service. And if someone walks on the street, they will see a Ruter color car in their red colors with their branding. And if they call customer service, it'll be a Ruter person answering the phone. So from a passenger facing perspective, it'll be a Ruter. Holo doesn't get that, let's say public exposure. And that's also perfect for us because we just want to handle the technical and safety aspects. On the other hand, we have ⁓ this MOIA who is the software provider. They work with Mobileye and so they build the platform. They give it to us. We make sure that it works functionally safely according to the service requirements from Ruter And then we have the day-to-day responsibility. I think one of the things where people maybe sometimes don't have the full overview of who's responsible or how it is. is the liability. Publicly, if the vehicle crashes, they will think the vehicle is responsible and by definition then Ruter Legally, it's the permit holder, so we are liable for what the vehicle does in any given situation. And then practically, It'll be the consortium. So we have a, of course, a contract with MOIA and Ruter and so on. And there can be many, many things that can cause a crash, right? It can be that maybe a component didn't work correctly. Maybe a pre-ride check wasn't submitted or maybe there was an issue that we missed entirely. But we would always look internally and check the data and information with partners to see what went wrong on this given perspective. But publicly. People will always think it's the vehicle ⁓ and by extension then the service provider. Daniel Abreu Marques: And then you have more or less ⁓ also in your contracts written down ⁓ who is responsible in which case and then you have some sort of I don't know black box who can exactly we where you can exactly determine what Anders Toft: Exactly. Exactly. Because many of these decisions are made by a computer. ⁓ If it's a level two permit, which is what we have in the ⁓ most pilot projects on level two, where the safety driver or the driver of the vehicle is responsible for everything that it does. Of course, they are the safety concept. So they can always intervene if the vehicle meets some sort of operational limitation. or if there is an edge case where they need to jump in and let's say steer the car away from whatever would happen. ⁓ But we do need to do ⁓ root cause analysis every time. So we will send the information, the timestamp, the drive sessions, and we will upload the data to the software provider, and they will give a root cause analysis of what happened. We won't see let's say the equations leading up to decision, but we will know if it's if it was the LIDAR it was the camera it was a Let's say a policy decision or it was a glitch or if it was an external factor So we will get that information from them so that we can always say if asked by the authorities what went wrong here and how do you plan to Fix it going forward Daniel Abreu Marques: Sense and if you have all these players, so where do the dependencies between these players create the most operational risk, in your opinion? Anders Toft: There can be sometimes a safety, let's say, interest and a commercial interest. We are spared from it in the R &D phase, but let's say we went to a scale-up scenario. We as the operator, we're liable for anything that happens, even in a level four capacity, right? We're still liable for what the vehicle does to the authorities. And the authorities, if they have a question, they will go to the permit holder every time. They won't go to the software provider and they won't go to the service provider. So we always need to understand what happened and we always need to make sure that we can say vouch for what the vehicle is doing and then we put it into operation. It can be, there could be a scenario where we find an issue. It can compromise the safety functions of the vehicle. Let's say a safety critical component doesn't work and we need to ground the entire fleet. Or we can only drive manually for the time being or only with safety drivers. But it's a service deterioration and it's not what the service provider wants. So we will ground it. And then the liable party, if it's a software component, will be the software provider. It'll be in their best interest to get a new update in as fast as possible. And then it'll be in our best interest to make sure that everything is as safe as possible. So there can be sort of how fast can we push an update versus how safe can it be? And these timelines sometimes don't always meet at the same point. ⁓ So it's not something we've experienced so far, but it's something that I could imagine could happen. Definitely. Daniel Abreu Marques: Makes sense and maybe when we're talking about ⁓ things going wrong. So w when something goes wrong at roads or how many organizations actually get pulled in to figure out ⁓ why this happened and also who ends up owning then the the answer at the end? Anders Toft: Yeah. It should always be the operator that can answer such questions in the end because they're the permit holder. But let's say we experienced that the vehicle does an unexpected maneuver on the road. Maybe it fails to yield to another vehicle or suddenly stops or something. ⁓ The operator is the frontline, let's say, component in this consortium. So we will always be the ones to see it first. It's our responsibility to tag it. Every vehicle platform has a way to tag issues and to report it to the software provider. And then they are then expected to make an analysis and come with a, let's say an overview of what happened, why did it happen and what do we do to fix it going forward. Depending on the severity, this can be very, very quick, immediate, or it can be put to the backlog, but let's say it's a critical issue. And then they will come with their perspective as to who owns the issue. Usually if it's software, it's the vehicle handling generally. It's a software provider. It could also be that a safety driver fails to yield or they stop too late or they don't follow their training. And then it would be us, but that's something that the software inevitably should show as well. Because we can see the reaction of the vehicle, the thinking. And we can also see the reaction of the safety driver per timestamp. But there's always a root cause analysis and then there is an alignment of whatever that information shows. We log it in the systems and then we have it for a software fix or for a new training or for a general alignment of procedures between the parties. Daniel Abreu Marques: Now we talked a lot about the the permit holder and I think this is a a role that most listeners probably never had thinked about before. ⁓ so maybe Steffi, could you in in plain terms e explain what what is the permit holder and what are you signing up to be accountable for as permit holder that no one else in the chain is doing? Stefanie Berliner: Mm. Sorry, I just have to think about that. I don't repeat everything Anna said. Maybe just take let's take that out. I think first first important thing is as a permit holder, you're accountable for the service and not for the vehicle itself, so to say. I think that makes a huge difference. And I love to work more more with examples. I think a good example is working in the public transport industry, because with the service comes also Like Anders already said, the operational safety and the compliance and the risk ownership, so to say, right? But now we are talking about we have we are working with different vehicle sizes, different areas. So maybe it's a hybrid solution at the moment. We have to discuss also, we have to be accountable that we have an integration in our existing systems, like the control center. Like, for example, when it comes to German law, the technische Aufsicht, so the remote assistance, is it called. But same also goes for Switzerland, right? You need always the support also from the technology provider, so to say. How do you cope with passengers who are stranded somewhere and they still have to get from A to B. So and you be accountable to get your passenger and your client, your customer from A to B safely compliant, best in time. It has to be reliable and so on. So it's more about the technology. It's really about to understand what do you need to run your service properly for your client, right? Because if they don't want to use it because it's a it's a shitty service, in the end the technology can be as good as it w as it should be, but it doesn't fulfill everything for for the customer in the end. And this is something where also we discussed dependency, but also where I see the difference. of accountability here. Daniel Abreu Marques: Make sense and and maybe one question that is ⁓ not on the question list, but I will ask it and then we can cut it out afterwards if you don't have or don't can if you cannot answer it. But ⁓ how is the the contractual ⁓ kind of organization? So do you have then the ⁓ contract with with Ruter? And Ruter has also the contract with ⁓ the virtual driver company and also with the OEM and something like that, ⁓ Do they outsource more or less everything to you and say, Hey, I want the vehicle or I want two vehicles that are driving themselves in Oslo? Stefanie Berliner: Maybe I s I start in because we we can answer that, but more in an overarching way, right? Because depending of course on the project. Most of the project we learned are a lot we're talking a lot about collaboration, partnerships, consortium. So you you work together, bring the case, what will be the roles responsibilities between the partner splits, so to say, right? Who is doing what exactly? And when that is clear, of course you build A contract behind it and makes it clear before, for example, you bid on a tender. Then, of course, the consortium has a contract with the one who set up the tender. Here in that case, it could be, for example, Ruter. And this could be one example. The other example is, for example, we hear a lot of partnership between the technology provider has to fulfill some duties towards the operator and the other way around. Yeah, maybe let's cut that out and restart because I think I made the case when I when I complained that you have different roles and responsibilities and you build that on a contract. And it's not like you have one contract only with one party, you have a lot lot of contracts between different stakeholders within within a project. But but we can dig deeper into that then you I'm just not sure. maybe it's easier to refer to a specific project and then this is not not yeah, not a secret to tell. Okay. Daniel Abreu Marques: Yeah. No worries, no worries. All good. So I will I will continue with our planned question. So ⁓ Authorities can also audit you and require incident reporting and what does a regulator actually want to see and also can most operators produce it? Anders Toft: Yeah, it's a very good question. think it's one of the very, things that make Europe also quite a difficult place because we have so many countries and each country has their own regulations and their own requirements. If you take Switzerland, where Stefanie lives and Oslo, where we operate, the requirements and the authority from the authorities, they couldn't be more different basically. In Oslo, I think they have a very open authority, the Statens Vegvesen They don't have any specific set of data set data points that they want to see at any predefined frequency per se. We obviously have a risk assessment in the application material where we defined risk, its potential impact and the potential for it happening. And that's data we need to track. And we need to also report to them if we see an increase in any of these predefined risks that go beyond what we initially expected. And we need to tell them if this has any sort of safety implications at what point and what do we do about it. In Switzerland, I can't remember specifically, maybe Stefanie, she can bid in, but they also have much more requirements in terms of actual disengagement and what they want to see. Maybe Stefanie, can, you're the expert there. Stefanie Berliner: Yeah, maybe to give you a few examples, right? There there are some different kind of measurements when it comes to the ordinance which is set in place in Switzerland. For example, you have all the measurements, of course, where's the vehicle located at every time ,live? You have to be able to to have always connectivity between the remote assistance and the vehicle. So they measure latency, for example, they measure disengagements, they also measure behavior for different road types. They want to understand over critical measurements how the vehicle behaved in in junctions, intersections, residual roads, motorways, whatever it is. And then you have to collect the autonomy state status for the moat. And and when you had disengagement and very specific, but in the end they want to have a measurement really to be able to implement a safety management system for the operator. On the other hand, They negotiate, you negotiate as an operator together with the authorities, really, to say, okay, does it really make sense? Because our Risk profile looks XYZ. So it makes sense, for example, to collect these amount of percent, or for example, in in autonomous mode for these intersections, because this is really critical intersection. Normally we have a lot of incidents and so on. This just an example, because you cannot say really precise what the authorities expect, what kind of measurements, what the amounts. This is dependent on the area you serve the service or you operate the service, and also of course of the technology, how many vehicles and and also for ex yeah, I think it makes it very easy. If you are driving on the motorway, it's way simpler than driving in the inner city of of for example a city like Zurich, Basel and so on, which is so much complexity when it comes to intermodality with bicycles, with passengers, with ⁓ with logistics and so on, trams, metro, train stations, and and all together that makes it way more complex. So I could assume that authorities want to see way more data in these areas compared to, for example, at driving only on the motorway. Daniel Abreu Marques: Makes sense and if you cannot trace why the the vehicle did something, you you cannot hold the the permit responsibly. So how hard is that traceability to build and also who co controls the data that you just mentioned? Anders Toft: I think it can be a little difficult sometimes when you have an external operator such as we are and then we ask, hey, why did the vehicle do this action in this roundabout? Why did it yield for the car? We of course need it as we have a responsibility to the authorities to always be able to vouch for what the vehicle is doing and understand it. If we don't understand the actions of the vehicle, We can't say that it's safe enough and we can't predict how it's going to behave. as let's say, as a developer of the technology, if you make the, ⁓ the policy, say, I'll start over as a developer of the technology, you have to give a certain amount of information to an external company. ⁓ we're not married, let's say to Mobileye We can also work with other companies, ⁓ in, right. So we'll get some sort of understanding of how there's technology works. And I think maybe they can be a little afraid that we'll use it to compare with another vendor on the other side or, but at the same time, we will still need the information to vouch for a safe operation. So the, the transparency is something that you have to fight for. I will say with, Mobileye specifically, we had a very good, we had a very long process here. It was their first, let's say real deployment with an external operator in Oslo. They have a little bit more now. And we pressured them quite a bit because we are very, very keen on all the data that Stefanie just mentioned that the Swiss authorities require. We just want it because we think it helps us, helps make us better. ⁓ And it was quite a journey, I think for Mobileye and for us to sort of merge our ways of working and our understandings because we want to know why the vehicle doesn't yield when it needs to. There are very specific requirements in Norway that also makes it very difficult. ⁓ But we also had to learn to work with Mobileye and to say, how does it work for them and how can we sort of make the best process for these issue analysis and the support tickets that we create and so on. But the data sits, of course, with the software provider. It's their system. It's their brain. The policy decisions are within ⁓ their computer. And it's not something we can see and it's not something we need to see. will never see the equation. We just need to see, let's say the end result. It's a super, super difficult, but also very interesting process when working with external companies and how much insight can you get into. And, but also I think Mobileye learned in the end, the more information that we feed to Holo, the better they can sort of package it and the better information and feedback we can give them later on. So it was a very good. iteration, I think, between all parties. And I think it really, really made us, made us better. It made me better personally, because it was also my first major deployment. And I think it also helped Mobileye into shape, shaping some of their products, for example, the map or ⁓ the, perception ⁓ policy decision-making. Daniel Abreu Marques: Makes sense. And y you frame safety also mostly operational and organizational, not just technical. So what does that d distinction mean and for how you run a service modus? Anders Toft: There are many safety aspects, of course, going. Let's say if you have a, if you have a bus company, you have the, of course, the certified people and they're trained to handling the bus. But now so many components, they go over to, let's say systems that have to handle it automatically and they have to handle it the same time every time. And they also need to know when to apply an exception and you remove the human element and you add more and more computer, computer based decisions. based on so many things. ⁓ So there's a human safety management system and the technical safety management system that you sort of need to merge here. becomes everyone that you employ that works around these vehicles needs to be trained a little bit extra in a sense because they really need to understand the impact of what does it mean if the LIDAR shows a little bit ambiguous data. What does it mean for the potential perception of a pedestrian later up? What does it mean if it's foggy? Does that have an impact on the perception capabilities of the vehicle? What does it mean if, let's say with a safety driver, I have a headache or I don't feel so good? Does it affect maybe my ability to intervene timely enough? So there are so many things that you need to consider. And with self-driving vehicles, it's a zero, what do you call it? It's a very low threshold business, zero tolerance business industry. That's what I wanted to say. If an AV does something, as we saw with cruise many, many years ago, it can shut down the entire business and everyone can lose their job. So even this small issue, not the cruise was a small issue by any means. But even a small issue can have major implications for the entire company, right? So understanding how safety sort of plays together and works and also that every person in the company has a huge safety responsibility. I can't understate it enough how important that is. So in Holo, we do quite frequent incident exercises and it's also part of our training program that there is, we have a page dedicated to safety. ⁓ We talk about it a lot because it's really, really important that everyone understands what they're doing, why they're doing it, and why don't we just do something different because they think maybe this way is smarter or faster, right? And I'm happy to say that they're very, very good at it. They do an amazing job, all the team members we have. ⁓ But it's something we already stayed in the hiring process and that we test so early. ⁓ Daniel Abreu Marques: And what sits inside a safety management system for an A V operator that a normal bus operator, let's say, w would not have? Anders Toft: ⁓ A good example is that, and I'll be honest, I don't know much about buses, but let's say the self-driving vehicles, they're evolving rapidly. And especially in the R &D project, ⁓ it's the expectation that maybe every month or every two months, you'll get a new software and this will change the way the vehicle behaves. For the better, hopefully, that's the intention. ⁓ But it means that the performance of the vehicle is always improving. That means that the ways of working within and without the vehicle, they can also change accordingly. ⁓ The software developer, so let's say MOIA in this case, they will be keen on improving the product ⁓ and making it as good as possible so it can handle all traffic scenarios independently from a human, let's say assistant. ⁓ So every month there will be a new software update. And it'll change the handling, the parameters of the LIDARS the cameras, or the way that it perceives specific vulnerable road users, or many, many things, the way it yields or enters or exits around about. Some of these definitely work, but it's not always guaranteed that the deployment in a new, let's say, specific ODD, such as Oslo, will work every time. So we always need to test and verify all software updates to make sure that the local adaptation and implementation also works. ⁓ So the change management process of reviewing software updates, testing, giving feedback, training, making sure documentation is updated. That's a recurrent theme in AV mobility. Daniel Abreu Marques: Let's say ⁓ something goes wrong and there is an escalation. So can you can you walk us through the the first few minutes after an incident? So what actually happens? Anders Toft: Yeah. So we differ first and foremost between crash and non crash incidents. ⁓ Not so I'll start over. ⁓ Let's say there is an incident and the vehicle hits ⁓ a lamppost or something. Now, let me start over again. I need to find a good example. Okay. Let I'll start up. Let's say there's an incident and the vehicle hits another car, the self-driving. Let me try again. Let's see. Stefanie Berliner: Maybe let's make honest j just bring up an an current ex or not a current, but an example we already had that makes it sometimes a bit easier, right? When the car, for example, almost crashed the wall. Maybe something like that, then you have a Anders Toft: Yeah. Okay, I can say that. So can I say that we had the incident or should I, should it be more what if? Stefanie Berliner: Yeah, what if but then you know exactly what what happened afterwards, right? Anders Toft: Okay. ⁓ Let's say that the vehicle does something suddenly that had huge safety implications and the safety driver that ran the vehicle needs to immediately disengage and do a counter steering. ⁓ The first thing that we do is that the safety driver on site, they take care of the, if there are passengers or any people on board, they secure the site, they make sure that the vehicle doesn't pose any further risk to the surrounding area. If it was a level four situation, the vehicle would then have to do the minimum risk maneuver and the minimum risk condition. ⁓ Immediately, the incident is escalated to our supervision team. That's like a control tower that's always ready and remotely monitoring all the vehicles and the operation. And they have an incident report that they fill out. So the onsite personnel, they're trained in reporting all the relevant information. say if the AV or the vehicle was an AV mode or manual mode, how did they disengage? What happened? Damages, injuries, witnesses, relevant. And they'll report it to the supervisor and the supervisor is the then incident manager. They'll make sure to assist the onsite personnel. They'll document everything. They'll make sure that everything gets logged in the correct systems. They'll make the tickets and they will notify and escalate to everyone. So from site, everything goes to the supervisor and the supervisor then funnels the information to all other relevant parties. So obviously the operator will need to know. I get a phone call. The project manager gets a phone call. There is maintenance, ⁓ people, HR, CEO, everyone with a C title. And then we also need to contact the service provider and the software provider, of course. Depending on the severity, there will be, let's say, a consideration of any public statements. ⁓ But one of the most important things that we do is to see, is it a local incident? Or is it a potentially global incident? If it's someone crashes into us, we're quite confident that they they drive it into the rear of the vehicle and we're driving at normal speeds. It's probably an external fault. If the software does something by itself that we can't really explain and it has safety implications, there is a chance that it affects the entire fleet. So very quickly in the flow, we make a consideration if it's a local or a global issue, and then we act accordingly. If it can be a global issue, we will ground the entire fleet immediately, and then we will wait for the software provider to tell us what they see basically. Daniel Abreu Marques: Makes sense and you measure autonomy per kilometer per road type, not just per kilometer. So why is the is the simpler number misleading and also what does the road type breakdown reveal? Anders Toft: Yes. There are the Oslo ODD, for example, or the Oslo suburbs. It has some very, very good roads. Let's say for people that don't know, a good road is typically a road with predictable behavior from other road users and very clear lane markings. and very clear separation between road users. So you have opposing traffic, but they're in a different lane and there is a white mark between them. cyclists are on a bicycle lane and pedestrians are on the pedestrian path. You have very clear signs and you have very clear traffic lights. That's on the good scale. And if we go all the way to the other end, we have the small urban residential roads where there is no separation between ⁓ the oncoming traffic. pedestrians, cyclists, everyone in cars, they use the same road. So the, let's say the behavior for the road users becomes incredibly unpredictable. You don't know if a kid's going to run out with a football, if the cyclist is going to be in the middle of the street. So the ODD becomes so complex and we see that a lot in Oslo. And initially we just had the rate of autonomy. How, say if you drive 10 kilometers. and it's autonomous for eight of them, okay, we have 80 % autonomy. But you say, yeah, but what about the small roads? Okay, that's actually a good question because those would probably have a much lower number than the large roads, right? That is why we started diving into it. And then we saw that there's quite a big discrepancy as expected between the normal roads, so to say, and then the small residential roads. And it's a very, very good point because at some point, the vehicle needs to drive on this road itself. If you want to service these areas, which Ruter wants obviously, the vehicle needs to be able to all traffic scenarios. And if we then get the data that shows on these roads we have, I'm just going to say a number like 80 % autonomy, on this road it's 70 and on this road it's 98, it's pretty clear what are the priorities going forward and where is the development effort needed. It's also good data to show to the authorities because we can say the risk assessment contains these issues on the small residential roads and these are our mitigation strategies. On the larger roads, we would want to apply for a level four permit now so we can drive level four there. And then you can sort of gradually roll out the level four into the ODD, but take one road type at a time. ⁓ It also helps in talking to Ruter So they have an insight into what is the, let's say, what is the expected service that you can expect as a normal user on various road types. So they at least have an understanding of that, right? But it gives it incredibly good granularity into the vehicle performance. So I think it's absolutely necessary to measure in those numbers rather than just normal mileage in total. Daniel Abreu Marques: Super interesting and maybe let's stay on and measurement. ⁓ so which metrics tell you that a deployment is genuinely maturing versus ⁓ just accumulating miles, so to speak? Anders Toft: There are so many metrics that can be in that. ⁓ There are service metrics. So how fast are you moving around to pick up people? There is the rate of autonomy, obviously. are we more or less autonomous from version one to version two? There is also, let's say, uptime generally. If we have an issue and we need to troubleshoot it, how much time does it take to reboot the system or to troubleshoot A, B and C? Is that improving or is it digressing from software to software? There's also the amount of issues we see. So we track every issue. If the vehicle can't service a mission, a test mission or a real mission, we'll put it down as downtime and we'll track the time it took for the vehicle to recover, let's say serviceability. If we then see that software 2 introduces new issues, then okay, it may be the rate of autonomy is better, but there are new issues introduced that decreases the overall performance. So there are many metrics, but maybe to sum it up, it's the service metrics. How fast can you get from A to B safely? There are the rate of autonomy metrics, how is it performing on various road types, and then there are the issue metrics. Are we seeing new issues? Are some being removed? And does it take the same time to troubleshoot or does it go faster, essentially? Daniel Abreu Marques: When you think about remote intervention, so what would what b would be a number that would kind of maybe worry you and what is one that signals you are ready to scale? Can you say that? Anders Toft: Yeah, so I think a good clarification to make is that at least in a European context, the remote assistant operator, which I think is a good distinction to make, would never actually perform the DDT, so the dynamic driving task of the vehicle. They won't remotely press the stop button and they won't control remote operate the vehicle. ⁓ At least not in Norway as far as it's indicating. ⁓ So. But there can be that a self-driving vehicle needs remote assistance because it's blocked and they need to, let's say you can do scenario A or B, or that a passenger has an issue that needs to be attended to very, very quickly. And I would say anything above two seconds becomes critical. If it's a traffic scenario and the vehicle is stuck and it's pending an A or B scenario from the remote assistance operator. Am I allowed to drive up on the bicycle path to overtake this truck that's unloading something, some cargo or something? It's a traffic consideration. The longer you wait, the more you impede traffic. And I think the remote assistance operator should be able to make that call within top five seconds. Anything beyond that, and it can have friction with the surrounding traffic. And if the surrounding traffic gets impatient, It leads to a decreased perception of AVs. And it can also lead to them then trying to drive around, which then increases risk in the area. If it's more safety critical and a passenger needs assistance, two seconds is the maximum. Because then you have the incident management procedure. You need to contact the authorities immediately. You need to send on-site personnel. So I think the reaction time should be as low as... humanly possible. Daniel Abreu Marques: Makes sense and ⁓ maybe another question on on the remote intervention or remote assistance. So when you think about that, about these five seconds threshold, so how how how does a like feasible remote operator to vehicle ratio look like in your opinion? Anders Toft: It's very technology dependent. think Waymo and Baidu and WeRide I don't think it's really information they disclose ⁓ I've Waymo situations ⁓ online where, you know, a vehicle is blocking a fire truck that's going to, ⁓ or an ambulance that's going to an active shooting and it was blocking it for 40 minutes. ⁓ And think that's unbelievably long. ⁓ It indicate. that there weren't enough remote agents to, let's say, to assist the vehicle, I can say. ⁓ But I think from a financial perspective, needs to be somewhere between one remote assistance operator for 15 vehicles, something like that, and then up from there to be financially productive. I think that's the whole name of the game with autonomous vehicles, right? That's why we all do it because it has the scalability that current mobility doesn't have. But it's very ODD dependent. I could, for example, foresee that during summer, let's just say we had one remote assistance operator for 15 vehicles, but during winter, maybe it could be one for eight because the ODD becomes so much more difficult. So there are so many things to consider there that ⁓ it's a very good question, but also very difficult to answer. Daniel Abreu Marques: Yeah, but but you did a good job. So we also talked about ⁓ this the software versions and that they maybe also change the behavior. So contracts comes with SLAs, penalties, thresholds, and how do you set a threshold for technology whose behavior keeps also changing with the software updates or the software version? Stefanie Berliner: I think one important point is especially talking about the support for each case, right? We heard a lot of case in in in the last question. So what what could actually happen? It is you have to you need thresholds, for example, when something is broken. So when it comes to maintenance, right? Because of course for the software in the end the provider is is ⁓ accountable and liable and also want to do the work by his own or or you need a license to do it on your own, for example, then you need ⁓ solution whenever something could happen because downtime means penalties, means a a bad business case. So this is really an important threshold. That's why it's so important to do your front loading really to understand what kind of service parts has to be fulfilled in your in your normally service accountability. So and based on that you have to understand where's the roles responsibility split and what do you expect. So even you as an operator are responsible but often for your tasks you need a lot of support From the technical provider and also the other way around. When it comes, for example, right now to develop certain tools, right? We haven't experienced level four yet in Europe. So we have to understand, for example, how to how to ensure safety within the vehicle. How to ensure, for example, that the doors are clear before you close it. What happens if someone with a handicap ⁓ wants to enter a vehicle, for example. With a wheelchair, blind person, all the things we have to think through and there we need support from each other. So this will also be part of the SLAs we z we will see in the next contracts and and also in the next projects that it's also about develop specific tools and whatever you need to have in place to offer an accountable service in the end. So it's not only about the technology is running safely, it's also to have the best service for for the customer. And so There were some examples where I think some of the thresholds are very clear. For example, when you expect if we have to ground the fleet, within twenty-four hours we have to solve the problem, right? Someone has to fly in if it makes sense, or we need some some spare parts. And this this might be the easy case. The more difficult case is for situation we haven't expected yet or haven't ⁓ experienced yet, for example. And there you need also yeah, you need to make some of the risk mitigation why you define some thresholds also in a service level agreement. So there's a lot maybe which is not defined yet, but I hope I could give you I could have given you some examples. Daniel Abreu Marques: Absolutely. And you talked about situation that you never encountered before. So edge cases and also ana anomalies are where AVs get tested and how do you capture them so a rare event in one city improves also the service in another. Stefanie Berliner: Yeah, sure. I think one of the more common edge case of of course seasonality. So when we are talking winter, driving in winter, this is maybe something which is easily transferable because I think the technology providers who had the chance to drive in Oslo, for example, or Norway, they will will experience a lot of icy roads and also snowbanks and so on. And and this knowledge you can use, for example, if you would use the same the or the same technology f in in other countries where we experience then snow and also slopes and inclining, declining, all the behavior which might be where you find a solution for for the technology. I think and and that makes it also so interesting what we mentioned before, the data you collect, for example, of of the autonomy mode and and the performance behavior for specific road types, intersections, junctions, this is also learning, right? You you can adjust or adapt to another area. So and these are all depending when something is new, of course, an edge case. So the first time this technology maybe drives over a border might be also a different thing, right? To understand there are different traffic rules you have to change. So this this might be something. Or an edge case will be when we first see a kneeling process for the buses, an automated ramp, something like that. This is also an edge case now, but in the future We know how to solve it. So and and this is the good thing that most of it is transferable as soon as you have experienced and made an autonomous solution out of it. Daniel Abreu Marques: So that it makes sense and you also depend somehow on companies whose software you did not write. So so where do operators and technology providers or also OEMs most often misunderstand each other operationally? Stefanie Berliner: ⁓ maybe to make make an easy example. For example, we we drive on the road and suddenly the vehicle stops because someone was running in front of the vehicle. So the technology provider, the OEM, would just say, Hey, the system worked as designed, perfect. But now the challenge starts with the operator itself because he thinks great, but now I have fifteen passengers stranded somewhere and we we blocked By a bus stop and the customer complains and now the real issue are starting. So that that might be something, right? We have a different point of view. And ⁓ of course, a technology provider wants to bring in the best technology to solve these problems, of course. But in the beginning he has not the operator view, he is just the technology view, right? The system worked as designed, but is it designed perfectly to bring in the best service for for the customer? Might might be an example, right? Anders Toft: I think there's also, if you as a service provider, you want to deploy autonomous vehicles and you already have a service, it's usually to also cover a gap that the ⁓ buses don't cover. Usually that would be the last mile or into the more difficult areas, such as the residential roads. So you would think, yes, we want to put the AVs into the small roads because then they can get off the bus stop or at the train station and take the AV home. And the technology provider, they'll come back and say, yeah, we don't think we should support these roads immediately because the rate of autonomy will be compromised to the extent that it doesn't make sense maybe to drive it autonomously there to begin with. Or that the, let's say the validation process to improve it is longer. So you will drive with a decreased service for X amount of months. So there's always also sort of a misunderstanding about what makes a good use case to deploy AVs sometimes. Daniel Abreu Marques: Super interesting. And and maybe one last stretch. So let's also talk about what experience actually buys you and where this with Holo is all heading. So Holo has over ten thousand hours on the Mobileye platform. So what specifically transfers to the MOIA project, for example, and how much does it ⁓ also the timeline f versus a competitor who would start cold more or less? Anders Toft: Yeah, it's a good question. I mean, we learned so much from Mobileye and I think we were also quite good at documenting. And I think we had a very, very good dialogue with Mobileye so that we could stay in touch. And we also had a meeting with them where we say what worked and what didn't work. And I think some of the things that we can really see now what works is, you say the permit. It took three months to write the permit and get the approval. which I think is incredibly quick, all things considered. We knew exactly what to write. We knew more or less what to expect because the safety concepts of the systems are the same. It's the same software provider. There are some new hardware components, of course, but it's the overall same concept. And we could make the risk assessment immediately because we gathered all the data over the two years with Mobileye. And we had a very transparent dialogue with Statens Vegvesen in a region public road administration and we've had that for every for all the projects we've had in Oslo. ⁓ So from a permit perspective it allowed us to basically fast track the permit ⁓ and get it all fixed in three months. I think from a training perspective or the staff all the safety drivers that we have we have ten in Oslo they already know what the systems do. There's a display, they know how to read it, they know what issues they can expect, they know the boot up sequence is a little bit new, but many of the troubleshooting sequences are more or less the same in a new packaging, so to say. It means that the training they had to get on the new platform could be shortened from five days to three days. It means that we go into this project and as an operator, we can already challenge MOIA and say, yeah, we know that The map is like this, the perception is like this, the systems are like that. So we already know what we need to ask and which systems we need to build to get the data. For the, let's say, operational insights. I already know what data points the vehicle can produce and what data points MOIA can access from Mobileye. So I can also, I can be a little bit annoying towards them because yeah, I know that I can get this and this and this data point. and we can have it every second so we can make these dashboards. So I think from our perspective, it made the entire deployment phase at least a year shorter. ⁓ For example, you need a place to pick up and drop off passengers, and those are for now predefined, but it'll be in zones as well. But we know what makes a good PUDO so pick up and drop off location. And we know what the vehicle, the system can handle. We know where it struggles a little bit. So immediately for the testing phase, we could just put in 80 PUDOs. They need to be tested of course, but we know these will work after a few iterations. We know what data we need. know in many ways what issues to expect. We know how it, it, let's say, reacts to vulnerable road users. We know how to troubleshoot and we know what information we can get in a root cause analysis. And all of this we built from scratch with Mobileye in the deployment of the Nio project that we started in 2023, early 2024. So I think we've saved between one year and 18 months, to be honest, because we could do it all so fast. One of the, let's say, things that took a little bit more time was just getting the formal approval and writing from the authorities. They had approved it unofficially, but they had to send, you know, the the legal text and all of that stuff, and it took a month. From our perspective, the permit was officially approved in only two months. There was a huge learning from it. ⁓ They've also been quite impressed, MOIA with our team members and our staff in Oslo and how fast they could, let's say, get into the systems. Daniel Abreu Marques: Yeah, I can imagine it sounds super fascinating ⁓ how fast you could get to this point. And maybe now let's let's look a bit ahead and as the EU regulatory framework also matures, how does the operator and the permit holder role also change and what should the rest of the industry be preparing for now? Stefanie Berliner: I think we made the case clear, right? That operators will need a much stronger software, data and safety capabilities, right? So the question is is more about how do we integrate these autonomous systems into our existing transport, mobility network and an operating model. So it's not only about plug and play, it's really to have the knowledge how to integrate these systems, right? So I think the last years we spent providing that autonomous vehicles can drive. Now We have the next decade is about proving that the service can operate. So I think ⁓ knowledge that you need these capabilities and also build these capabilities, right? You have you need a different skill set, nothing nothing which cannot be educated, right? So I'm I'm not afraid that we lose jobs. It's just an interesting and it's a different approach working with software. And especially in Europe, for example, compared to the US or the Chinese market, the permit The permit holder holds also the safety case in the end, not for the technology, for the service. And maybe also ⁓ was interesting to read your your article, right? Because the the roles and responsibility split, for example, for technology pri ⁓ provider in the US and and its operator in Europe will be different based on the fact that the operator will be the permit holder. And you always need it next to a type approval, you always need also an operating permit. And this is a really big not bitch big this is a really this is a really big case in Europe actually to get this permit, understand it, to understand the process behind, you need standardization, you need to handle release management, change management, the incident management. You have to to not only write everything down, you have to live it and you have to prove it and you have to be auditable. And that makes a totally different game. And also to understand that you need a shift as an operator as and it's not enough you have the right driving license to drive the specific vehicles in the future. Daniel Abreu Marques: Yeah. So that was a really, really packed ⁓ episode. And I thank you, Stefanie and Anders, so much for for coming on the Autonomy Insider podcast. So we learned a lot today. And thank you for taking the time sharing all those experiences with you. And I'm looking forward to see also your progress and ⁓ news on other deployments. ⁓ and then for sure you have an open invite to come back on the podcast ⁓ to share your your newest experience. So thanks again ⁓ for taking the time and See you soon. Anders Toft: Thank you so much for having us. Yeah. Stefanie Berliner: Thanks so much. It was really a pleasure.