Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Tesla Reengineered Its Autopilot Stack for Pure Vision
Transcript
- Lucas: If these conversations are useful for what you're building or running, you are in the right place. I'm Lucas. Luna: And I'm Luna. This is The CTO Podcast. Lucas: Today we are talking about one of the most audacious architectural migrations in the history of automotive software: Tesla's decision to rebuild Autopilot from a sensor-fusion system to a pure vision stack. Luna: And not just a software change. They physically removed the radar and ultrasonic sensors from the vehicles. That is a supply chain decision as much as a code decision. Lucas: Right. So to set the scene — up until around 2021, every Tesla with Autopilot had a forward-looking radar, plus ultrasonic sensors around the car, plus eight cameras. The system fused data from all those sensor types to build a picture of the environment. Luna: And radar has certain strengths. It handles fog, rain, direct sunlight — conditions where cameras can struggle. Why would you give that up? Lucas: Good question. Tesla's argument — and Elon Musk was very public about this — was that radar creates a crutch. The neural network learns to rely on radar for depth estimation, but radar has its own failure modes. It can't distinguish a metal road sign from a pedestrian. It can't see color. It bounces off the ground. And most importantly, if the camera and radar disagree, the system has to have a conflict-resolution layer that adds latency and complexity. Luna: So they decided to solve the hard problem: make cameras work in all conditions, instead of fusing imperfect signals. Lucas: Exactly. And this required a complete redesign of the perception stack. Under the old system, the cameras were doing object detection, the radar was doing depth estimation, and the ultasonics handled close-range parking. The new system — Tesla calls it Tesla Vision — processes all eight camera feeds through a single neural network that outputs a vector space representation of the world. Luna: Vector space representation — what does that mean in practice? Lucas: Instead of detecting individual objects and then tracking them, the network learns to predict the entire scene as a set of spatial coordinates — lanes, obstacles, traffic lights, pedestrians — all in a common coordinate frame. It's an end to end approach. The network sees the raw pixels from all cameras simultaneously and outputs a unified bird's-eye view. Luna: That is a massive increase in input complexity. Eight cameras at 36 frames per second, each at 1280 by 960 pixels — that's something like 36 million pixels per second that need to be processed in real time. Lucas: Right. And that's why the hardware had to change too. Tesla's Hardware 3.0 — which they started shipping in 2019 — includes two custom AI chips designed by the Tesla team. Each chip delivers 36 trillion operations per second. That gives the car enough compute to run a large neural network at real-time latency without relying on the cloud. Luna: So the architectural bet was: build a sufficiently powerful neural network and enough compute on the car, and you can match or exceed radar performance without the fusion complexity. Lucas: And the data pipeline to train that network is staggering. Tesla has been collecting video clips from its fleet for years — every time a driver intervenes or a disengagement happens, that clip gets flagged and sent back to the training cluster. They built a supercomputer called Dojo specifically to train these vision models. Luna: But here is where it gets interesting from an engineering culture perspective. In May 2021, Tesla started shipping cars without radar — even though the vision-only software wasn't fully validated yet. Some features like Autosteer were initially limited to 75 miles per hour. There was a period where the cars had the hardware but not the full software capability. Lucas: Yeah, that is a bold move. Most companies would validate the software first, then remove the hardware. Tesla did it in reverse. Andrej Karpathy, who was leading the AI team at the time, said they wanted to force the team to solve the hard problems. If radar is still there, engineers will keep using it as a fallback. Luna: It is like turning off the backup generator to force the team to make the primary system work. Lucas: Exactly. And that culture of moving fast and breaking things — applied to a safety-critical system — is controversial. But it did accelerate the development. Over the following months, Tesla gradually increased the speed limits, restored features, and eventually the vision-only system surpassed the old fused system on metrics like mean time between disengagements. Luna: Honestly, if today was worth a coffee to you, that is the link — buy me a coffee dot com slash fexingo. Lucas: We keep the show ad-free, and listener support is what makes that possible. So if you get value from these deep dives, that is a simple way to give back. Luna: And we mean it — no pressure. A coffee's worth. That link again is buy me a coffee dot com slash fexingo. Lucas: So back to the architecture. One of the key design decisions in the vision stack is that the neural network doesn't just detect objects — it predicts their motion. The network outputs a vector of future positions for every detected object over a horizon of several seconds. That is called occupancy flow. Luna: Occupancy flow — that is different from traditional tracking where you detect, then track, then predict. Here the prediction is baked into the network output. Lucas: Right. And it's trained on massive amounts of real-world data. Tesla uses a technique called 'imitation learning' — they record what human drivers do, and train the network to mimic that behavior. But they also use simulation to generate edge cases: a child running into the street, a car cutting you off, debris on the road. Luna: Simulation at scale is hard to get right. The simulated environment has to be realistic enough that the network doesn't learn quirks of the simulator. Lucas: Tesla addresses that by constantly comparing simulator performance to real-world performance. If the network behaves differently in simulation versus reality, they adjust the simulator. It's a feedback loop. Luna: So what did the engineering team actually rebuild? Was it just the perception module, or did they touch the entire stack? Lucas: They touched everything. The perception module was the biggest change, but the planning and control layers also had to adapt. The old system had a radar-based occupancy grid; the new one uses a vision-based occupancy grid. The lane detection, the traffic light recognition — all had to be retrained. And they had to build a new calibration system because cameras need precise alignment to produce accurate depth. Luna: And the calibration is done in the factory, I assume? Lucas: Yes, but Tesla also developed a self-calibration routine that runs on the car. If the cameras shift slightly due to temperature or vibration, the system can recalibrate itself while driving by comparing static features like lane markings. Luna: That is clever. It reduces the need for service visits. Lucas: Exactly. And it ties back to the philosophy of simplification. Fewer sensors, less hardware complexity, fewer service points. The car becomes a software-defined device that improves over time. Luna: But there is a cost. Removing radar means the system has to work in all weather conditions. How does pure vision handle heavy rain or fog? Lucas: It uses a combination of techniques. The network is trained on data from all weather conditions — Tesla has collected data from snow, rain, fog, night. The cameras also have wide dynamic range to handle low-light situations. And the system uses temporal information — if a camera is momentarily obscured, the network can still predict object positions from previous frames. Luna: It seems like the trade-off is that the vision approach requires more compute and more data, but reduces hardware cost and complexity. Lucas: Exactly. And that trade-off is central to the engineering philosophy at Tesla. They believe that software and AI can solve problems that traditionally required expensive hardware. It's the same philosophy behind their decision to use a single central computer instead of distributed ECUs. Luna: To wrap up — do you think the pure vision bet will pay off in the long run? Or will Tesla eventually add back a radar or lidar? Lucas: I think they are all in on vision. They've said publicly that they believe vision is sufficient for full self-driving. And the architecture is designed to improve with data and compute. But the challenge is proving safety — regulators in many countries still require radar for certain features. So the engineering is there, but the regulatory path is uncertain. Luna: So it's as much a regulatory bet as an engineering bet. Lucas: Exactly. And that is the kind of challenge that makes this story so interesting. It's not just about the code — it's about the conviction to remove hardware before the software is fully ready, and the engineering culture that makes that possible.