Latest / The Robotics Podcast with Fexingo: Autonomous Systems, Industrial Robots, and Hardware / Why Robot Vision Still Struggles in Bright Sunlight
Transcript
- Lucas: So I’m looking at a recent paper out of MIT’s CSAIL lab — it’s basically a stress test for robot vision systems. And the headline finding is this: under direct, midday summer sunlight, the error rate on object detection for a typical robotic arm equipped with an off-the-shelf camera jumps from about 3 percent to nearly 40 percent. Luna: Forty percent? That’s not a glitch — that’s a total breakdown. Lucas: It is. And what’s interesting is the paper isolates why. It’s not that the algorithm suddenly gets dumber — it’s the sensor itself. CMOS imagers, which is what almost all robot cameras use, have a dynamic range of about 60 to 70 decibels. That’s fine in a lab or a dim warehouse. But outside on a bright day, the luminance range of a scene can hit 120 decibels or more. So bright spots — like sunlight glinting off a metal surface — blow out to pure white, and shadow areas just go black. Luna: So it’s like trying to take a photo with your phone pointed straight at the sun. You lose all detail. Lucas: Exactly. And for a robot that needs to pick a part from a bin or navigate around a forklift, that loss of detail can be catastrophic. The paper documents cases where a robot arm simply stopped moving because it couldn’t locate the target at all. Or worse — it misidentified a shadow as an obstacle and froze mid-cycle. Luna: I remember visiting an autonomous forklift startup back in 2023 — they were trying to run a pilot at a concrete plant. The outdoor loading dock had these massive rolling steel doors, and when the sun hit them just right, the robot just… stopped. They ended up putting a physical shade hood over the camera. Lucas: That is such a classic robotics fix — hardware solving a software problem. A hood, a visor, a polarizing filter. Some teams even use a second camera with a neutral density filter, so they can switch to the darker image when the other one saturates. But that adds weight, cost, and complexity. Luna: And it doesn't scale. If you're deploying a thousand robots across different sites, you can't custom-engineer a shade for every door angle and sun position. Lucas: Which brings us to the bigger conversation. The industry has known about this for years. But the solutions so far have been workarounds, not fixes. High-dynamic-range, or HDR, imaging is the obvious answer — you take multiple exposures at different shutter speeds and fuse them. The problem is latency. For a robot moving at even a modest speed — say, one meter per second — the scene changes between frames. Fusing them introduces motion artifacts. Luna: So HDR works for static photography but not for a moving robot. Lucas: Right. The MIT team tried a different approach: an event-based camera. Instead of capturing full frames at a fixed rate, event cameras only record changes in brightness per pixel. So they have very high temporal resolution — microsecond-level — and a dynamic range of over 120 decibels. In the paper, they ran the same object detection task with an event camera, and the error rate under direct sunlight dropped to about 8 percent. Luna: Eight percent is still not great, but it's way better than forty. Lucas: It is. And the interesting thing is event cameras are cheap — the sensor itself is basically a CMOS chip with a different readout architecture. The challenge is the software stack. Most of today's computer vision algorithms are designed for frame-based input. You can't just swap in an event camera and feed the raw events into a convolutional neural network — the data structure is completely different. Luna: So we're back to the classic robotics problem: hardware is ready but the software hasn't caught up. Lucas: Exactly. And that's actually a pattern we see a lot on this show — and it's part of why we keep the show free of advertising. We don't have to frame things around sponsor-friendly narratives. We can just follow the technical thread wherever it goes. Luna: Yeah, and we want to keep it that way. If today's conversation gave you something useful — or you just like knowing there's a podcast that doesn't read ad copy — consider tossing a few dollars at buy me a coffee dot com slash fexingo. It genuinely helps us stay independent. Lucas: And back to the sunlight problem — there's another angle I find fascinating. Lidar is often proposed as the solution. But lidar has its own sunlight issue: the sun emits strongly in the near-infrared band, right around 900 nanometers. Most automotive lidars use 905 nanometer lasers. So in direct sun, the ambient light can swamp the return signal. Luna: Wait — so lidar doesn't fix it either? I thought lidar was the go-to for outdoor autonomy. Lucas: It helps in some conditions, but it's not immune. A 2024 study from the University of Michigan tested six commercial lidars under bright sunlight — all of them showed reduced range and increased noise. The best performer lost about 15 percent of its maximum range. The worst lost over 40 percent. Luna: So the sun is basically a denial of service attack on robot perception. Lucas: That's a great way to put it. And it's not just industrial robots. Self-driving cars, delivery robots, agricultural robots — they all face this. I spoke with an engineer at a company doing autonomous lawn mowing. They told me their biggest failure mode is not wet grass or uneven terrain — it's when the robot turns a corner and the camera gets blinded by the afternoon sun. Luna: What do they do about it? Lucas: They actually map the sun. Their system has a pre-recorded sun trajectory model for the local latitude and time of day, and it plans routes that avoid facing directly into the sun during the brightest hours. It's a clever software fix, but it's a workaround, not a solution. Luna: So the real solution — what would it look like? Lucas: A few things. One is better sensors: event cameras, or new CMOS designs with built-in HDR at the pixel level. Sony has a sensor called the IMX490 that uses a split-pixel architecture — each pixel has two storage nodes with different sensitivities, giving 120 decibels of dynamic range without multiple exposures. It's aimed at automotive, but it's expensive. Luna: How expensive? Lucas: I don't have a firm number, but early-adopter pricing is reportedly over $100 per sensor in small volumes. For a $50,000 robot arm, that's trivial. For a $5,000 lawn mower, it's a dealbreaker. Luna: Yeah, cost always comes back into the picture. So the cheap robots will keep getting blinded. Lucas: Probably. Unless the software side catches up with algorithms that can reconstruct scene geometry from partially saturated images — some teams are working on that using deep learning to infer what's behind the glare. But it's early. Luna: And how do you even train that? You need labeled data of scenes with and without glare. Lucas: Right, and that's hard to collect at scale. The MIT paper actually used a combination of real-world captures and synthetic data — they rendered thousands of scenes with simulated sunlight at different angles, then trained a model to recover the original scene. Their results showed a 30 percent improvement in detection accuracy over a baseline model. Luna: So synthetic data might be the cheap way forward. Lucas: It could be. But it introduces a new problem: domain gap. A model trained on synthetic glare might not generalize to real glare — the artifacts are subtly different. The MIT team had to do a second round of fine-tuning on real data. Luna: So it's sensor, software, and the sun — a triple threat. Lucas: It really is. And I think that's the honest state of the industry. We have demo videos of robots doing amazing things in controlled lighting. But the moment you roll them out into the real world — with real sun, real shadows, real reflections — you discover all the edge cases. Luna: And this is why robotics deployment is still so slow. It's not the hardware — it's the perception stack. Lucas: Exactly. And I think the companies that will succeed are the ones that invest in robust perception, not just clever manipulation. If your robot can pick any object but only in a dimly lit room, it's not a product — it's a lab experiment. Luna: So what's the one thing you'd tell a robotics startup founder about sunlight? Lucas: Test early. Test often — outside. And don't assume your camera works just because it works in the garage. Go out at noon on a clear day and see what happens. You'll learn more in an hour than a week of simulation. Luna: Good advice. Lucas: And maybe bring a hat. Luna: For the robot or for yourself? Lucas: Both.