Latest / Automated with Brian Heater / Ali Agha on Robot Hallucinations, Physics, and Real-World AI

Transcript

Ali Agha on Robot Hallucinations, Physics, and Real-World AI · Automated with Brian Heater

• 0:00
  SPEAKER_00: When you go to the real world, everything is gonna become off-nominal very soon.
• 0:05
  SPEAKER_00: Everything is gonna become edge case and corner case.
• 0:07
  SPEAKER_00: And most of your theoretical frameworks break in the real world.
• 0:13
  SPEAKER_00: I feel from day one, we're taking a step back and designing the architecture with that appreciation of what we are gonna face in the real world.
• 0:23
  SPEAKER_00: In text and language, we required hundreds of trillions of tokens of data.
• 0:27
  SPEAKER_00: In robotics, who knows?
• 0:28
  SPEAKER_00: The variation is even even higher.
• 0:31
  SPEAKER_00: And with that hundred trillions of tokens of data in conversation, you still hallucinate all the time.
• 0:38
  SPEAKER_00: When it comes to real world and physical world, the tolerance is much, much lower.
• 0:43
  SPEAKER_00: One of the reasons we are in environments like construction is the site changes every minute.
• 0:50
  SPEAKER_00: Yeah.
• 0:50
  SPEAKER_00: So the richness and quality of data is unbelievably higher than any other domain we have deployed in.
• 0:56
  SPEAKER_01: More and more companies that have started off as big physical AI model training companies are also building like a full stack, but that seems like that's not something that you're interested in at all.
• 1:08
  SPEAKER_00: We don't rule that out.
• 1:09
  SPEAKER_00: At some point we might do something, but hello everyone.
• 1:24
  SPEAKER_01: My name is Brian Heater.
• 1:26
  SPEAKER_01: I am the managing editor at the Association for Advancing Automation.
• 1:30
  SPEAKER_01: We are back with another live recording of Automated.
• 1:34
  SPEAKER_01: This one is coming to you from the Automate Show Theater at Automate in Chicago a couple weeks back.
• 1:40
  SPEAKER_01: We're speaking with Ali Amakah, the uh CEO and founder of Field AI.
• 1:46
  SPEAKER_01: Uh Ali also has a fascinating background involving things like spaces, helicopters, and March and Caves.
• 1:54
  SPEAKER_01: We'll dive into all of that.
• 1:55
  SPEAKER_01: Uh it's a great conversation.
• 1:58
  SPEAKER_01: Thanks to everybody who's subscribed recently.
• 2:00
  SPEAKER_01: Getting those numbers up is exactly what allows us to do the show each week.
• 2:04
  SPEAKER_01: So please like and subscribe if you haven't already.
• 2:06
  SPEAKER_01: Uh and in the meantime, please enjoy this conversation with Aliaga of Field AI.
• 2:12
  SPEAKER_01: Vision and AI are changing what's possible in industrial automation from quality inspection and predictive maintenance to robotics, logistics, medical devices, aerospace, and manufacturing.
• 2:23
  SPEAKER_01: At the advanced vision and AI conference, you'll hear from industry leaders, see real-world AI applications in action, explore the latest in machine vision, computer vision, sensors, and computing and automation technology.
• 2:36
  SPEAKER_01: You'll also be able to connect with the people who are shaping what comes next.
• 2:40
  SPEAKER_01: Whether you're building, buying, or implementing these systems, this is a chance to stay ahead of the curve and see how vision and AI are moving from promise to practical impact.
• 2:50
  SPEAKER_01: Learn more and make sure to buy your ticket early at the Advanced Vision and AI conference September 23rd and 24th in Santa Clara.
• 3:02
  SPEAKER_01: Thanks for sticking around an extra day.
• 3:04
  SPEAKER_01: Thank you.
• 3:05
  SPEAKER_01: You were with us yesterday at the Humanoid Robot Forum.
• 3:08
  SPEAKER_01: And I said, watch, I said this last time you and I spoke several months ago when you had your big fundraising round.
• 3:16
  SPEAKER_01: I said, we need to get you on the podcast for a lot of reasons.
• 3:20
  SPEAKER_01: One of which is I really want to talk about your background.
• 3:24
  SPEAKER_01: I want to talk about I want to talk about field AI, I want to talk about physical AI, we're gonna get there, but I want to talk about all of the stuff that you worked on prior to this.
• 3:34
  SPEAKER_01: I want to talk about your time at Qualcomm, I want to talk about your time at NASA.
• 3:39
  SPEAKER_01: Primarily, I want to talk about space helicopters.
• 3:42
  SPEAKER_01: What is a space helicopter?
• 3:45
  SPEAKER_00: Yeah, um, maybe I can go a little bit back in time.
• 3:48
  SPEAKER_00: Um and um I think about maybe uh now 15 years ago, I met my co-founder back in MIT.
• 3:56
  SPEAKER_00: Back in the day, we used to work on coordinated assets like Arial and Grand Robots, working together for a variety of different missions, commercial and federal.
• 4:09
  SPEAKER_00: Um my co-founder Shai, he left for DeepMind, building foundation models.
• 4:13
  SPEAKER_00: Uh I left MIT for Qualcomm to put algorithms and autonomy solutions on Qualcomm's small, lightweight, low-power processor, a Snapdragon, that is on many of our phones.
• 4:28
  SPEAKER_00: And it was an exciting experience.
• 4:31
  SPEAKER_00: We did their the first CES show with quadcopters and Snapdragon.
• 4:36
  SPEAKER_00: And um, the journey to NASA and space started there, and NASA used to come and knock on the door in those days and uh tell us, hey, there's this uh helicopter, we are gonna fly on Mars.
• 4:50
  SPEAKER_00: It was a secret mission.
• 4:51
  SPEAKER_01: This is like getting the men in black coming to your door, knocking on your door, saying they had a secret space helicopter, they don't have to work on it.
• 4:57
  SPEAKER_00: Exactly.
• 4:57
  SPEAKER_00: And they were trying to reduce the weight and power on every aspect of the solution because you're flying something on a thin atmosphere on Mars.
• 5:07
  SPEAKER_00: And uh Snapdragon at the time was one of the lightest weight, lowest power processors out there.
• 5:13
  SPEAKER_00: And they were exploring the feasibility of using that for a for a space mission.
• 5:17
  SPEAKER_00: So that's how I ended up joining NASA JPL.
• 5:20
  SPEAKER_00: Um, and during my journey there, I became group leader for Aerial Mobility Group there, and and learned a ton.
• 5:28
  SPEAKER_00: As you know, Mars Helicopter did fly on Mars with Qualcomm Chip as the main processor on it.
• 5:34
  SPEAKER_00: And then I started writing grants, and I knew that there are elements in robotics and specifically in characterizing risk and uncertainty that allows these solutions to be a lot more resilient and hence deployable in the real world from complex space and Mars environment to uh a commercial setting on Earth.
• 5:58
  SPEAKER_00: And um initiated more than 10 different projects in NASA JPL, built an RD team, and two of the projects that I initiated there were latest in the series of DARPA challenges.
• 6:12
  SPEAKER_00: And and those sort of created a backbone for what we are doing today at at Field AI.
• 6:19
  SPEAKER_00: But I'll pause here, I'll tell you a little bit about the Search for Life and space and stuff we did there.
• 6:23
  SPEAKER_01: Yeah, well let's get there a little bit.
• 6:25
  SPEAKER_01: I mean, I think I assume like a lot of people here and most of the people listening to the show are familiar with DARPA challenges.
• 6:30
  SPEAKER_01: Um we've been talking about it quite a bit on the show lately because I'm I really get the sense that they're almost underappreciated for the way in which they've kind of really helped to create the modern robotics industry.
• 6:45
  SPEAKER_00: Yeah, absolutely.
• 6:46
  SPEAKER_00: I mean, these events are I would say holy grail for us AI robotics people.
• 6:51
  SPEAKER_00: Um, they're they're shaping and creating industries, right?
• 6:54
  SPEAKER_00: If you look at the first ARPA challenge back in 2004, 6, 5, 6 around self-driving cars, right, really created the entire industry, right?
• 7:04
  SPEAKER_00: Massive number, a lot of number of startups came out of those events.
• 7:08
  SPEAKER_00: And today, many of successful startup companies we see were PhD students, professors, or uh companies that were participating in that ARPA challenge.
• 7:19
  SPEAKER_00: Similarly, the DRC focused on manipulation and humanoids back in 2012-13 accelerated a lot of developments uh across the board uh uh in that category of robotics.
• 7:34
  SPEAKER_00: And uh the latest ones that we were part of were heavily focused on autonomy.
• 7:40
  SPEAKER_00: So DARPA challenges were growing from solutions that you had at Rode and GPS and handling in early days self-driving solutions in relatively more uh inform environments that you have relatively more prior information, to now settings that you're sending robots with absolute zero information about what they're gonna see and face, right?
• 8:08
  SPEAKER_00: Uh and and that was a very big push that uh pushed this state of the arts quite significantly, I would say, in the last decade.
• 8:15
  SPEAKER_00: And our story of getting there was prior to the ARPA challenge, we uh in through our space efforts, we were building the first solutions and autonomy solutions for legged platforms, walk-in platforms, to go explore Mars analog caves.
• 8:34
  SPEAKER_00: One of the most likely places in our solar system to find life are our caves because they shelter uh any potential life from cosmic rays and more stable um situations and settings for life to potentially thrive, right?
• 8:52
  SPEAKER_00: And we were exploring solutions uh with different types of mobility systems, including four-legged robots, to go into these environments on Mars and also in Titan.
• 9:02
  SPEAKER_00: We published quite a bit on these, and we did the very first deployment in human history of legged robots going several kilometers with zero human touch, no prior map, no GPS, no pre-uh defined information into these Mars analog caves.
• 9:20
  SPEAKER_00: It was very fun times from cave networks in Oregon to uh to Kentucky and so on.
• 9:26
  SPEAKER_00: And we were doing this, and at that time, this the DARPA Challenge proposal came out, call came out, which was supposed uh the name was DARPA Subterranean Challenge.
• 9:36
  SPEAKER_00: So super aligned with what we were doing.
• 9:38
  SPEAKER_00: The idea was DARPA wanted to send a team of robots into subterranean environments, industrial facilities, explore these, create digital twins, and uh do variety of different missions and operations.
• 9:53
  SPEAKER_00: Um so we were very well aligned.
• 9:55
  SPEAKER_00: We submitted a proposal, we got in one of the very few teams that DARPA funded, and uh our team grew at some point to more than 70, 80 people, and we ended up winning pre-COVID the urban phase of this competition worldwide.
• 10:12
  SPEAKER_00: And that was an incredible experience because we were putting brains and autonomy on a heterogeneous set of assets.
• 10:19
  SPEAKER_00: At the peak of our operations, we sent 11 robots, four of them with legs, four of them with wheels, and three of them flying, going into these fully unknown environments, discovering it, and and carrying out a variety of different missions as they discover the world and as they autonomously dispatch themselves.
• 10:39
  SPEAKER_00: And um, that was an incredible experience, pushed the state of the art quite a bit forward, and then we moved on to another DARPA challenge where now we put similar technologies on much larger vehicles going many, many kilometers in off-road environments, again with no map, no GPS, no roads.
• 10:57
  SPEAKER_00: And throughout these years of space missions and DARPA challenges, we started to push the boundaries of field robotics and keep deploying these solutions out there, understanding what are the corner cases, understanding and creating this appreciation of when you go to the real world, everything is gonna become off-nominal very soon.
• 11:20
  SPEAKER_00: Everything is gonna become edge case and corner case.
• 11:22
  SPEAKER_00: And that appreciation was the backbone of the idea for creating field AI, that we started realizing to actually have the most applicable and feasible solution out there, you really need to take a step back and change your architecture and theoretical frameworks because most of your theoretical uh uh frameworks break in the real world.
• 11:49
  SPEAKER_00: Things are harder, there is no Gaussian noise, there's no linear systems, there are no predefined uh uh uh paths, and that's where we um uh uh we basically started thinking of these solutions can go to millions of robots and can be commercialized and change the game for the industry.
• 12:07
  SPEAKER_01: So this is a radically different way of thinking about autonomy and navigation.
• 12:13
  SPEAKER_01: Um everybody's thinking about edge cases, right?
• 12:16
  SPEAKER_01: Everybody's thinking about long tail.
• 12:18
  SPEAKER_01: Um anyone who's familiar with self-driving knows that um, you know, these are the the edge cases are when you talk about like the 99.999%.
• 12:28
  SPEAKER_01: But it sounds like to a certain extent what you're getting at is almost like expecting the edge cases or having edge cases kind of be like essential to the development of the system.
• 12:40
  SPEAKER_01: And the edge cases themselves aren't edge cases.
• 12:43
  SPEAKER_00: That's 100% true when you're in on a structured environment.
• 12:47
  SPEAKER_00: Yeah, right.
• 12:48
  SPEAKER_00: Um and a safety critical setting and scenarios.
• 12:53
  SPEAKER_00: So you have sometimes applications where you can control the environment to a certain degree, right?
• 12:58
  SPEAKER_00: Um, you can optimize it for the robot operations.
• 13:02
  SPEAKER_00: Um then you have applications like self-driving cars that you don't optimize the world, but the world has quite a bit of a structure.
• 13:10
  SPEAKER_00: You know the number of lanes ahead of time, you know where the exit is, you have access to GPS, you know there's a limited set of objects that you're expecting to face and encounter like cars and pedestrian and sidewalk and so on.
• 13:23
  SPEAKER_00: Then you go to a construction site where every room has like as much variation as you might see in the entire self-driving car uh setting, because it's just you're dealing with objects that are even sometimes difficult to describe what it is: broken pieces and wire hanging and water dripping and scaffold half-built setup and stairs are being put and people walking around.
• 13:49
  SPEAKER_00: So it's extremely complex, or looking at just the environment around here, right?
• 13:53
  SPEAKER_00: Around us, which is a lot more structured than construction setting, but it's still like uh full of varieties from vegetation here to chairs and people walking and all sorts of variations on the terrain.
• 14:05
  SPEAKER_00: So when you are looking at the largest TAM out there, the largest market in front of robotics, where is it's defined by you basically adding robots like a human into human-purpose environments to one-to-one or one-to-many increase the capacity of people.
• 14:24
  SPEAKER_00: Suddenly you're dealing with very unstructured setting that, as you said, dealing with corner cases and edge cases is cannot be an afterthought.
• 14:34
  SPEAKER_00: You cannot say, hey, let me build something in the lab, let me build an architecture and solution that works at a certain setting, and then I will go to the real world and try to come up with the wrappers, come up with a uh safeguards, uh uh uh safety-wise for my solution.
• 14:53
  SPEAKER_00: Right?
• 14:54
  SPEAKER_00: And uh and that's the approach we are taking at field.
• 14:57
  SPEAKER_00: At field from day one, we're taking a step back and designing the architecture with that appreciation of what we are gonna face in the real world.
• 15:07
  SPEAKER_00: All of the things from variation in lighting, people walking, dust covering half of the camera, and so on, and how to build an architecture that from day one knows that in the field it's gonna deal with the edge cases, it's gonna deal with hallucination.
• 15:23
  SPEAKER_00: And that resilience is what allowing us to be able to deploy, right?
• 15:28
  SPEAKER_00: So now today we are on three continents uh hundreds of sites, doing thousands of missions, uh, and that all of that comes back to that appreciation, re-architecting the solution from day one and going for safety, going for uh resilience of the solution as the main building block of the architecture.
• 15:51
  SPEAKER_01: One of the primary conversations I'm having right now around generalization, specifically around humanoids, is uh the conversation around full generalization versus how quickly can we deploy systems.
• 16:06
  SPEAKER_01: The most pragmatic humanoid companies will tell you okay, when we're getting systems out into the world, we're gonna get them out, they're gonna do one thing, they're gonna do it well, they're gonna learn that way, that's gonna kickstart the flywheel, that that's how they're gonna learn the space.
• 16:22
  SPEAKER_01: Uh moving totes around a factory, right?
• 16:24
  SPEAKER_01: That's what a lot of them are doing right now.
• 16:26
  SPEAKER_01: It sounds like you're taking a very different approach in that to a certain extent you're kind of starting with the most difficult problem.
• 16:36
  SPEAKER_01: How long, if at all, though, does that extend the roadmap?
• 16:42
  SPEAKER_00: Yeah, um absolutely.
• 16:44
  SPEAKER_00: At field maybe high level, I would say we are as you know, a software company, right?
• 16:50
  SPEAKER_00: We build brain that is agnostic to robots, a brain for different types of embodiments, different environments, different tasks, right?
• 16:57
  SPEAKER_00: So today the brain is on large multi-ton vehicles, all the way to a small dog robots, two humanoids, and operating across a variety of different types of environments.
• 17:09
  SPEAKER_00: Um to your question, uh there are now maybe two you can look at this uh space from two lens.
• 17:20
  SPEAKER_00: One is generalization lens, or loosely making it equ equivalent to data lens, and the other one is architecture lens, which I would translate it to deployability and safety lens.
• 17:35
  SPEAKER_00: If you think of this sort of 2D axes, um you can uh come and translate the success of Chat GBT and conversational AI to the physical world and say, hey, I'm gonna repeat that approach.
• 17:53
  SPEAKER_00: I'm gonna take this large monolithic neural network and instead of words and text, I'm gonna pass images and lidar uh scans and so on, and uh basically train this to generate the next optimal action for the robot.
• 18:09
  SPEAKER_00: Yeah.
• 18:09
  SPEAKER_00: Repeating the same philosophy in robotics.
• 18:12
  SPEAKER_00: The challenges with this is this is uh massively data hungry, right?
• 18:17
  SPEAKER_00: In text and language, we required hundreds of trillions of tokens of data.
• 18:21
  SPEAKER_00: In robotics, who knows, the variation is even even higher.
• 18:25
  SPEAKER_00: And uh but but even more importantly, uh with that hundred trillions of tokens of data in conversation, you still hallucinate all the time.
• 18:35
  SPEAKER_00: So and that's acceptable because in text and language the the tolerance for risk is high.
• 18:40
  SPEAKER_00: You can fix a few words in your article.
• 18:42
  SPEAKER_00: When it comes to the real world and physical world, the tolerance is much, much lower, right?
• 18:48
  SPEAKER_00: So we uh look at field AI to this problem from a quite different lens.
• 18:54
  SPEAKER_00: We don't take a traditional vanilla transformer architecture and then uh uh to to to then come and see it as a data problem, the solution.
• 19:05
  SPEAKER_01: It's funny that you're saying traditional uh transformer because that's already like a huge.
• 19:11
  SPEAKER_01: Very new.
• 19:12
  SPEAKER_01: I mean, it's you know, because everyone's talk when we're talking about physical AI and we're and we're talking about um things like like transformers, we're talking already talking about this this huge jump over hard coding, right?
• 19:22
  SPEAKER_01: That's like a huge jump in promise over what systems can do autonomously and how systems can adapt to the world.
• 19:29
  SPEAKER_01: And you're you're already talking about something sort of separate and apart from that.
• 19:33
  SPEAKER_00: It's not I wouldn't say it's 100% apart from that, but we take that architecture and we uh say, okay, how this is a purely data-grounded architecture.
• 19:44
  SPEAKER_00: You give good data, certain amount of data, you cover certain uh set of skills, but to solve that hallucination problem fundamentally, right, we we come and we uh uh look at how you can bake physics and more importantly, how you can bake uncertainty quantification to these models so that they're not just purely grounded in data, but they are grounded in the reality of the real world.
• 20:13
  SPEAKER_00: And this is something, again, we have been doing for a long time, for last decade.
• 20:18
  SPEAKER_00: We have been pushing the state of the art and developing uh models and developing solutions that quantify uncertainty.
• 20:27
  SPEAKER_00: Now we are baking those into the belly of these models and giving them this sense of oh, this is if I'm generating an you know an output from perception to decision, associated with it, this is a confidence measure.
• 20:45
  SPEAKER_00: This is an uncertainty measure that is not just grounded in data, but also grounded in physics.
• 20:50
  SPEAKER_00: So no matter how much data you provide, there is this sort of a training wheel, as you will, uh, around a kid learning to ride a bicycle that gives you this safety around it that you understand these are the bounds and guarantees what I can do.
• 21:05
  SPEAKER_00: And if things start deviating, there are indicators that okay, I'm gonna start hallucinating.
• 21:09
  SPEAKER_00: Let me slow down, let me be safer around this, let me look at the environment and collect a little bit more data.
• 21:15
  SPEAKER_00: So that has been our approach to actually go and do a surgery on the architecture side, go a little bit deeper, create an actual version of Transformer for the physical world, rather than picking the vanilla version and think of hallucination and thinking of safety as an afterthought, right?
• 21:34
  SPEAKER_00: So we bake that from day one, and that is increasing the resilience of the system significantly.
• 21:40
  SPEAKER_00: But uh connecting it to your question uh as a side effect of it, it also reduced the level of data you require dramatically.
• 21:51
  SPEAKER_00: Because now it's not it's not the case that I'm not looking at this world and just based on raw pixels of image, want to make sense of everything.
• 22:00
  SPEAKER_00: Which requires trillions of tokens of data.
• 22:03
  SPEAKER_00: I am now providing and making the network a lot richer, and it has access to a lot of physical representations enriched by uncertainty.
• 22:12
  SPEAKER_00: That it all it needs to do is learn from where that all we have provided is, where the boundaries, from there to the real world.
• 22:20
  SPEAKER_00: And that requires orders of magnitude, less data.
• 22:23
  SPEAKER_01: Are you talking about on-device processing?
• 22:26
  SPEAKER_00: Everything on device.
• 22:26
  SPEAKER_00: Everything on device.
• 22:27
  SPEAKER_00: For us, everything, no connection to cloud, everything purely on the robot, no 5G, no cloud connection, no Wi-Fi.
• 22:34
  SPEAKER_00: And and that's that reduction in data now goes back to the philosophy you were talking about.
• 22:39
  SPEAKER_00: That for us, that flywheel starts from day one because there's that resilience.
• 22:45
  SPEAKER_00: On day one, you might be a bit slower, a bit inefficient, but you're safe.
• 22:49
  SPEAKER_00: That allows you to collect data.
• 22:50
  SPEAKER_00: You collect data, you get better.
• 22:52
  SPEAKER_00: And in fact, we fall in the camp of uh diversifying data.
• 22:57
  SPEAKER_00: We believe the best way is not just doing a tote or one single task and grow from there.
• 23:04
  SPEAKER_00: In fact, we diversify our data significantly, but through the architecture, we require, we reduce the requirement on the volume of the data significantly.
• 23:15
  SPEAKER_00: But now we have robots in construction, in energy sector deployed, in urban operations, and uh in delivery applications.
• 23:27
  SPEAKER_00: And that variation actually we see helping the solution more much more than if we had data from one specific sector or one specific task.
• 23:38
  SPEAKER_01: So are so we're effectively we're talking about the difference between building a world model, you know, including physics, but it's still necessary on top of that to collect physical data in order to train the robots to do certain things within that world.
• 23:57
  SPEAKER_00: Correct.
• 23:57
  SPEAKER_00: I mean we we do use all sources of data.
• 24:01
  SPEAKER_00: We heavily use simulation, we heavily use video data, we heavily use you know teleoperation data.
• 24:07
  SPEAKER_00: But what we see is changing the game dramatically is the fourth channel.
• 24:11
  SPEAKER_00: That is the data coming from the real world, real customer workflow collected by the robot.
• 24:17
  SPEAKER_00: And and that is exactly as you said, comes in and creates this representation of the world that is transferable across different robots, different embodiments.
• 24:27
  SPEAKER_00: But traditionally, without this breakthroughs on the in the architecture, it is hard to do because if you don't have an solution, an architecture resilient to hallucination, you fall into this chicken and egg problem.
• 24:44
  SPEAKER_00: Right?
• 24:44
  SPEAKER_00: Um, you might come back and say, Ali, you're telling me that you're deploying these in the real-world customer workflow to collect data, but you need the data to have a model to deploy, right?
• 24:55
  SPEAKER_00: These two are dependent on each other.
• 24:57
  SPEAKER_00: It's a chicken and egg problem, and that's absolutely true.
• 24:59
  SPEAKER_00: If you are doing this end-to-end monolithic solution, because that is a data-driven solution.
• 25:05
  SPEAKER_00: Everything is grounded in data.
• 25:07
  SPEAKER_00: You need to have collect a lot of data to be able to do one niche application, and it's still with a potential chance of hallucination.
• 25:15
  SPEAKER_00: And and that chicken and egg problem is is the is a vicious sort of a cycle that would uh uh uh postpone everything that you're doing, and you need to require uh you require quite a bit of data, and even then it's a hypothesis to verify.
• 25:33
  SPEAKER_00: For us, the way we get out of this chicken and egg dilemma, and we don't have that problem, is by changing the architecture piece.
• 25:41
  SPEAKER_00: And architecture comes and says, okay, here I'm encountering something that is out of distribution.
• 25:48
  SPEAKER_00: I might have never seen and not trained on it.
• 25:51
  SPEAKER_00: So my data-driven component of the solution is gonna start hallucinating.
• 25:56
  SPEAKER_00: Now the physics piece will will come in, will allow you to be safe around it, a bit maybe slower, a bit of efficient, but safe.
• 26:04
  SPEAKER_00: And that safety allows you to collect data from that element that you did not know about.
• 26:11
  SPEAKER_00: And that maximally contributes to the evolution of your model tomorrow, and you get better, you deploy more, that leads to more data that make your model better, and that is start spinning that flywheel faster and faster about every single deployment because these robots start learning from each other.
• 26:28
  SPEAKER_01: Well, when I talk to people about uh about physics, like simulation comes up a lot, the work NVIDIA is doing uh a lot uh comes up as well.
• 26:37
  SPEAKER_01: Um but primarily speaking, what's your sense of what most people are getting wrong about physics and why it's so hard to create a I guess consistent model of physics?
• 26:53
  SPEAKER_00: Um that's that's a great question.
• 26:54
  SPEAKER_00: I think when we say physics, we mean something quite different.
• 26:58
  SPEAKER_00: So when you talk about uh um physics and simulation, that means uh get uh a simulation environment, get a set of data that r uh respects physics, right?
• 27:10
  SPEAKER_00: Get the best possible data that when you're collecting, when you're simulating this uh mission, you have already baked in the physics constraint.
• 27:20
  SPEAKER_00: But it still is a data-driven solution.
• 27:23
  SPEAKER_00: You just are improving and getting the best possible data because in collecting the data, you're respecting the physics, you're baking more and more physics in simulation.
• 27:32
  SPEAKER_00: So it's a better data, but it's still a data-driven solution.
• 27:37
  SPEAKER_00: For us, that's not what we are talking about when we say physics.
• 27:40
  SPEAKER_00: We are we are not a pure data-driven solution.
• 27:45
  SPEAKER_00: For us, when the system is moving in the environment, in addition to all it has learned, there's a very heavy algorithmic piece that actually analyzes the physics right there.
• 27:58
  SPEAKER_00: And looks at the the speed of the light and sensor and light reflection and the environment and penetration of your LiDAR and the cameras uh operation and IMUs and so on, and tries to understand based on the the physics around you based on the data that you come you're you're getting in an algorithmic sense.
• 28:21
  SPEAKER_00: And you have that piece and you have the data-driven piece.
• 28:24
  SPEAKER_00: And these two pieces are doing a pretty complex handshake to understand when the data-driven pieces start hallucinating, getting into a regime that might not have experience, and when the physics parse is very confident about certain things and tell, okay, on this one, leave it to me.
• 28:40
  SPEAKER_00: I know exactly what to do.
• 28:41
  SPEAKER_00: And this again goes back to what we talked on the background because that has been the world that we have been living and leading for the last decade, right?
• 28:50
  SPEAKER_00: So for uh many years, we have been deploying these solutions from 900 feet underground in most complex unknown environments with no prior information to Mars, to Mojave Desert, to Camp Pendleton.
• 29:02
  SPEAKER_00: And on that front, there has been a lot of breakthroughs that how you analyze physics, how you characterize uncertainty, how you solve a Foker Plan Colmon growth PDE that evolves a probability distribution on your stochastic differential equation and tells you if you take the right path versus left path, what would be the uncertainties, right?
• 29:26
  SPEAKER_00: So now you're taking all of that algorithmic piece and telling the network that you don't need to learn everything from scratch.
• 29:34
  SPEAKER_00: I don't need to have hundreds of billions of parameters, and I don't need to have hundreds of trillions of data for you to uh to learn all of this from scratch.
• 29:45
  SPEAKER_00: I can give you so much of this information that you, with uh orders of magnitude less amount of data, order of magnitude less smaller size of network, you can still get there because someone else algorithmically provided those at the right places within the right uh uh uh parts of the network with the right tokenization.
• 30:05
  SPEAKER_01: Like it's determining probability and odds based on input?
• 30:09
  SPEAKER_00: Correct, correct.
• 30:10
  SPEAKER_00: You're basically enriching rather than let's say I take a photo and I get pixels of the raw measurements, right?
• 30:18
  SPEAKER_00: Rather than that, you have an algorithmic piece that, in addition to that raw data, tells you a lot about the physics of the world.
• 30:26
  SPEAKER_00: A network says, oh, okay, I have access to all this raw information, but I also have access to this output of this algorithmic piece that is telling me how the shapes of the world look like, how the geometries and elevations and traversability and localizability of the world.
• 30:43
  SPEAKER_00: And that makes the network to move a lot faster because it doesn't need to recreate all of that based on raw data.
• 30:52
  SPEAKER_00: It can.
• 30:53
  SPEAKER_00: This is the beauty of it.
• 30:54
  SPEAKER_00: Yeah, maybe very interesting aspect that actually network can do that because it has access to the raw images and raw LiDAR scans.
• 31:03
  SPEAKER_00: It can ignore everything we provide to the network and say, no, I don't need this.
• 31:09
  SPEAKER_00: It's called fallbacks.
• 31:10
  SPEAKER_00: I can I can go end-to-end and learn from raw measurements, but in reality, we see quantitatively that is not happening.
• 31:17
  SPEAKER_00: When you're providing all these physics awareness, uncertainty that is quantified about different representations of the world, network actually uses those significantly and it contributes to reduction in the loss function much faster and rapid, more rapid reduction than if you had provided purely raw measurement.
• 31:38
  SPEAKER_01: This is this is where it breaks my like completely breaks my you know humanity's major brain is I, you know, I I assume that like the light quality in a subterranean environment is gonna be very different, uh next to nothing because it's a subterranean environment.
• 31:54
  SPEAKER_01: That itself, the lack of access to the sun, is gonna have a huge impact on the light quality.
• 31:59
  SPEAKER_01: So, how does that relate to the physics in a particular environment?
• 32:04
  SPEAKER_00: Yeah, 100%.
• 32:06
  SPEAKER_00: This is um when you go to environments that are perceptually degraded, right?
• 32:11
  SPEAKER_00: You have dust and fog and smoke and variable lighting and and so on.
• 32:16
  SPEAKER_00: Um, so that's like perfect actually example for how physics come in.
• 32:21
  SPEAKER_00: Because now you have a variety of different sensors with different wavelengths, like you have thermal camera, penetrates through dust, you have LIDARs, different behavior with fog, dust, and and light.
• 32:32
  SPEAKER_00: You have cameras that it's low light, just you know, have challenges, right?
• 32:37
  SPEAKER_00: And and now you don't necessarily need a neural network to make sense of every single aspect of this.
• 32:46
  SPEAKER_00: You can come in with physics and tell what are the consensus between these sensors, what are the contrasts and correlation to create resiliency.
• 32:56
  SPEAKER_00: You can look at uh based on physics space and algorithmic outputs that hey, I'm at a setting that my lidar is starting to become degenerate because my such and such eigenvalues of matrix is telling me that LIDAR-based component of my localization is about to get degenerate, is about to fail.
• 33:17
  SPEAKER_00: And then thermal camera kicks in.
• 33:20
  SPEAKER_00: So RGB camera kicks in.
• 33:23
  SPEAKER_00: So you have uncertainty associated with these different channels, and based on those uncertainties, you know how to weight them and who comes in and who dictates what will happen.
• 33:34
  SPEAKER_00: And that you do all of that calculation, and that's just a suggestion to the network.
• 33:39
  SPEAKER_00: You still have your neural network data-driven piece that is the main component.
• 33:44
  SPEAKER_00: But now, data-driven piece, in addition to making sense from the raw data, seeing all of these uncertainties, you have calculated all of these outputs and representations you have built through different types of sensors that is trying to make sense of the world.
• 34:00
  SPEAKER_00: And neural network looks at the raw data, looks at these, and decides what how to create the output based on all of these different types of inputs.
• 34:10
  SPEAKER_00: I'm of course grossly oversimplifying here, but that helps the neural network to learn a lot faster and use the raw data to interpolate and do things that the physics parts cannot do.
• 34:25
  SPEAKER_00: But at the same time, it helps the neural network to not redo the parts that you have provided and hence lower amount of data, hence higher level of resilience and better physically grounded measures of uncertainty and confidence, which all translates to higher level of deployability in the complex environment.
• 34:47
  SPEAKER_01: So, you know, you're talking about we're talking about construction, we're talking about uh oil fields.
• 34:53
  SPEAKER_01: I know like Fukushima was really big for you, you know, uh disaster relief that you know that took um like a huge amount of time to do disaster recovery for that.
• 35:04
  SPEAKER_01: And from a very, I guess, kind of like commercial standpoint, construction, huge industry, you know, oil natural resources, huge industry, but again, very difficult environments for reasons we've discussed.
• 35:17
  SPEAKER_01: Now, the flip side of that is we were, you know, just over on the HRF stage, and everybody over there saying, Oh, you know, we're we're in warehouses and factories because the home is so unstructured, it's so hard.
• 35:30
  SPEAKER_01: These these sorts of like learnings in these kinds of environments, you know, working in like a cave or working in a in a uh a construction zone, are those sorts of instruct uh uncertainties, are those sorts of learnings things that you could even apply to like a domestic environment?
• 35:50
  SPEAKER_00: Um absolutely.
• 35:53
  SPEAKER_00: In fact, one of the reasons we are in environments like construction is not only it's a massive market,$17 trillion market, extremely hungry for for new technology to increase productivity, safety, and all of that.
• 36:09
  SPEAKER_00: But also the site changes every minute, yeah, every hour.
• 36:14
  SPEAKER_00: Yeah.
• 36:15
  SPEAKER_00: So the the richness and quality of data is unbelievably higher than any other domain we have deployed in.
• 36:21
  SPEAKER_00: When I'm I we have robots in manufacturing setting today, we have robots in energy setting, we have robots in logistics, as you mentioned, and data in all of them are very useful.
• 36:31
  SPEAKER_00: But in many cases, after sometimes after two, three minutes in assembly application or in some of maybe security or some of the logistics, after maybe hours or days or weeks, the the data is not as interesting.
• 36:48
  SPEAKER_00: Because you always are looking for what would contribute maximally to the evolution of your learning, what is out of distribution, what is new.
• 36:57
  SPEAKER_00: And in construction, the data is never boring.
• 37:00
  SPEAKER_00: Every hour uh it gets better.
• 37:02
  SPEAKER_00: And I'll give you a very concrete example.
• 37:05
  SPEAKER_00: Very first time we went to uh uh an oil and gas refinery deployment.
• 37:11
  SPEAKER_00: Right?
• 37:12
  SPEAKER_00: We had never we never had a single data from environments like that.
• 37:16
  SPEAKER_00: We had been on many, many, many different construction sites.
• 37:20
  SPEAKER_00: Um but on day one, robot had a reasonable performance.
• 37:25
  SPEAKER_00: Of course, three months after that, performance was much better because it learned more and more about that class of environments.
• 37:32
  SPEAKER_00: But the the construction setting was rich enough, it had seen so much of a variation that we're able to make enough sense of that oil and gas refinery that it has never seen before to perform reasonably well in the very first deployment, first five minutes of uh us unboxing the robot and deploying in the environment.
• 37:53
  SPEAKER_00: So I we believe uh in addition to the size of market, in addition to this being a very hard problem, which is a good solution, good problem for a startup to build a moat around, um, it is very good and rich environment because of the quality of data.
• 38:11
  SPEAKER_00: Because I don't have to hire thousands of people to go and change the environment for me or teleoperate something.
• 38:17
  SPEAKER_00: I have autonomous systems that every minute, every hour, getting massive petabytes of influx of data uh bring into the to the to the training uh set.
• 38:27
  SPEAKER_01: You know, I obviously I speak to a lot of physical AI companies, and there's a lot of companies doing really amazing things out there.
• 38:35
  SPEAKER_01: What's interesting is it seems like more and more companies that have started off as big big you know physical AI model training companies, the two that come to mind immediately are uh Rode AI who have announced that or have have told me that they are going they're going to do a humanoid, and uh Genesis who have announced a humanoid.
• 38:57
  SPEAKER_01: So like it seems like more and more of these physical AI companies are also building like a full stack, but but that seems like that's not something that you're interested in at all.
• 39:07
  SPEAKER_00: Absolutely not.
• 39:09
  SPEAKER_00: Uh I mean we I should take it back.
• 39:12
  SPEAKER_00: We don't rule that out.
• 39:13
  SPEAKER_00: At some point we might do something, but because of our touch point with customers, because of you know the scale of our operations today, we we rarely hear a customer tells us that we want that robot or this robot or or it always has is been uh is around the use case.
• 39:36
  SPEAKER_00: We want this solution to be solved, right?
• 39:38
  SPEAKER_00: And usually it's again not it's not very uh common or uh or uh frequent that you have use cases that you can put the same platform.
• 39:49
  SPEAKER_00: So I think uh the the philosophy we are taking at field is to go after the largest TAM, uh to go after these use cases and uh go and penetrate the market at the fastest pace.
• 40:06
  SPEAKER_00: We should not lock ourselves to specific uh embodiments.
• 40:10
  SPEAKER_00: Today we are on 34 different embodiments and um quite a bit of them now in the field deployed with paying customers, and these are quite different platforms from each other.
• 40:24
  SPEAKER_00: And as we see the demands are coming and growing, we see less and less convergence, more and more of a oh, I want to optimize, I want to increase my ROI, I want three hands on this robot, uh, I want one more finger, I want the wheels on a bi-manual on a wheelbase.
• 40:40
  SPEAKER_00: And we we believe locking to one uh uh uh uh model would um would reduce the opportunity.
• 40:48
  SPEAKER_00: Um and again, we don't rule it out, right?
• 40:51
  SPEAKER_00: As we are growing, and uh if there are we get to a point that there are use cases that we are convinced that you should build a robot for that specific use case at millions, we will we will go and do it.
• 41:08
  SPEAKER_00: Uh uh we believe now with the hardware commoditization and understanding of how it's done, it is it is feasible, it is doable.
• 41:19
  SPEAKER_00: Um and uh but but the philosophy we are taking at field is heavily um robot agnostic philosophy.
• 41:28
  SPEAKER_01: Turns out uh generalization is gonna require a lot of generality in the end, it sounds like.
• 41:32
  SPEAKER_01: Ali, uh thank you so much for appreciate it.
• 41:36
  SPEAKER_01: Thank you.
• 41:39
  SPEAKER_01: Thank you so much for joining us today.
• 41:40
  SPEAKER_01: Thanks to Ali and the folks at Field AI for helping lock that one down.
• 41:45
  SPEAKER_01: Thanks to you as always for tuning in.
• 41:47
  SPEAKER_01: If you've been enjoying the show, please like and subscribe wherever you get podcasts.
• 41:51
  SPEAKER_01: And don't forget to subscribe to our weekly newsletter of the same name at automated.fm.
• 41:57
  SPEAKER_01: We will see you the same time next week for another episode of Automated.