Latest / RobTalk / How Robots Learned to Walk: From Hand-Engineered Control to Reinforcement Learning

Transcript

How Robots Learned to Walk: From Hand-Engineered Control to Reinforcement Learning · RobTalk

• 0:00
  Speaker 0: reason behind the human form factor, right?
• 0:02
  Speaker 0: That we see in humanoids now is that you essentially have an example of how this works really well which as humans and you can try to imitate from them.
• 0:11
  Speaker 1: So when we had this thing with the optimization problem it's also minimizing costs but while the robot is running there are a lot code that actually feeds into these cost functions.
• 0:24
  Speaker 0: I mean major limitations were anything that the robot was supposed to handle had to be inside the plan and this big optimization which we
• 0:37
  Speaker 1: were doing online.
• 0:42
  Speaker 1: Welcome back to Rob Talk, our podcast where you talk about all things robotics!
• 0:47
  Speaker 1: Today I'm going a little bit of deep dive – We've already touched upon locomotion as an issue with dancing robots but today there is special guest who knows much more than me And so I thought it would be a good time to talk with Felix Frank, who's tech lead in our Robot Intelligence team.
• 1:09
  Speaker 1: Welcome, Felix, to Rob Talk!
• 1:11
  Speaker 0: Thank you, Clemens.
• 1:14
  Speaker 1: RobTalk the autonomous robotics podcast physical AI no theory just reality.
• 1:26
  Speaker 1: Alright and my name is Clemens.
• 1:29
  Speaker 1: So we're going right in.
• 1:32
  Speaker 1: Felix.
• 1:34
  Speaker 1: Tell us a little bit about your journey.
• 1:36
  Speaker 1: Now you're working on manipulation tasks, so dealing more with the hands or the grippers of a robot but You've come a long way and until you got into our team.
• 1:47
  Speaker 1: So how was your journey there?
• 1:49
  Speaker 0: Yeah I can maybe start like when i really get in to robotics which was two thousand seventeen With a master.
• 1:57
  Speaker 0: see this back then.
• 1:59
  Speaker 0: Back.
• 2:00
  Speaker 0: Then I Was looking at How To do certain vision-based grasps as part of a thesis at Volkswagen.
• 2:09
  Speaker 0: And then I stayed at Volkswagen for six years, i think working on different topics all the way from sim to real and a lot of reinforcement learning model based RL time series models but all in.
• 2:29
  Speaker 1: That's a lot of topics, but what was the application that you were dealing with?
• 2:34
  Speaker 1: A large car manufacturing company.
• 2:39
  Speaker 0: The group I was in at VW is mostly focused on fundamental research so we didn't have super-defined applications all the time.
• 2:49
  Speaker 0: There are more breakout projects where take some of the methods which we had developed during research and use that inside the company.
• 3:02
  Speaker 0: One example is the air pressure control, inside the VW plant where we essentially learned forecasting models for how certain compressors would work with different temperatures in different ambient conditions.
• 3:24
  Speaker 0: more energy-efficient usage of compressed air inside the plant.
• 3:28
  Speaker 1: Okay, so making factories more efficient by use data driven methods?
• 3:35
  Speaker 1: Correct
• 3:36
  Speaker 0: yeah that sums it up pretty nicely.
• 3:39
  Speaker 1: and then you moved on to working on something completely different which was lagged robotics right.
• 3:44
  Speaker 0: Yeah.
• 3:45
  Speaker 0: So basically I started last year in March kind of working for a US startup that works with humanoid robotics.
• 3:57
  Speaker 0: And there I worked all over the tech stack, but specifically my main focus was on whole body control locomotion and how to basically use the humanoid not just walk around But then also interact with environment.
• 4:13
  Speaker 1: So it spans Not only legs what the behavior should be when you want to actually do stuff.
• 4:21
  Speaker 0: Correct, yeah?
• 4:23
  Speaker 1: So let's now go a little bit deeper.
• 4:25
  Speaker 1: so last time we talked about Lek Robotics in context of for example these dancing robots.
• 4:32
  Speaker 1: but Lek robotics goes back a lot further.
• 4:37
  Speaker 1: maybe You can talk a little more on how The whole thing worked three four five six years ago.
• 4:45
  Speaker 0: Yeah, so in leg robotics there has basically been major changes.
• 4:52
  Speaker 0: In like.
• 4:52
  Speaker 0: what is the standard methodology?
• 4:55
  Speaker 0: In order to make these robots walk and dance or do other things?
• 5:00
  Speaker 0: because if you think about how everything was done ten years ago this was mostly a very hard engineering problem where he would spend A lot of time on tuning certain engineering controllers in order to just do the fundamental walking capabilities.
• 5:21
  Speaker 0: So you would have a stack, which basically consists of some sort of like footstep planner that will essentially maybe given certain terrain or may be giving us certain direction Would plan okay where do I place my feet?
• 5:40
  Speaker 0: and then based on that it would know the entire model of the robot like all the kinematics how the weights are distributed.
• 5:47
  Speaker 0: And you'd essentially planned for The center-of-mass being inside Like the feasible area, but your like plant footsteps would allow?
• 5:59
  Speaker 0: Then all this will run as one big online optimization some Yeah, so something you would basically have to optimize at up to one kilohertz and which will essentially try to balance the robot given defeat contact.
• 6:18
  Speaker 0: And give him the like trajectory demands that he
• 6:22
  Speaker 1: was tough.
• 6:23
  Speaker 1: So this is all be?
• 6:24
  Speaker 1: The idea of it can break a problem into its constituents correct then solve everything independently in an Indian put together
• 6:33
  Speaker 0: Yeah, and I mean there you would really have every small sub-problem of this.
• 6:38
  Speaker 0: Would be a certain engineering problem right that?
• 6:41
  Speaker 0: You would spend a lot of time on modeling accurately And then solving with some custom algorithm and basically combining it back together into a stack Like the sensor fusion alone like how you would essentially determine How far you have moved already.
• 6:59
  Speaker 0: That will basically require use your I am used on the robot, but then also Use the kinematic tree and you're known joint positions and joined velocities in order to get some sort of odometry signal.
• 7:13
  Speaker 0: fuse that with some calm unfilter like approach And Get essentially a velocity for your pelvis.
• 7:21
  Speaker 0: Then do some estimation.
• 7:23
  Speaker 0: how far have i gone inside this plan?
• 7:25
  Speaker 0: All these needed to work flawlessly together.
• 7:29
  Speaker 0: Any little disturbance like plan for what basically break the system.
• 7:35
  Speaker 1: So all these sensors is like IMU as we have in cell phones to determine accelerations, or how fast you turn or how fast you go.
• 7:46
  Speaker 1: So we know it, but all these sensors have noise as well and so right?
• 7:51
  Speaker 1: And then first to do filtering... To understand really what's going
• 7:56
  Speaker 0: on.
• 7:56
  Speaker 0: I mean one key thing if we want to go there is essentially You would need to know What feet are currently in contact with the ground.
• 8:04
  Speaker 0: Right!
• 8:04
  Speaker 0: If you know that your left feed for example Is now in contact With the ground Then basically from There you can assume It a stationary.
• 8:12
  Speaker 0: build the kinematic tree to your other IMU locations and then fuse it based on that.
• 8:18
  Speaker 0: But, basically also means you needed at least contact sensors in defeat And most robots actually used very expensive force sensors To like get accurate estimations of Are you slipping or are really in contact?
• 8:32
  Speaker 0: Or you're like firmly planted your feet underground.
• 8:36
  Speaker 0: Yeah I mean all this was a lot of hand engineering work.
• 8:40
  Speaker 1: Maybe just to explain a little bit.
• 8:42
  Speaker 1: So kinematic tree is how the different pieces of the robot fit together and what effect one part has on another, right?
• 8:49
  Speaker 0: Yes so I mean essentially if you like from very basic level.
• 8:55
  Speaker 0: we know that all the joints they have encoders where you can measure the angle between inside the joint.
• 9:04
  Speaker 0: And then since you know how you constructed your robot What are the locations of all the limbs, hands and everything given that you know how tree looks like?
• 9:17
  Speaker 0: How things connected to each other.
• 9:19
  Speaker 0: And then how angles in important joints.
• 9:24
  Speaker 1: So when we do it right than should get a digital twin where you have derived from sensory data.
• 9:32
  Speaker 1: Right
• 9:32
  Speaker 0: Correct
• 9:35
  Speaker 1: I mean i've seen this on others fields as well.
• 9:37
  Speaker 1: I remember, twenty years ago when you went to natural language processing conferences.
• 9:45
  Speaker 1: The problem of language processing was split up into syntax and semantics and entity recognition and machine translation And there were all these work streams that are parallel.
• 9:58
  Speaker 1: Now it's been replaced by chat GPT.
• 10:01
  Speaker 1: So what would be the limitations on this approach?
• 10:08
  Speaker 0: I mean, major limitations were essentially that anything like the robot was supposed to handle had to be inside the plan and this big optimization which we're doing online.
• 10:23
  Speaker 0: So even small things.
• 10:25
  Speaker 0: if you have an additional contact with your hand let's say open a door or something... That basically needed included in how the robot stabilizes itself with its environment.
• 10:44
  Speaker 0: Things like picking up objects, things like leaning to a wall and opening doors... All of these were very hard problems because you essentially had come-up ways on how to include extra contacts into your dynamics model compensate for the effects that these external forces basically have on your robot.
• 11:07
  Speaker 0: Because usually when you're just walking in free space, and I mean let's say... You are just on a flat surface right?
• 11:13
  Speaker 0: Then you've got very defined contact!
• 11:16
  Speaker 0: Basically there is a flat floor with two feet.
• 11:18
  Speaker 0: And those are only contacts to be dealing with.
• 11:21
  Speaker 0: When you walk it can kind of think about as controlled falling or leaning forward.
• 11:29
  Speaker 0: You're falling forward and then you're catching yourself with your feet.
• 11:32
  Speaker 0: And if doing that in the right dynamic way, Then the contacts you have with your feed they basically absorb like gravity would just push it to ground.
• 11:43
  Speaker 0: But now add another contact.
• 11:46
  Speaker 0: If we want open door We need compensate for these extra forces which propagate from hand to rest of body.
• 11:55
  Speaker 0: Humans do this very well, because we spend countless hours as children playing and trying all different sorts of things.
• 12:06
  Speaker 0: But if you look at this as an engineering problem when you try to hand tune and optimize for all these special situations online that becomes very cumbersome in challenging.
• 12:19
  Speaker 1: So the idea is there?
• 12:22
  Speaker 1: You would be somehow able to engineer walking Real in the real world where the real World is just a very Very unordered semi-structured Domain.
• 12:35
  Speaker 1: and so maybe that thinking was a little bit too simplistic.
• 12:38
  Speaker 1: then what did you actually optimize for like?
• 12:40
  Speaker 1: You said it's an optimization problem, but What kind of error would you try to minimize there?
• 12:46
  Speaker 0: So I mean The main thing you will try to minimise.
• 12:49
  Speaker 0: Would basically be the overall actions that you would have to do.
• 12:54
  Speaker 0: The bigger thing in this optimization problem is the constraints that you have.
• 12:58
  Speaker 0: Because essentially, your footstep planning and then if my feet had to be there so those are already constraints on the kinematic structure because I know okay... My left foot has been at a certain place.. My right foot was also at some point of time.
• 13:17
  Speaker 0: Okay, where does my center of mass have to be?
• 13:20
  Speaker 0: and that basically already places your body in a certain location.
• 13:24
  Speaker 0: And then if you now add manipulation on top of it.
• 13:26
  Speaker 0: You would have certain end effector target positions like wear your hands or wear your elbow Or something Like.
• 13:33
  Speaker 0: That is supposed To Be and merging all Of These Constraints Basically Leaves You With A Limited Set Of Options.
• 13:41
  Speaker 0: Then You Usually Just Distinguish Between Them By something that uses the least amount of overall torque or some constraint is also keeping your body upright, these sort things.
• 13:54
  Speaker 0: But then actual minimization isn't a key thing.
• 13:57
  Speaker 0: it's just additional variables you would basically use to figure out where do I actually want go?
• 14:08
  Speaker 1: So part of engineering comes up with right costs and design them in the right way.
• 14:15
  Speaker 1: One of them is energy minimization, but then there could be a lot other constraints?
• 14:20
  Speaker 0: Yeah I mean it would also be that you want to be as fast possible which influences how you place your feet and so on.
• 14:26
  Speaker 1: yeah So people got already quite far if we work with this problem for ten years.
• 14:34
  Speaker 1: i remember they were robots doing parkour kind of movements like five six seven years ago.
• 14:40
  Speaker 0: Yeah, I mean the Atlas robot from Boston Dynamics is kind of a really good example for how good you can actually get if your really perfect technique.
• 14:50
  Speaker 0: And it showed really impressive capabilities in terms not just walking but also running the parkour that you've mentioned and did backflips or all these sort things with lots of engineering work basically.
• 15:04
  Speaker 1: So many years off tuning and fine-tuning to get at certain level?
• 15:11
  Speaker 1: People got really to some kind of boundary there like a frontier that you know where they couldn't get better.
• 15:18
  Speaker 1: Yeah, or suddenly we had a shift.
• 15:20
  Speaker 1: So what was it?
• 15:21
  Speaker 1: Kind of shit.
• 15:22
  Speaker 0: yeah I think the.
• 15:24
  Speaker 0: As you said for language right people were looking at ways too.
• 15:29
  Speaker 0: basically instead of doing all the Like splitting up in sub problems and doing a lot of engineering on each sub problem And then combining it back to a bigger architecture The idea was Okay, how can we approach this problem more from a learning perspective?
• 15:45
  Speaker 0: So um... We essentially want to use data uh in order to yeah and then some sort of black box function approximator like a neural network right.
• 15:56
  Speaker 0: And try to solve some of these sub-problems or try It's a bit hard to pinpoint the exact time when this shifted, but I guess for quadrupeds.
• 16:17
  Speaker 0: This was a bit earlier.
• 16:18
  Speaker 0: The earliest quadrupet paper that really showed proper walking without any major engineering around it... ...was about twenty-nineteen to twenty-twenty and then humanoids came later more like twenty-three or twenty-four years ago.
• 16:38
  Speaker 0: So actually not that long ago.
• 16:39
  Speaker 0: And there were a bunch of things, where necessary in order to make this happen.
• 16:44
  Speaker 0: for humanoid specifically the availability of cheap humanoids was like super important because twenty years ago you could only do research on humanoids and a few labs That basically had built their own custom humanoid.
• 17:01
  Speaker 0: This is expensive, this one's brittle.
• 17:05
  Speaker 0: By now we have humanoids kind of as a consumer product, not that they're incredibly useful to like an average person but.
• 17:14
  Speaker 0: As a research tool yeah there are quite good mm-hmm.
• 17:18
  Speaker 1: so that was the whole cheetah development from MIT I guess?
• 17:22
  Speaker 0: Yeah Cheetah Development From MITry.
• 17:24
  Speaker 0: But then for humanoid's unitree The G one.
• 17:27
  Speaker 0: uh yeah i mean this Was Quite Important.
• 17:31
  Speaker 0: and i mean if we look in terms Of shift of the methodology The the real key thing that happened is, we got also GPU parallelizable simulations.
• 17:43
  Speaker 0: That was a bit earlier but I mean using this for humanoids or quadruplets for locomotion in general.
• 17:50
  Speaker 0: um yeah it was around that time.
• 17:53
  Speaker 0: and uh the key thing Is?
• 17:56
  Speaker 0: A lot of these learning methods.
• 17:57
  Speaker 0: they're not very data efficient.
• 17:59
  Speaker 0: Um But if you have simulators that can just produce endless amounts of data In a short amount You can basically use that to still learn good and robust behaviors.
• 18:12
  Speaker 1: Okay, then there's now two new developments right?
• 18:15
  Speaker 1: One is this repetition of human motion And Then There Is A Much More Open Ended Domain Of Actually Solving a Task.
• 18:23
  Speaker 1: So Maybe Let'S Go Again Really Quickly Into This Repetition Part of the work,
• 18:30
  Speaker 0: I would actually split it a bit finer.
• 18:32
  Speaker 0: because so if you look at locomotion in general there.
• 18:36
  Speaker 0: It wasn't purely like trying to repeat what humans are doing from the start.
• 18:43
  Speaker 0: that was really.
• 18:46
  Speaker 0: You essentially have a simulation which has your robot inside So some digital twin off of your actual hardware and then you will essentially do What we call reinforcement learning?
• 18:57
  Speaker 0: Which is trial through like trial by, yeah or learn by trial and error kind of.
• 19:05
  Speaker 0: And the human wasn't in a loop at that point in time.
• 19:09
  Speaker 0: so you would essentially have custom reward functions that would consist out of ten to fifteen terms.
• 19:17
  Speaker 0: something Do you keep your upper body upright?
• 19:25
  Speaker 0: You have to lift your feet more than x centimeters into the air.
• 19:30
  Speaker 0: Make sure that you stay close to some default pose, follow a certain velocity command and then a bunch of more terms.
• 19:39
  Speaker 0: basically all around this.
• 19:41
  Speaker 0: And here there was no external data involved.
• 19:44
  Speaker 0: so the only way that data would get in through the simulation Randomly initialize policies and they would have to try an make this.
• 19:57
  Speaker 0: Humanoid or quadruped a robot in general walk on the only signal it will get is that he had to follow certain velocity trajectory.
• 20:05
  Speaker 0: so it has to go forward backward left right whatever.
• 20:09
  Speaker 0: And based on the cost custom costs function, there's what be some sort of reward shaping problem where you essentially learn policies that then fulfill all of those rewards as good as possible.
• 20:23
  Speaker 0: And it's kind of amazing because these policies, they are really robust.
• 20:30
  Speaker 0: so one key thing is you only learned them in simulation and basically never see any sort of real robot data.
• 20:39
  Speaker 0: but there a technique which was called domain randomization where essentially Inside your simulator you vary the physics so much that Your robot has to learn or your policy as to learn a very robust behavior which basically fits To any sort of variance inside like that concerns physics.
• 21:01
  Speaker 0: So different mass distributions inside your robots, different friction parameters Also small variations in the kinematic structure or Yeah, all these sort of things that you would randomize in simulation And then the resulting policy will basically be good enough That it can be transferred to the robot zero shot.
• 21:26
  Speaker 0: So You would spend a lot of time Tuning the simulation and tuning the rewards now.
• 21:32
  Speaker 0: so the engineering problem kind of went away from.
• 21:36
  Speaker 0: We like hand tune all this algorithms for the sub problems too.
• 21:40
  Speaker 0: We tune the simulation, we tune the reward functions and then we would spend hours inside this simulator training policies.
• 21:50
  Speaker 0: And I'm getting robust behavior out which will work on the real robot.
• 21:54
  Speaker 1: So let me try to understand a difference between stuff that was talked about before.
• 21:59
  Speaker 1: so when you had this thing with optimization problem it's also minimizing costs but while the robot is running right?
• 22:08
  Speaker 1: that actually feeds into these cost functions.
• 22:12
  Speaker 1: Whereas in reinforcement learning, we try to learn a model upfront right?
• 22:17
  Speaker 1: But we do it uh...in the way that We get feedback from data.
• 22:22
  Speaker 1: In this case The data is generated through simulator.
• 22:25
  Speaker 0: That's pretty good summary.
• 22:27
  Speaker 0: So I mean the whole idea behind Learning a neural network for such problem would basically be This complicated optimization problem gets amortized into a function approximator.
• 22:40
  Speaker 0: So essentially, in the end this neural network outputs a solution to this complicated optimization problem without actually running an actual optimization online.
• 22:51
  Speaker 0: but it has tried this a lot of times before and simulation And uh...has gotten so good at that kind-of zero shot answer is good enough.
• 23:02
  Speaker 0: you can keep your system.
• 23:05
  Speaker 1: So there is no online optimization running at all anymore?
• 23:11
  Speaker 0: There are works where you then do certain online adaptations to the real world.
• 23:16
  Speaker 0: The whole line of papers, which basically goes in this direction that you learn a fairly robust model and sims And on-line you basically adapt your robot into real time.
• 23:27
  Speaker 0: But let's say the standard approach does not have any more online optimisation.
• 23:32
  Speaker 0: I mean, technically the state estimation.
• 23:36
  Speaker 0: a lot of people still relied on classical state estimations.
• 23:39
  Speaker 0: So you would have some Kaiman filter that essentially estimates the velocity off your center-of-mass and... That is also sort of an optimization running there.
• 23:49
  Speaker 0: Yeah
• 23:50
  Speaker 1: so it's still requires quite a bit of domain knowledge about how the robot actually works.
• 23:58
  Speaker 1: in terms physics We have simulators now which can just deal with a good chunk of what we know about physics.
• 24:09
  Speaker 1: It has to be Newtonian physics, so it's well-known thing.
• 24:13
  Speaker 1: and the surprising I guess is that you take such model from simulator and apply this to real world but saw how it works right?
• 24:23
  Speaker 0: The key here are techniques before domain randomization simulation as robust or like, as diverse as possible such that your policy has to learn a very robust behavior.
• 24:40
  Speaker 0: There were also works where you essentially had a lot of domain randomization in SIM and then part of your model was some sort of adaptation which essentially figured out okay.
• 24:52
  Speaker 0: what are the specific physics I'm dealing with right now?
• 24:55
  Speaker 0: And would be specialized towards.
• 24:57
  Speaker 0: But yeah, overall the key is that your simulation basically displays such a wide variety of physics which are all realistic and come down to difference in like the floor you're walking on or differences.
• 25:14
  Speaker 0: Like manufacturing for robots because they won't get the same robot twice.
• 25:21
  Speaker 0: so their policy had to be robust with all these perturbations
• 25:26
  Speaker 1: Mm-hmm.
• 25:27
  Speaker 1: I remember two lines of work there, one is that you would even also make the shape off the robot a variable of this whole thing.
• 25:35
  Speaker 1: so then The robot could actually start to look different in order to solve a particular problem.
• 25:41
  Speaker 1: And the other is said he will just throw things between the legs and really try To have the robots stumble and recover from it?
• 25:48
  Speaker 0: i mean A big part Of the randomization aspect was also external pushes.
• 25:53
  Speaker 0: So the robot would basically be trained while constantly being pushed around and shoved around.
• 25:59
  Speaker 0: And it would have to be robust for all of this, so um... This is also where comes in that if you suddenly bump into certain things or if you stumble over a small step something like your policy has seen variations by being shoved about the entire time.
• 26:18
  Speaker 0: It's good at absorbing these shocks then trying to continue.
• 26:22
  Speaker 0: nonetheless
• 26:24
  Speaker 1: If we do all Are we getting motion behavior that resembles that of a human?
• 26:30
  Speaker 0: No, okay.
• 26:31
  Speaker 0: So this is where now the human aspect comes in.
• 26:34
  Speaker 0: so there's basically two lines of like research that kind of converged at some point and The one line is what you've basically mentioned before which is imitating Yeah certain motions that are human.
• 26:48
  Speaker 0: does so the entire architecture.
• 26:52
  Speaker 0: You take a human and the human does certain motion, I don't know he walks or runs something like that.
• 26:59
  Speaker 0: And then you would use motion capture suits and essentially capture exactly how... ...the human skeleton moves while doing this sort of motion?
• 27:10
  Speaker 0: The idea is to instead using custom cusp functions which basically shape solution landscape for policy in way that you hand engineer, like you stay upright.
• 27:22
  Speaker 0: You lift your feet and this sort of stuff?
• 27:25
  Speaker 0: And instead you basically change to okay I just wanna be... Like i want to track what the human is doing!
• 27:31
  Speaker 0: I basically wanna imitate exactly the motion which was done in which it's presented by the human.
• 27:37
  Speaker 0: There are a few key things there that need to do.
• 27:41
  Speaker 0: one is called retargeting.
• 27:43
  Speaker 0: so You have a human motion and the human skeleton has, uh... The number of bones that it has.
• 27:51
  Speaker 0: But now your robot looks certainly different!
• 27:53
  Speaker 0: It's bigger or smaller?
• 27:58
  Speaker 0: And you essentially need to through the retargeting process take the emotion which the human was doing and map on an emotion Which is doable by your robot given the kinematic constraints.
• 28:11
  Speaker 0: So re-targeting is one thing.
• 28:14
  Speaker 0: And then in the second step, you would essentially learn a policy and now like the only reward for this policy will basically be okay stay as close to kinematic trajectory which is coming from retargeting.
• 28:31
  Speaker 0: Then get away with all of custom rewards tuning that we have to do.
• 28:37
  Speaker 0: it boils down to track what humans are doing.
• 28:41
  Speaker 0: so line one I guess started in probably with like deep Mimic works around that, where the first time we had these sort of reward functions which are solely based on tracking a human.
• 28:57
  Speaker 0: And then there's another line-of work which basically integrates.
• 29:01
  Speaker 0: what we talked about before is custom cost function learning Where the idea was okay.
• 29:06
  Speaker 0: now i don't want to track specific motion But I basically want to have movements which are closer to what the human is doing.
• 29:16
  Speaker 0: So there's a technique called adversarial motion priors, where you learn a discriminator essentially that tries to tell if the emotion it sees comes from humans or not and then part of the reward for like learning how to walk would essentially be fooling that discriminator.
• 29:42
  Speaker 0: So looking, you're walking a human and that way You could essentially For the first time add Human data To these sort of learned policies And they will not only have to Walk stay upright Move forward But also try and look Close what humans are doing.
• 30:06
  Speaker 1: Okay, so I mean it all boils down to always in some sense the same thing.
• 30:13
  Speaker 1: We we give it some target to learn now.
• 30:14
  Speaker 1: We're shifting that target and then we measure The difference too.
• 30:18
  Speaker 1: what?
• 30:18
  Speaker 1: We actually want to see and Then we can trickle down the errors through a very defined process with with neural nets right called back propagation.
• 30:27
  Speaker 1: Yeah, maybe then adjust the model and somehow then we see when we do it right that actually over time becomes better and better at doing that.
• 30:38
  Speaker 1: Okay, so the other development is you want to take out more of this manual tuning or pretty much remove code if they need to train such a thing.
• 31:00
  Speaker 0: This then goes to like the let's say current gold standard that we or I mean, current way.
• 31:07
  Speaker 0: That most humanoid manufacturers and also companies try to solve this which is essentially you get even more data driven And instead of imitating single motions You're basically trying to learn policies that can imitate a generic set off motion.
• 31:24
  Speaker 0: so you would capture a large data set of humans doing different locomotion modes and that can range anywhere from walking, running.
• 31:36
  Speaker 0: You also have stair-climbing, crawling or any type of moving around as human with different styles.
• 31:46
  Speaker 0: I've seen datasets where we had happy walking style and drunken walking style an injured walking style.
• 31:54
  Speaker 0: You would basically take all of this data and you will first still do the retargeting, so that kind of engineering shifted away from custom cost functions.
• 32:07
  Speaker 0: Now towards doing retargeting really well which is also a tricky problem in itself.
• 32:13
  Speaker 0: but once essentially your retargueted dataset Yeah, set of motions which your robot can do kinematically.
• 32:26
  Speaker 0: Um but you now have to learn a policy that can essentially deal with all of them in terms of yeah compensating for the physics that happened during executing those emotions.
• 32:39
  Speaker 0: so um like there was a paper from Nvidia last year called Sonic There's a bunch of other works around this.
• 32:47
  Speaker 0: That's probably the most well-known, where you really turn this into a big data problem because they collected I think on the order of seven hundred to a thousand hours off humans doing different types of locomotion behavior.
• 33:05
  Speaker 0: And then you would retarget all of these and you will essentially learn policies that have to deal with the entire variety to all different sorts of contacts, because I mean it's a very different problem if you walk where you have these like two defined contexts with your feet on the ground or if you are crouching around.
• 33:26
  Speaker 0: Where maybe be on your knees and then you're really using your entire body in contact with the environment to move around.
• 33:38
  Speaker 1: And so If we take this retargeting problem It all hints Why, even if we're not solving like robotics in particular.
• 33:51
  Speaker 1: The fact that were trying to learn as much from humans is possible.
• 33:54
  Speaker 1: it still makes sense to sort of follow the humanoid form factor right?
• 33:59
  Speaker 1: Humanoid kinematics and having arms...
• 34:02
  Speaker 0: I mean there's a big reason behind this human-form factor which you have An example of how this works really well, which is humans and you can try to imitate from them.
• 34:18
  Speaker 0: But I mean one thing to be very careful about that.
• 34:21
  Speaker 0: for locomotion This...I mean it's not solved.
• 34:26
  Speaker 0: but like we got quite far because We have the tools for recording Humans moving in-the world with motion capture systems for a while.
• 34:38
  Speaker 0: But when you now look at manipulation, it's much harder because You have very fine contacts with small objects and like Typical motion capture system wouldn't even capture these sort of forces.
• 34:51
  Speaker 0: So there they really need something else.
• 34:55
  Speaker 1: Okay let's stick to the locomotion for a little bit.
• 34:57
  Speaker 1: so One thing that what we've been talking about was mostly The robot being completely blind in depth.
• 35:07
  Speaker 1: So, how far do we get when the robot doesn't have eyes and other senses?
• 35:16
  Speaker 0: Yeah.
• 35:16
  Speaker 0: I mean essentially what you can get out of it is a really good whole body controller that basically knows It's like small variations in the terrains.
• 35:36
  Speaker 0: if there are small steps, that usually not a problem.
• 35:39
  Speaker 0: And they're also robust to certain amount of contact.
• 35:41
  Speaker 0: so it can bump into walls or yeah... They can be flat on the ground and still move around.
• 35:46
  Speaker 0: That's fine.
• 35:48
  Speaker 0: but I mean There is whole line research that this sort terrain aware motions where you would then try walk upstairs Or do other things that basically need some sort of knowledge.
• 36:04
  Speaker 0: I mean, we know that blind stair walking can kind still work.
• 36:09
  Speaker 0: Humans also do it.
• 36:11
  Speaker 0: as long the stairs are normed but breaks at a level where suddenly there's drop or you have ladder to really hit individual rungs And if you like are just off by a few centimeters, it's basically a catastrophic failure.
• 36:30
  Speaker 0: Right?
• 36:30
  Speaker 0: I mean that's where the blind policies Where they reach their limit.
• 36:35
  Speaker 1: and so now You might need to know much more about the semantics of the environment Know what a ladder is or no What stairs are yeah, and also do some kind of prediction right where we would have To be in a few seconds.
• 36:48
  Speaker 0: Yeah So i mean The current lines Of work.
• 36:51
  Speaker 0: They go along this direction.
• 36:54
  Speaker 0: you have some sort of component that is basically terrain aware.
• 36:59
  Speaker 0: And then often this is some sort laser scan off your environment, like exactly your surrounding areas and some grid.
• 37:09
  Speaker 0: how what's the Z value?
• 37:12
  Speaker 0: What's the height of specific point in space?
• 37:15
  Speaker 0: Then you have a certain component.
• 37:17
  Speaker 0: takes this terrain essentially plans ahead some motion although this planning is very implicit, it's just like the usually the sepence in some sort of latent space that both the planner component and then also this blind whole body locomotion component they have to know the same sort of space or way of talking with each other.
• 37:43
  Speaker 0: And Gets a high level command.
• 37:48
  Speaker 0: Hey, you want to be at point xy and then it knows okay in order To get there I have to walk up or down And basically generates an appropriate motion trajectory which than the low-level blind component just follows And is robust.
• 38:04
  Speaker 0: two certain perturbations.
• 38:06
  Speaker 1: So sounds like this is still a little bit in the research space.
• 38:09
  Speaker 0: yeah This is kind of edge off current research not just terrain aware, but also then your intent.
• 38:20
  Speaker 0: where like how to combine now these blind locomotion or whole body policies with a higher level intend.
• 38:30
  Speaker 0: Like you have to do a certain thing You have to interact with the environment in a certain way Or you have go In a certain place and your terrain right?
• 38:37
  Speaker 0: Like combining These two things that's kind of yeah The current edge of research.
• 38:42
  Speaker 0: I would say
• 38:43
  Speaker 1: Right And That is currently Not fully data driven, you still have to put up some engineering again like we used to do for the other parts as well.
• 38:51
  Speaker 0: I would say that we're getting closer and closer to it being Data Driven.
• 38:55
  Speaker 0: so there's more and more works on having this planner component also from data.
• 39:01
  Speaker 0: So essentially humans walking upstairs or vaulting over boxes.
• 39:08
  Speaker 0: You kind of learn, okay how do these motion trajectories look and you start to combine or you start doing this end-to-end learned.
• 39:15
  Speaker 0: But overall yeah there's still a lot open problems in that aspect.
• 39:20
  Speaker 1: So it means if I have flat surface like an N factories the alternative is to use wheeled robots Of course.
• 39:27
  Speaker 1: so maybe out your practical experience Can you tell me a bit about what it is to actually do research with these lag robots and the challenges there are?
• 39:39
  Speaker 0: Yeah, I mean in general doing research with lag robotics or like working with them Is quite challenging.
• 39:46
  Speaker 0: You need A lot To work well In order to be able to Do anything at all because if you just assume that There's no power The humanoid will basically Just collapse And fall down.
• 40:00
  Speaker 0: Whenever you're experimenting, you are usually doing it with some sort of patient carrier or something that essentially holds your robot in place when its off.
• 40:09
  Speaker 0: Or there's no power for any failure.
• 40:12
  Speaker 0: so... You need a physical harness around the robot to make sure it is safe to operate around and if you really want work on an electric robot It needs to be on battery because You want to walk around, it cannot be tethered anymore.
• 40:31
  Speaker 0: So there's a whole lot of complications that you basically add simply because in its natural form the system is not stable right?
• 40:39
  Speaker 0: If you don't do anything It just falls down and is on the ground And doesn't do any thing anymore.
• 40:45
  Speaker 1: Yeah we've seen that In some very embarrassing public presentations
• 40:48
  Speaker 0: as well.
• 40:48
  Speaker 0: We see them at recent demos.
• 40:49
  Speaker 0: actually Yes but this still happens.
• 40:51
  Speaker 0: I mean Any sort of small issue Being like The power Of the robot or being at some sort of code failure, right?
• 41:00
  Speaker 0: If you run into some critical state the robot will basically just collapse and fall to the ground.
• 41:05
  Speaker 0: Yeah
• 41:06
  Speaker 1: Okay so I mean that all points from a safety perspective.
• 41:12
  Speaker 1: obviously a fixed robot might be more stable And also wheeled robots might be much more stable than an electric robot.
• 41:21
  Speaker 0: yeah i think in general fix robot is can be tethered it's fixed in place like If it usually has certain breaks, if power goes out.
• 41:30
  Speaker 0: It just stays there.
• 41:32
  Speaker 0: but also a wheeled robot right?
• 41:34
  Speaker 0: I mean the wheel system in and of itself... The way its designed is stable so.. ...it won't fall over or collapse if theres no power.
• 41:43
  Speaker 0: So you can basically put some place turn off the power and wait whereas for humanoid You would essentially have to hang somewhere Or like put into a certain fixture that keeps it in place Mm-hmm.
• 41:56
  Speaker 1: Okay, so a lot of challenges on the leg robots also quite a bit of power consumption as well which we haven't really talked about.
• 42:05
  Speaker 1: So but I think this rounds it up The whole topic of its own.
• 42:11
  Speaker 1: thank you Felix for being on the podcast today.
• 42:15
  Speaker 0: Thank You for having me.
• 42:16
  Speaker 0: Yeah,
• 42:17
  Speaker 1: hope you enjoyed it as well.
• 42:19
  Speaker 1: And I hope yeah please share this episode if you liked or like it and subscribe.
• 42:27
  Speaker 1: We're gonna see each other hopefully soon in the next version.
• 42:30
  Speaker 1: when it says Rob talk.
• 42:34
  Speaker 1: This is Rob Talk a podcast by Robco Subscribe on Spotify Apple Podcasts
• 42:40
  Speaker 0: and
• 42:41
  Speaker 1: YouTube.
• 42:42
  Speaker 1: See You On The Factory Floor.