Latest / Strachey Lectures / Strachey Lecture: Doing for our robots what evolution did for us

Transcript

Strachey Lecture: Doing for our robots what evolution did for us · Strachey Lectures

0:00
Okay, George. Welcome to the Hillary term 2019 Strictly Lecture, organised by the Oxford Computer Science Department.
0:20
First of all, I'd like to say a huge thank you to our sponsors, Oxford Asset Management, who make these lectures possible,
0:28
make it possible for us to invite very distinguished speakers from across the world,
0:32
and this is the fourth year that they're now supporting these lectures.
0:35
So we're very grateful for that support.
0:40
And I'd also now like to welcome our speaker, Professor Leslie Pack Hellblade,
0:47
who is the Panasonic Professor of Computer Science and Engineering at the Computer Science
0:52
and Artificial Intelligence Laboratory at the Massachusetts Institute of Technology.
0:57
Okay. I hope I slept that out. Correct. You'll be wanting to hear what she has to say.
1:07
But just a very brief introduction for me to say that Cowling has made research
1:12
contributions on a huge range of areas to decision making under uncertainty,
1:17
learning and sensing with applications to robotics.
1:22
Her research has a particular focus on reinforcement learning and planning in partially observable devices and the Sky Computers and Thorp Award.
1:33
She's been elected fellow of triple-A I. And she was founder and editor in chief of the Journal of Machine Learning Research.
1:40
So I'm delighted to be able to welcome her here today to talk about doing for our robots, what nature did for us.
1:57
All right. Thank you so much. Okay, let me not reverberate.
2:00
Thank you for inviting me. I'm excited to have the chance to talk to you.
2:04
I enjoy backtalk and feedback and stuff.
2:08
So as I go along, if you want to ask a question or complain about something I've said, I'm very happy.
2:12
And also, of course, we can discuss at the end. So my goal,
2:17
my research goal for my whole life really has been to understand the computational
2:22
mechanisms that we need in order to make a really general purpose intelligent robot.
2:27
So I'm not trying to solve any particular robotics problems,
2:30
but I just want to understand the nature of intelligence and how we can put it inside a physical system that interacts with the world.
2:38
So the way I think about a robot is that it's a transducer so fundamentally.
2:44
So I should also say that I worry about the software part of a robot. So let's assume that that's some hardware specs.
2:50
Then what I want to think about is what program it is that I need to put in the head of my robot.
2:56
And fundamentally what it is, is a program has to take a history of actions and observations.
3:01
So okay to the star, some history of what it's observed and what it's done itself and decide on the next action.
3:09
And really that's the entire job of a robot engineer is to figure out what pi program pi to put in the head of the robot.
3:17
So that's what I want to talk about today and get you to help me think about it. So how should we frame that problem?
3:23
How should we think about what pie should go in the head of the robot? So.
3:28
First I want to have you think about the robot factory, right?
3:31
So I'm going to be a robot factory. I'm a robot engineer.
3:34
I'm going to pick a program PI to put ahead of the robot.
3:39
And the question is, what programs should I pick? And what I'm going to argue is that I should pick the program that works as well as
3:47
possible in expectation over the possible situations the robot might find itself in.
3:55
So if you say to me, Robot factory, I need a robot to weld a particular model of a particular car in my factory,
4:05
then I can make a very specific program to do that because I know exactly what it's going to need to do.
4:10
If, on the other hand, you say robot factory, you make a robot that come to my house and do whatever I ask for it to do.
4:16
Well, that's a different problem. It's a harder problem, but it's a problem of the same kind.
4:21
Right. So in expectation over the environments that this robot is supposed to work in, I should the program should should perform well.
4:28
And so right now, there's a lot of argumentation about the role of learning and the role of reasoning and whether programs
4:35
should learn or not and how much and I want to say that that's really there's no point in having an argument.
4:41
We just want that program. If we can find the program that works best in expectation of the environment, if the environments are very different,
4:48
it's going to have to learn something from the particular environment it finds it finds itself in.
4:53
If the environments are very similar, it won't. Okay.
4:58
So then I take my job to be the job of designing the robot factory.
5:02
So how is it that I'm going to think about finding good programs to put in the robot so that
5:07
it can perform well when it goes into the distribution world that needs to go into it.
5:11
So there's a bunch of different ways to think about it in one way,
5:14
which somehow at the moment seems kind of prevalent, is that the robot should just learn everything, right?
5:20
So I should put approximately nothing in the head of the robot.
5:22
The robot should go out and it should learn everything from its experience in the world.
5:27
Well, okay, so that's actually not even remotely sensible, right?
5:30
Would you buy a robot that didn't know anything and allow it into your house and
5:33
have it like break a bunch of stuff and try to figure out how to do things?
5:36
Right now, another strategy is to say, well,
5:41
you should just hire really smart engineers and set them at a desk and they should type in that program and that should do very well.
5:48
And that's historically maybe been the approach, but that's actually really hard for the engineers.
5:53
Most engineers don't have good access to what program they should write in computer vision.
5:57
We've learned the lesson, right? People used to try to write computer programs that would recognise faces.
6:02
Lots of very smart people tried really hard to do that and it completely failed.
6:06
What succeeded was that they could write programs that could learn to write faces.
6:09
And I'll come back to that. Another strategy could be we figure out what humans do and then just do that.
6:17
And I think that that's an interesting and important enterprise.
6:19
My own personal bet is that that might not be the quickest way to get to where I want to get through.
6:23
So people should do that. But that's maybe not what I'm doing. Or maybe we could recapitulate evolution, right?
6:30
Maybe we could just get robots to somehow evolve or learn.
6:35
Not in the particular niche that they're born into, but over some longer time.
6:38
And maybe that would get us somewhere. So I don't know.
6:42
None of these things is completely appealing, but I just want to kind of explore a little bit more.
6:47
So let's think about learning or evolution in the factory.
6:52
Right. So I said that when humans tried to write programs to recognise faces, that didn't work out very well.
7:00
But humans actually were really good at writing programs to learn to recognise faces.
7:05
And so maybe we can do something like that. Right.
7:07
So we want to come up with some strategy for behaving in the world that works well in lots of different worlds.
7:15
To somehow do that in the factory, we kind of have to replicate the variability of the domain that the robot's going to have to go into.
7:21
We have to replicate that variability in the factory so that we can test our program
7:26
so that we know if they're going to do well when we put them out in the world.
7:29
So we need we could, though, maybe we can formulate that in terms of a certain space, an objective function, some kind of a test distribution.
7:36
So this is maybe not certainly it's a well-formed approach to the problem, but there's a debate again raging right now on the machine.
7:46
Learning community has had big arguments with people recently at the Europe's conference about this.
7:52
And people want to say you should do no harm. Write that that if you build anything into your system,
7:57
into your machine learning system or robot system, if you build something in, you risk being wrong.
8:03
And if you build in a wrong thing, you've doomed your robot or your whatever it is to being suboptimal.
8:08
It can't overcome that. And that's true, although I would say that we should.
8:14
We're going to have to take the risk. Right.
8:16
Those of you who raised children probably did some mildly suboptimal things in the process, but probably mostly they came out okay.
8:24
Right. So I feel that way about robots too. Like we have to make a decision. We're the engineers, we have to build the system.
8:29
We might do things, we might build in some things that are not exactly right, but we don't have time to wait to let things be completely generic.
8:36
So you could imagine running a completely generic algorithm that just enumerated programs in order of complexity until you found a good one.
8:43
But that's crazy, right? For reinforcement learning people, you could do something roughly the same.
8:48
Okay. So what are we supposed to do? So one strategy, I don't know, it's sort of appealing would be just set up some evolutionary process.
8:55
It's going to take a really long time. We go to the beach and then eventually the problem gets sorted for us.
9:00
Yeah. Okay. So that could be good. But I'm worried that we might not be alive at the time that that thing finishes or even gets anywhere interesting.
9:09
And so I'm going to talk about a somewhat actually much more boring strategy,
9:13
which is to combine aspects of all of these things in a way that might help us get from
9:19
where we are to really interesting and flexible robots in a short amount of time.
9:25
Okay. And so the story that I want to kind of. Make tell.
9:30
Here is one where we do metal learning in the factory.
9:33
So metal learning is a fancy name. It means learning to learn.
9:37
It means in the factory we run something like a machine learning algorithm.
9:43
Research algorithm that arrives at an algorithm that has the property that when we put it in the world, it can learn effectively.
9:51
So I want my robot to be able to come in to your house and make tea.
9:55
That's what I want. It's going to have to learn. Every different house is different.
9:58
It's going to have to learn how it's organised, how you like your tea, all these things in the factory.
10:02
I would like to be able to meddle, learn how to do that. There's all kinds of interesting constraints.
10:07
I've been speaking a bit with Liz Pelkey, who knows a lot about human development, and she says that.
10:16
Human babies and probably mammals in general are are basically born with a lot of fundamental things about the world,
10:24
that there are other agents, that there are clumps of matter that cohere, that there's space that we move through.
10:31
And so I think I feel kind of licensed to build those things into my robot.
10:35
There are invariants in the worlds that we care about that we could build in theory.
10:38
Maybe I know the kinematics of my robot, and I think there are also some interesting constraints that we have to respect that don't come from
10:47
the problem that but that come from the fact that humans are actually going to engineer these systems.
10:51
So if I'm going to build a system. Even if I'm going to build the metal learning system,
10:56
it has to have some degree of modularity just because I personally can't understand the whole thing.
11:01
I have to understand the pieces and parts and put them together in some kind of a systematic way.
11:06
So these are some constraints that we can bring to bear on the problem. Okay.
11:10
So I'll tell you just some little bit of history and then I'll talk about actual technical stuff.
11:16
So my first job when I got graduated as an undergraduate in Philosophy of All Things was to work at a research lab,
11:25
and they were just building a robot. And nobody there really knew anything about robots.
11:29
And my job was to get the robot to drive down the hallway using the sonar sensors.
11:32
And the sonar senses were terrible. They didn't get very reliable returns, and I didn't know anything.
11:39
And so what I did was I wrote a program. The robot would run into the wall and I would fix the program.
11:44
The robot would run into the wall in a somewhat different way. I would fix the program.
11:47
The robot ran into the wall for weeks. I fix the program, the robot, and while I fixed the program, the robot and as the wall.
11:53
Eventually I learned about how the sensors interacted with the environment and the control system and so on.
12:01
I learned enough about that so that I could write a program that drove the robot down the hallway without running into anything.
12:06
And the lesson I took away from that was that I didn't want to be in that loop anymore.
12:12
Right. So the robot could darn well learn how to interact with the world.
12:15
And I didn't want to be in the middle of that. All right.
12:18
So I'm going to be outside designing the learning algorithm lets the robot interact with the world.
12:22
So then I reinvented reinforcement learning and I did it kind of bad and wrong and eventually got introduced to people who knew something better.
12:28
So that was good. And I made a little robot that did reinforcement learning and I actually learned something during my thesis defence,
12:34
which was kind of cool, but by like the mid nineties, which was when neural networks were cool before.
12:41
So you might read this is kind of like the third time right now the neural networks are cool.
12:46
They were the second time they were cool was kind of like in the nineties.
12:49
But in the nineties there was the same story is now everyone said, Oh, this is awesome.
12:53
We're just going to put neural networks in there and they will figure everything out.
12:58
But I just for sample complexity reasons, for some bunch of technical reasons, I don't actually think that that's possible.
13:04
Okay. So what to do instead? So I've been working over maybe about the last ten years with that colleague Thomas Perez,
13:13
who knows a lot about robot kinematics and robot planning and that kind of stuff.
13:18
I, in the intervening time, learned something about planning under uncertainty and model learning.
13:23
And we've been working together and we're taking the following approach.
13:29
So our view is that there are some basic inference algorithms.
13:34
And representations that are justifiable based on regularities in our environment and some kinds of fundamental computational sorts of facts.
13:45
Convolution is a great example. Right.
13:47
So the same people who like to say that it's a terrible idea to build things into their networks, do convolutions.
13:53
But if they're doing convolutions, they're building something into their network,
13:56
which is an understanding of some translation invariance or some local spatial regularities and stuff like that.
14:02
So they're building a lot of knowledge. And I think that there is a handful more mechanisms like that.
14:07
I am not going to argue that this is the right particular set of mechanisms, but I would argue that there are there is some set of mechanisms,
14:14
and I'm hoping it's like six or ten and not 700, because otherwise it's going to be hard.
14:20
So we're building in some basic principles and mechanisms.
14:24
And what we were doing until very recently was actually hand building a system.
14:30
And so we're learning people hand building a system.
14:33
Well, it was to get an understanding of the whole arc of the system,
14:40
of how you could go from perception through estimation and reasoning and action and so on, to build the system that was pretty, pretty competent.
14:48
So what I'm going to do is tell you about that. And then I'll tell you about how we're adding learning.
14:56
So this is a this is a photo I like to show in all my talks lately.
15:01
It is not my kitchen, I promise.
15:05
But imagine what it would take to make breakfast there or clean it up and think about what?
15:13
What makes it so. That's hard. That seems like a hard problem for a robot and what makes it hard.
15:17
So there's a bunch of reasons that make it hard.
15:21
One of them is that it's in a sense, in a very high dimensional space.
15:25
Right. Robot people like to talk about how many degrees of freedom their robot has, you know, six or ten or 20 or something.
15:30
But how many degrees of freedom does the kitchen have? You can't count them.
15:36
It's not just the positions and orientations of all the objects. It's like whether the grapes are mushy and when the people are coming home, right.
15:41
So that all this stuff very hard to even think about the state space of that kitchen.
15:47
The horizon is really long.
15:49
If you imagine how many like little linearly interpolated motions the robot might have to make to clean that kitchen, that's really a lot.
15:56
And the uncertainty is fundamental. So, again, there's.
16:00
If you talk at a robotics conference, often people will say, but don't don't worry about uncertainty.
16:05
We'll just make the sensors better and then you won't have to worry about it.
16:08
And for some kinds of uncertainty, it's true that making the sensors better will cure the problem.
16:12
But making the sensors better won't let me know what's inside the cupboard or your head.
16:18
Right. So there's uncertainty. That's just that's kind of very fundamental.
16:21
And I can maybe get information about these things, but it requires careful, explicit action to do that.
16:29
Okay. So we have a kind of an architecture with boxes and arrows and it's not very
16:33
surprising or different from other people's architectures with boxes and arrows.
16:36
But I'll tell you what I think are some of the salient points of how we address this problem and show you a demonstration.
16:42
Okay. So we have this thing called live space hierarchical planning in the now.
16:47
Fundamental to it is the idea of reasoning and belief space.
16:52
So everyone who's ever had an atomic theory course or did way back in the day or
16:56
something knows that you can basically take any machine apart into these two pieces,
16:59
one of which remembers something about the history and the other part that decides what to do based on what it's remembered.
17:06
So we'll call the state estimation and action selection. And for us, the arc that goes between the boxes is BUI,
17:12
if you can think about it as a probability distribution over the possible states of the external world.
17:19
So the first box is trying to estimate what's going on,
17:21
and the second box has the job of taking that belief that distribution over the state of the world and deciding what action to do.
17:31
So that's the space we kind of live in.
17:35
So the first question is what what goes along that wire?
17:38
And if you've read papers about common filters or dips or something.
17:44
There's been talk about state representation and so on.
17:48
But if you think about a robot that has to clean the kitchen, it's state representation can't be like some lovely little vector.
17:54
It's a very complicated story. So first of all, we don't know in advance how many objects there are or what they are.
18:02
It's not like we can say, Oh, I have a ten dimensional state space, so we have an open world.
18:07
We keep. And again, I don't want to argue that this is the one true way.
18:10
I just want to tell you a thing that kind of works.
18:14
So we keep a kind of a database of objects that we believe exist in the world and some distribution over their properties,
18:22
like their relative positions in their mass and so on. We keep a representation of what space we believe is free and what space we believe is occupied
18:31
because we have to reason about whether it's safe to go somewhere that we haven't looked at yet.
18:36
We keep distributions about what kinds of objects tend to occur near other ones so that we might search more efficiently in certain kinds of places.
18:43
So this is our belief, something complicated. We have also to worry about an integration that very few people worry about.
18:52
Lots of people worry about robot motion planning. Just how do I move the robot from one pose to another one?
18:57
And that's a non-trivial problem. And there's a million algorithms and AI people worry about high level symbolic actions.
19:03
What should I do and in what order? At some very abstract level. What's interesting is that these things can't be isolated from one another,
19:11
that the geometry can actually completely affect what high level actions you should do and in what order.
19:19
The geometry might tell me whether I can drive my car down a certain alley or not,
19:24
or whether I can fit in a certain place, or whether I can put these two pans on the stove at the same time.
19:29
So we spend a lot of time worrying about reasoning about the interaction between discrete things and continuous ones.
19:37
Okay. So probability geometry, discrete stuff, the planning problem, the optimal planning problem in our domain is like unthinkably difficult.
19:49
So we kind of fall back on ideas from control theory.
19:53
So in control theory, an important idea is feedback.
19:57
And the important thing about feedback is that you could do a slightly wrong action.
20:01
And if you're just very quickly look to see what happens, you can decide to do something else instead.
20:06
That might make up for the slightly wrong thing that you did before.
20:08
So you don't have to pick optimal controls, you just have to pick like not terrible controls.
20:12
I'm all about being not terrible. Optimal is not in it, right?
20:16
There's no way. So I've made peace with not being optimal.
20:21
So here's our strategy. Our strategy is we make it really weak, really very approximate model of the dynamics of our world.
20:31
And do planning. Right. So here's my beliefs. I have a distribution over the state of the world and make some kind of plan.
20:39
I'm going to take the first step of that plan executed in the world, get an observation to see what happened.
20:45
And I'll take my belief and plan again.
20:48
And an important thing to think about when you see this system is the perspective that you take when you're making decisions.
20:57
So the spec, the perspective that we take, I don't know if you can see that kind of grey,
21:01
you can a little bit from the planner's perspective, it's interacting with an environment, but its environment includes the belief update.
21:13
So it's a control system that operates in belief space.
21:17
We give it its objectives and belief space, right? I tell the robot,
21:20
I would like you to believe with high probability that the kitchen is clean or that the green boxes on the left hand side of the table or something.
21:27
I can't give the robot goals in state space because the robot doesn't have access to state space.
21:33
It can't promise me that it can change the world in a certain way, but I can ask it to come to believe something now.
21:40
And it's not allowed to just like delude itself and just say, Oh yeah, I believe it, no problem.
21:45
It has to actually do the work and run the bays update and so on so that it really does believe this thing.
21:49
So I ask it to come to believe something and when it chooses actions,
21:53
it has to think about not just the effect of the effects of the actions on the state of the world,
21:58
but actually the effects of the actions on its own belief. And that's why it looks right.
22:03
You look not to change the world, but to change your belief.
22:07
And what's nice is that you can treat all your actions.
22:10
Actions that gather information. Actions that change the state of the world. Actions that do both things at once.
22:15
Which really most do all in the same framework. So that's a lesson from DPS, but it applies here too.
22:23
So we think about planning and building space. We think about planning to take actions that will control our own state of information about the world.
22:32
Okay. One more kind of high level idea. I promise I'll get more even more technical in a minute.
22:36
But I just kind of want to give you the story here.
22:39
Another kind of high level idea here is, okay, so planning is difficult and inefficient and exponential on the horizon usually.
22:46
And so we can't stand to have a very long horizon. And if we thought about how many actions it takes to clean that kitchen, the horizon is horrible.
22:54
So we do something hierarchical. And people have talked about hierarchical planning a lot.
23:00
Usually when they talk about hierarchical planning, they use the idea of hierarchy to make planning for a completely worked out plan more efficient.
23:11
But we're going to do something a lot more aggressive than that.
23:14
So what we're going to do is we start maybe with some high level goal and we make a plan at some level of abstraction.
23:20
So, for instance,
23:21
I made a plan to come to Oxford involved going to the Boston airport and flying around and walking through Heathrow and doing some things like that.
23:28
But I mean, it is a pretty high level of abstraction.
23:31
Now, partly I did that because I'm kind of computationally lazy and I figured I could work out the rest later.
23:37
But partly also I did it because I didn't have enough information. I couldn't have planned my trajectory through Heathrow because I didn't know
23:45
what the map was like or what gate we would come into or any of that stuff.
23:48
So we make a high level of abstraction and then we kind of commit to the first step.
23:52
So when we make a plan, I should say the little that you can think of the yellow boxes and as abstract actions,
23:59
you can think of the blue boxes as what we call primitives.
24:02
They're like sets of states that we have to go through kind of sub goals.
24:07
So we take the first signal and we make a plan for that.
24:10
So get to the Boston airport and then we take the first goal of that and we make a plan for that.
24:16
Like, I don't know, get an Uber. And so finally I get down to some primitive action and I execute it.
24:22
So I'm being optimistic that I can work out the rest of the stuff later.
24:27
And eventually right now, we're hand building these models.
24:29
Eventually, I'm going to have to learn, for instance, to predict whether it's reasonable to walk through Heathrow in 20 minutes.
24:36
I don't know. You can tell me whether that's reasonable or not. Okay.
24:40
So we have this hierarchical planning thing. It also lets us remember I said whenever we took an action we re planned.
24:47
The fact is that if we have this structure, we can decide whether we need to replan very efficiently.
24:54
We can ask the question, I just took this action. Did it did it lead to a blue box that I was expecting?
24:59
If it did, I can do the next one. If it didn't, I can pop that low level thing off the stack.
25:04
So let's say I was planning to take an Uber to the Boston airport, but I discovered that there aren't any.
25:09
Okay, so I pop my bottom plan. I don't give up the idea of going to the airport or the idea of coming to Oxford or my academic career or, you know.
25:20
Right. You know, people possibly who reason too much at the high level, but it's not healthy.
25:25
A little bit is okay, but too much is not. So you'd like to control your reasoning in the structure lets you do that.
25:30
So that's also kind of a nice thing. Okay, we put this together, we get the robot to do some stuff.
25:36
So I'm just going to show you some movies and kind of talk while it does.
25:38
What's important about these movies is that the robot is doing different kinds of things subject to different goals, and it's all the same code.
25:46
In this case, we asked it to put the green block on the corner.
25:50
The Green BLOCK is too big to pick up, so it has to push it. It reasoned that it had to take the orange one out of the way.
25:55
It also knows that pushing is really unreliable, and so every time it pushes it, it looks to see where it went and says, that wasn't so good.
26:01
I better, you know, replan it, pushing the same code again.
26:04
We asked the robot to go out of the room. It knows just it knows about space and it knows about occlusion and obstacles.
26:12
And it says if there's something in my way, I can't go through it.
26:16
First. It says, if I want to move for some space, I have to look at it and believe that it's free.
26:20
If I look at it and find something in the way, I have to move it out of the way.
26:23
So it's reasoning very generally, doing all these same kinds of things.
26:29
Before it was picking up the oil model to see if there was oil in it. Okay, so this was good and bad.
26:37
Oh, this is a crazy robot in Singapore. Okay, well, whatever.
26:41
A little too much similarity on that one. All in the same row, all the same code.
26:47
But so that was good. Kind of general purpose. Kind of general purpose, reasoning, planning, estimation, and so on,
26:52
but kind of not so good because there's no learning or add activity in there at all.
26:58
There's no learning in the moment and there's no learning in the factory.
27:01
This is basically me and my writing code. So not an extensible strategy for making general purpose robots in a factory.
27:08
So the question now is how can we think about making a system like this learn?
27:16
Okay. So in particular, we want to think about what kinds of things can we learn and how to do it.
27:24
And so there's there's kind of an interesting distinction.
27:28
I really think that there's two importantly different kinds of learning, which, again,
27:32
people used to talk about a lot back in the day, but I think they tend to model them now a little bit.
27:39
So a classic kind of learning is to learn about the world, right?
27:44
So I'm a robot. I don't know what happens if I push this button, so I'm going to try it in your kitchen and see what happens or I'm not sure.
27:51
So there's some things about how the world works that I don't understand.
27:54
And I have to do things to gather information to figure out how the how does the world work.
27:58
I might learn observation models, how to my sensors. Tell me about the world transition models.
28:02
What happens if I do this? Most of the work right now in robot learning is focussed on these two lower boxes, right?
28:08
So one thing is object detection, which has been huge, right?
28:11
So various kinds of perception has been very important to us and also primitive policies,
28:16
strategies for picking things up or reorienting something in my hand or riding a bicycle or walking.
28:22
Right. So those are all very important kind of you can think of them as sort of closed sensory motor loops.
28:28
So this is this is one kind of learning which is really gathering information about the world that time.
28:34
There's another kind of learning which is at least as important, I think, which is learning to reason more efficiently or effectively.
28:42
And I would argue that learning to play chess or go or Starcraft, is that right?
28:47
Especially chess and go. Maybe not the Starcraft thing just to go to figure out, you know, the rules.
28:51
Once you read the rules information, theoretically you were capable of making an optimal first move.
28:58
It's just that you're too dumb to compute that right.
29:01
If you just had a better computer, you wouldn't have to learn any new information.
29:06
So you have to put the information in a different form, but you've got the information.
29:10
And so there are lots of opportunities within a kind of an estimation and planning and
29:14
reasoning architecture like the one I told you about to also do that kind of learning.
29:19
And so we're thinking about what kinds well,
29:22
I'm going to do now is talk now concretely about some particular thread of work
29:27
that's going on right now in my group involving learning transition models.
29:32
And a little bit about learning sampling for planning because when you plan in continuous domains,
29:38
you have to do something to manage the the continuity, the continuum of the possible actions that you can take.
29:46
Okay. So old story. Lots of models in the world.
29:49
They're all wrong. Some are useful. And useful for what?
29:53
So we have to think about if we're going to learn a model of the actions the robot can take.
29:56
What kind of a model could we learn? I had a post-doc named George Kennedy Ross, who thought,
30:02
I think very nicely and usefully about the continuum and abstraction in learning models for robots.
30:09
He likes to talk about the swamp. You can imagine there's some system of partial differential equations that really governs how the world works.
30:16
But that's and you could imagine that being a very accurate model, but not a model that you could plan with easily.
30:23
AP When people think about these beautiful, totally abstracted,
30:27
wonderful symbolic rules of symbolic modelling and so on, and they're lovely and you can play with them efficiently.
30:35
And then the question is, well, how do you make what kind of connection can you make?
30:40
Right? So what we need to do is think about how can we abstract chunks of swamp into into kind of nice abstract symbols up at the high level.
30:49
So I think we need both levels. And the view that we're taking right now is that we have local control loops that kind of operate in the swamp.
30:56
So picking things up and moving things in your hands and walking that all kind of the swamp level stuff.
31:01
It gets you from this blob of the swamp to that blob so you can do some stuff like that.
31:06
But then what we want to do is learn models of how those low level control loops work.
31:13
And those models don't have to be perfect. They just have to be kind of like good enough.
31:17
We hope to be able to abstract over objects, right?
31:19
So I don't want to have to learn a model of the dynamics of this particular prob, but I know still if I let go of it, it will dry up.
31:28
I know what'll happen if I throw it. I know a bunch of things about it, so I know some, some things abstractly.
31:35
And what I hope is that I can get some kind of virtuous category diagram thing
31:39
going on here so that at the high level there are arrows that are maybe nearly
31:44
deterministic that move me from sets of states at the high level to other sets
31:49
of states which are really embodied in these kind of swampy control loops.
31:54
So. Actually Skip that's permanent.
31:59
So what we're going to try to do what I'm going to talk about now in detail.
32:05
Is learning a model of the preconditions and effects of a low level control.
32:11
So imagine someone has learned an awesome policy for picking something up or stirring something or pouring liquid.
32:19
And now I want to learn and abstracted model of it so that I can do planning.
32:24
That's what what we're up to here and what we've learned recently in the world of planning for mixed,
32:33
continuous and discrete systems is that is that we can get really great leverage on these problems if they're articulated.
32:41
If the models that we have were articulated in a certain way and a really important aspect is that they
32:47
be factored that we talk about the state of the world not as an under analysed thing like State 94,
32:53
but that we describe it in terms of state variables. And the state variables have values that we can change.
32:59
It's also true that that constraints are a very useful language for describing the effects of actions in the world,
33:06
especially geometric kinds of ways. So we're going to we're going to look for models that are articulate, articulable in this kind of style.
33:15
Okay. So how can a competent robot acquire a new ability, assuming that my robot already knows how to pick things up, how to move around.
33:22
But we've learned to do a new thing, like pouring or stirring, and we want to add it to the robot's repertoire.
33:28
So that's what we're up to. So here's an example. Here's pouring in two dimensions.
33:36
I can describe the situation using a bunch of continuous valued variables,
33:41
things like the size of the aperture of the cup that I'm pouring out of the size of
33:45
the thing that I'm pouring into the relative pose of the centres of the two things,
33:51
the way in which the robot is grasping the cup. You could imagine some game parameter in the controller for doing the pouring.
33:58
You could imagine conditioning on the viscosity of the stuff in there.
34:02
We're not doing that, but you could imagine that. So there's a bunch of parameters that govern the situation.
34:08
And what I'm interested in understanding is under what conditions will my pouring operator actually work?
34:15
Pretty well, that is to say, get the stuff into the target.
34:19
That's what I would like to do.
34:21
So one way to think about it is that we could write a kind of symbolic ish looking description of this operation of pouring.
34:31
But it has a bunch of continuous parameters, right? The grasp and the sizes of things and the relative pose and so on.
34:38
And what I want to learn is a constraint. A constraint on the values of those continuous variables.
34:46
But has the property that if it's true and I execute the action under these circumstances, then the goal will probably be satisfied.
34:55
The effect will probably happen. But if it's not true, then probably not.
35:01
So that's that's the thing that that I want to try to learn when will this operation have the desired outcome?
35:08
And because I actually would like to do this with an actual robot, I would like for it to not take too many samples.
35:13
So I'm going to be serious about that, too. Okay. So one way we can think about this problem is as a regression problem.
35:22
So we could say, instead of saying, well, my constraint is either satisfied or not, I might say it satisfied or not to some degree I can.
35:32
You could imagine making a score of pouring. For pouring. It's easy. We just measure the number of particles that end up in the in the target place.
35:38
And that's kind of a score. And we say, oh, we would like the scoring function to be higher than some value.
35:44
We'll just call that value zero for now. It doesn't matter some constant.
35:49
And so then what I'm going to do is I'm going to do some experiments and I want to
35:53
learn the mapping from the values of all those continuous variables into the score.
35:58
Right. So and if I can learn that and if I have the scoring function, then I know for any assignment of values to those variables,
36:04
the likelihood that the or the, you know, the amount of liquid that I expect to fall into the cup.
36:11
We can formulate this as a bunch of different kinds of regression strategies.
36:14
We're going to use a Gaussian process regression for probably some of your experts.
36:19
And this is some of you don't know what it means, so I'll try to talk to everyone.
36:23
It's a way of of articulating our own uncertainty about this mapping so that we can do experiments effectively.
36:30
So what we do is we do some set of initial outpourings and we get some for each.
36:35
Each time we try it, we have some assignment of values to those variables, right?
36:39
We try it with different sized caps and different relative positions and different games of the controller and so on.
36:44
And for each one of those we get a score. Okay, so the way it goes in process works.
36:50
You think along the x axis here I can only do one dimension that's really in a lot of dimensions, but I can only do one.
36:56
Those are the the the parameters were very.
37:00
And we're interested in knowing when is this g function bigger than zero?
37:05
That's what we would really like to know.
37:07
So whenever we do an experiment, so that's one of these little blue x's, we get an observation of that function and we can,
37:17
using some Bayesian reasoning, compute a function that is sort of the mean and posterior mean.
37:23
We have a distribution over the actual function. And every time we get an observation, we can compute a distribution on the function.
37:29
So the dark red line is the mean of that distribution over functions.
37:33
And the pink area is unlike standard deviation, couple of standard deviations in the other dimension.
37:40
Now lots of people will use Gaussian processes to do a lot of different things.
37:43
Often people think about it. They're interested in finding the optimum of the function.
37:47
We're interested in something else. We're interested in.
37:52
The level set we're interested in knowing what ranges of theta is is g above zero
38:01
for what ways I can do pouring like I don't want to learn pouring in just one way.
38:05
I'll explain why in a minute. But for what arrangements of pouring is it going to work out and for what arrangements will it not?
38:12
So this block area there that I've drawn in,
38:14
that particular figure is right now the region of theta space where I believe with high probability pouring will work.
38:27
Can you be sure that that makes sense here? Right.
38:31
So for right now, there's just this little region where I'm convinced it's going to work well with high probability greater than 0.95 or something.
38:39
So but now what I would like to do is some active experimentation to try to understand the boundaries of that level set.
38:47
I would like to know well what other configurations will give me good pouring and which ones won't.
38:54
And so we use an algorithm called the straddle algorithm, which is pretty interesting.
38:57
I'm not going to go over it in detail, but it has this notion of an acquisition function.
39:01
So for different theta is it tells us which values of theta will give us the most information,
39:06
not about the maximum, but about the boundaries of the level set.
39:11
I want to know the boundaries of successful pouring.
39:15
So this particular acquisition function likes to try experiments in places where the mean is near zero, right?
39:22
Because that's we're probably near a boundary of good versus bad and where the standard deviation is higher and it combines this in a suitable way.
39:30
And what that means is that we can take a small number of samples and update our belief about this function.
39:38
Okay. So what we find is that this kind of active learning is is data efficient.
39:43
So if we try experiments at random, it takes really a lot of experiments before we get a good idea of the of that super level set.
39:52
If we try this other approach, I have to tell you just a tiny story, because the first paper we did about this was, again, just me and Thomas,
39:58
and we did it using some kind of feedforward neural network because we thought it would sound cool and it kind of worked, but not very well.
40:05
And but mostly what it did is it enraged our students. So they did a better job.
40:09
So students did a better job. The Gaussian processes, this red thing, it's awesome.
40:13
So with not too many trials. No, this iteration here is how many experiments we had to do.
40:20
It's not at times ten to the third, it's just like ten or 20 or 30.
40:24
So we learned something from experimentation. There's another piece of the story which I'm actually going to skip and I'm going to show you.
40:33
So first in simulation and then in the real robot, what again we do is we take something that has the ability already.
40:41
It already understands picking up objects and putting them down. It learns pre images for pouring in this case and stirring.
40:50
We asked it to make a cup of coffee. To make a cup of coffee.
40:53
There has to be cream in there. There has to be sugar. There has to be coffee. It has to be mixed.
40:58
It has to be on the green thing and it has to be served at the end of the table.
41:01
That's the thing. We do not tell her what steps to do or in what order.
41:04
So it's using a general purpose planner to do that. And it uses these learn it uses these learned to print images.
41:12
To to kind of get new descriptions of these operations of stirring and pouring and scooping, and it puts them together to make these plans.
41:19
We have to watch it stirring because it's fun. There we go.
41:23
I like this nice little simulator. More fun.
41:26
Is this okay? So here's our robot doing basically the same thing.
41:30
In this case. We just learned the pouring and the pushing it already picking up.
41:35
What's interesting about this is that the goal varies. That is to say which.
41:40
Yeah, I know. So you can come in fixed my motion planner if you want to, or you could just giggle.
41:48
But so we give it objectives. We move the objects around.
41:51
We ask you to do different things at different times. It kind of does it.
41:55
It's not a thing of beauty, but it's actually reasonably reliable.
41:59
I'll show you some outtakes at the very end. This time we told you that we wanted the thing on top of the block and the stuff in it.
42:06
And I guess I would argue that as we make these kinds of scenarios more complicated and the goals more complicated, look there,
42:14
it pushed the ball so that it was in the usable workspace so that compare with the other hands that was like mildly clever and did it.
42:24
Now not off the table as we make these scenarios more and more complicated, it seems, at least to me,
42:31
with my limited imagination, harder and harder to just straight up learn a policy to do this.
42:36
And it seems to me that some kind of planning is actually kind of important to the process.
42:42
Okay. Oh, last one. Okay, Guru. Right.
42:46
Um, let's see. 10 minutes. Okay. This is good.
42:51
So good. So what did I talked about there? Learning.
42:56
Assuming we kind of had the framework for a description of the effects of an action.
43:01
And now I want to talk about how we can actually learn the framework.
43:07
Right. So in that case, I, I said which aspects of the domain were important to making that prediction?
43:14
I said, the sizes and shapes of the cups mattered. I said, the gain and the controller matter.
43:18
I said all that stuff. And you just had to learn that constraint on a fixed dimensional kind of problem.
43:25
But that's not a reasonable setup, really. If I'm trying to put myself out of a job a little bit as the person who writes all this stuff down.
43:33
So another thing that seems to be important is deciding which objects and which properties of those objects
43:39
both affect the success of doing an operation and which might be actually changed by doing that operation.
43:48
So I had some old work that did this in a kind of a logical framework.
43:52
And so what we tried to do recently was recast that in a more hip, new neural network way.
43:58
I don't know if it's better. I actually think it probably is. But yeah.
44:02
Okay. So the idea. Let me just skip forward here.
44:08
Okay. The idea here is. We have a representation of the state of the world.
44:15
And, uh, but, but it's going to have different size in different instances of the problem.
44:22
Right. So most setups for neural network learning and, and functional approximation and so on assume some kind of fixed dimensional representation.
44:32
And when they don't, then they feed things in sequentially. Recently there's been work on something called graph neural networks,
44:38
which makes me laugh because it's kind of like marked off random fields, which is an old idea.
44:43
So this is a new name for an old idea, but it's a good idea,
44:46
which is that you can learn something about local relationships among objects or properties or values in your model and propagate those values.
44:54
And you can learn those local models in a way that makes them independent of the arity of the problem that you'll face today.
45:01
Right. So today I have to clear two things off the table. Tomorrow I have to clear 20.
45:06
But I hope that the models that I learn about how to do that will transfer automatically, will work independent of the size of my problem.
45:15
So in any given problem instance, I might have a representation right at the moment of my current belief about the world.
45:22
In this case, imagine that it's not even uncertain, although it could be.
45:26
So I right now I know about some objects, and for each object I know about some properties.
45:30
And what I'm interested in doing is learning a transition model.
45:33
That is to say, what will happen if I do this action right.
45:37
So one way to think about it is that it will depend on some properties of some objects in the
45:44
current state and it will affect some properties of some objects in the resulting state.
45:50
And what I'm going to focus on now is just telling you a little bit of a story about how we can find a model that has the sparseness property.
46:01
Okay. So there's this, again, kind of old idea from I, which also comes from natural language and the notion of didactic reference.
46:09
So [INAUDIBLE] is, I guess, in Greek means pointing to so this remote or that chair or the water bottle on the lectern.
46:18
Those are all TikTok references. Okay.
46:23
So what we want to do is decide which objects and which properties of objects are relevant to making the predictions that we want to make.
46:34
And so what we're going to do is we're going to start out and say, I know there's at least one object in the world that's important.
46:39
Let's think about pushing an object.
46:41
So if I want to learn the model of what happens when I try to push an object, then I know which object I'm pushing.
46:47
Okay. That's good. But there may be other objects that matter or that are affected by doing this action.
46:55
And I'm going to refer to those objects using dynamic references in this work.
46:59
We have some fixed set of dyadic references which we should increase and learn, but for right now they're fixed.
47:05
I can talk about an object above this object or below it, near the nearest object and so on.
47:09
So there are some relations which you can think of when applied to this object well after some other set of objects.
47:17
The set might be empty. The set might have one object, the set might have many objects.
47:21
But whatever their way of talking about other objects in relation to one object I already know, and then you can apply that kind of recursively.
47:29
So imagine that I wanted to talk about pushing an object.
47:33
We'll call that object object one and I can say LED object to the way to read that stuff that right
47:39
there is LED object to be the object or possibly the set of objects that's above object one.
47:45
And let object three be the set of objects that's above object to and so on.
47:50
So if I had a scene like the one on the right.
47:54
I could abstracted is a graph of relations. And then I could say, well, if a is object one.
48:02
Then these other objects. These particular objects in my particular world play the roles of object to an object to an object for.
48:13
All right. So I'm going to use this mechanism to have a flexible representation of an object I'm operating on and the
48:22
other objects that are relevant to it in a way that applies no matter how many objects are in my scene.
48:29
Okay. So if we have a set of these didactic references which reach out and name some other objects relative to the object I'm operating on then.
48:40
And we maybe we figure out which properties of those objects are important.
48:43
Then we have a kind of a straight up neural network learning problem, right?
48:48
Which is now we have a fixed dimensional input. We have this object and it's other objects that are relevant and some properties.
48:55
And we map into some properties of that object and some other objects.
48:59
So that's plain old numeric regression problem. We know how to train a neural network if we know what data to give it.
49:07
Okay. And so and then you can apply it, right? What's important about this thing is that it's meant to it's supposed to apply anywhere, right?
49:15
It says if these guys have these properties on the endpoint, this is the properties that the other guys will have in the output.
49:21
Okay. And so then we have an algorithm for learning this, and I am not going to go into detail, but it takes a kind of a normal training set.
49:28
A state is an arrangement of all the objects in the world. I take an action, a resulting sentence, then I get a resulting state,
49:34
and I have an outer loop that kind of greedily operates on the structures of the rules and an
49:42
inner loop that does some M stuff and an inner inner loop that does neural network train.
49:50
Okay. And so what happens is that we can do things like learn the results of pushing objects on a crowded table.
49:59
Right now, we're not testing this inside a planner. So this is a really preliminary work.
50:03
We're just checking to see if the model that we get predicts the data that we trained on.
50:07
Well, I mean, critics held out data well, but it's just based on likelihood.
50:12
And we compared it to two other strategies.
50:16
Right. So there's our learned rule based model.
50:18
We compared it to just a straight up neural network in which we encode the positions of all the objects on the table.
50:27
The problem is that you don't know what order to put them in. And so we picked what we thought was the most helpful order.
50:32
But it's a very hard thing to do. And we compare it against a graph neural network, which is, again, a kind of a modern,
50:39
structured neural network that abstracts away from the individuals in a nice way, but doesn't have exactly the right bias for this kind of problem.
50:49
And what we found so purple is our thing. Blue is the graph.
50:52
Neural network in red is a plain old flat neural network in this case with just three objects in all the scene.
50:58
So we're not testing generalisation over multiple objects,
51:02
but we find again that the sparse rule learning can learn officially very quickly how to make good predictions in this domain.
51:10
More importantly is the fact that it's relatively unaffected by clutter, right?
51:15
So as you add more objects into the world, the neural network suffers because they're all in some arbitrary order and it doesn't know what matters.
51:26
The graph neural network does reasonably well, and the learning thing works still more reliably, you might ask.
51:36
I certainly asked when I first saw these results. You might ask why does it get better as there get to be more objects in the world?
51:43
That seems counterintuitive. The answer is that if there's a bunch of stuff on the table and I'm pushing one object that might push another object,
51:49
but almost everything stays the same.
51:53
So predicting everything stays the same is not so bad and you just have to learn the things that don't stay the same.
51:59
So if you average over the number of objects in the scene and you predict how well you predict what happens,
52:03
then you measure how well you're predicting what happens to them.
52:06
The more objects that don't move, actually, the easier it becomes if you have the right bias.
52:11
But it becomes harder for the flatterer on our. Okay.
52:16
So this is just like a tiny, tiny tip of an iceberg, but it's very exciting.
52:19
I feel like we have the tools and the pieces and the parts to figure out how to make generally intelligent robots.
52:26
I kind of do. We have a bunch of work we have to work on connecting the vision algorithms that exist now,
52:32
which are awesome but not quite what we need into speed estimation in a useful way.
52:38
Right now we have learned policies down at the low level and planning at the high level, but that should be more fluid.
52:44
We should be able to catch the results of planning in a way that lets us routinised the things that we do very frequently.
52:50
We're not doing that. I think end to end learning is a blessing and a curse.
52:56
My colleague Tomas likes to call it dead end learning.
52:59
I'm not sure. So what's interesting about end to end learning, right?
53:04
So that's when you say, I have this giant system and I'm not going to try to give it intermediate signals of success,
53:11
but rather just measure the quality of the whole thing based on the final actions it takes.
53:17
That's the right thing. You can't argue with it intellectually, right?
53:21
It is exactly the right thing. You don't want to say my state estimate it needs to be awesome.
53:26
According to some criterion, that's only about state estimation.
53:29
Really. All I care is that the status metre does a job that helps the planner do a job that causes the controller to emit the right torques.
53:38
That's all I care about. But the idea that you could back propagate from errors on the talks all the way
53:45
through the plan or in the state estimate or to me seems like not so clear.
53:50
So I think we have to figure out ways of combining local reward signals and end to end reward signals.
53:56
We're talking about interacting with humans, all kinds of stuff. So I brought back one more ancient slide.
54:01
This is from 1995, but it's still kind of like my view of what's going on.
54:06
And then we have to think about learning a lot of different kinds of levels of abstraction and how to make them actually not divided into layers.
54:13
I think that right now, especially in robotics, a lot of people are working at what I would call the skill level,
54:19
although, you know, I think in robotics, that's roughly right.
54:24
What's interesting is that we can kind of do skills and then we can kind of do like fancy stuff up at the top, like play go.
54:30
But we're terrible at just like basically making breakfast or even walking out of this lecture room.
54:35
So that middle ground I think is interesting and important, and I want to recruit more people to work on it and think about it.
54:42
So there's a bunch of people who helped with this, and I'm grateful to them for what they have done.
54:48
And with that, I will say thank you and let you watch the robot make mistakes.
54:51
So thanks.