Latest / Strachey Lectures / Strachey Lecture: Can one Define Intelligence as a Computational Phenomenon?

Transcript

Strachey Lecture: Can one Define Intelligence as a Computational Phenomenon? · Strachey Lectures

• 0:02
  I. Okay.
• 0:10
  Welcome, everybody, to the strategy lecture.
• 0:13
  I'd like to start by expressing our gratitude to Oxford Asset Management, which has really generously supported the lecture.
• 0:21
  So I want to thank them for that. I also want to start with one announcement, which is that there will be refreshments after the lecture just outside.
• 0:30
  So everybody is invited. Please do join. And then the most pleasant task will be introducing less volume to is going to be
• 0:39
  speaking to us about whether one can define intelligence as a computational problem.
• 0:45
  So let's delve into one of these people who, as many of you know, invents one research field after another.
• 0:55
  So when I thought of which things to tell you, I just picked a small selection of them.
• 1:00
  And so one is less did a lot of work in algebraic complexity theory.
• 1:05
  So the complexity classes, VHP and BNP, which is still a very active area, the V stands for Valiant.
• 1:13
  Then we moved on and started the complexity of counting and the class number, and there's a whole community that works in that.
• 1:22
  Then we went on and founded Computational Learning Theory, so the book probably approximately correct was really influential and basically
• 1:30
  was the first rigorous study of what can learn after that not being enough.
• 1:38
  Let's became interested in classically simulating quantum, and he discovered holographic algorithms and that whole area, which is another concern.
• 1:50
  Then went on to write his book, Circuits of the Mind, which is about computational analysis and studying the human brain.
• 1:58
  And I think many of these threads will be brought together today.
• 2:03
  It's traditional in these talks to mention prizes, but Les is actually one actually pretty much every prize.
• 2:12
  So I picked out just four so that we would get on to the talk.
• 2:16
  So he's won the Never Linda Prize in 1986, became an FRC in 1991, won the Knuth Prize in 1997, and the Turing Award in 2010.
• 2:28
  And I looked up letters on mathematics, genealogy and found 109 descendants.
• 2:35
  But I'd like to point out that 13 of them are right here in our department.
• 2:39
  So we we all owe quite a lot to Les, not just for his intellectual stimulation, but also for many mentoring.
• 2:48
  So we're really. Oh, and I forgot. Please fill out the question.
• 2:52
  And so I'm delighted to introduce Les Valiant.
• 3:10
  Well, thank you very much, Leslie, for the very kind introduction, and thank you very much for inviting me here.
• 3:16
  So also, about 30 years ago, I spent a very happy sabbatical year in Oxford.
• 3:21
  I was treated very well. So I do have very happy memories of Oxford and I'm very glad to be to be back.
• 3:29
  So what I'm talking about is kind of a theoretical approach to a I.
• 3:34
  And in brief summary, it's a way of reconciling machine learning and reasoning.
• 3:40
  So it's a topic which is close to my heart and has been for a long time.
• 3:45
  But in giving in talks, often the hardest thing to understand is why someone is doing this kind of thing at all.
• 3:52
  So I'll be slightly self-indulgent and try to explain the motivation of of this kind of approach.
• 3:58
  And so first, I want to discuss this notion of a computational phenomenon, which not many people discuss, but to no avail.
• 4:09
  So, you know, algorithms have been around for a long time, and Euclid had had a very good algorithm given to numbers.
• 4:18
  You can find the greatest common divisor and efficiently.
• 4:24
  And so if now, for example, if I give you one number and you want the divisor, you know, which is factoring it, you know, that takes exponential time.
• 4:33
  So no two numbers can find a common factor.
• 4:36
  So there's something very striking already.
• 4:38
  And and what people knew about the algorithms a couple of thousand years ago and many of the best algorithms we know are ancient.
• 4:48
  So what is computer science contributing in general?
• 4:52
  Well, of course, the big change was Turing's paper in 1936.
• 4:58
  So I'll start by trying to spell out my view of what the big event was.
• 5:04
  And I will. Discuss this notion of a computational phenomenon, which for Turing,
• 5:12
  the phenomenon involves computation itself and the model of computation, which for him was a Turing machine.
• 5:18
  And the best way of explaining this, these ideas is, is by making an analogy with physics.
• 5:24
  So I'm not trying to say that computer science is physics are the same, but analogies do serve some purpose.
• 5:31
  So here the claim is that so what do you have in physics?
• 5:35
  You have some laws like F equals M.A. and the Gravitation Law.
• 5:44
  So you've got some laws which are believed to hold generally, but what they really are supported by mathematical theorems which are consequences,
• 5:53
  deductions from the laws and with which you can really understand the incredible breadth of, of what the law means and the analogy.
• 6:03
  Okay, so we learned about this a long time ago.
• 6:07
  So as a computer scientist, sometimes I have wondered what are we offering in a comparable to what the physicists have been doing?
• 6:14
  And on reflection, I think what we are doing is what's Turing did is that what corresponds to the law in physics is a model of computation.
• 6:25
  So he defined a model Turing machine.
• 6:28
  And and the general pain was that it kind of it captures computation in the real world in a very significant and general sense.
• 6:38
  And this is a big statement, but it's supported by mathematical consequences exactly like physicists do.
• 6:44
  So, for example, important consequences is that there's a universe give us a Turing machine, which, without his notion,
• 6:50
  he can discuss the non computable problems and other very important things about models of computation.
• 6:56
  Is that the robust if you make small changes, it shouldn't change the power.
• 7:02
  So I think this is what the main thing computer science offers and what the rest of us since have been trying to emulate in other ways.
• 7:13
  And so the idea here is that as an even more general level, of course, what Newton implied is that you can capture these laws of physics by equations.
• 7:26
  And I'm claiming that because the general statement is that that is phenomena in computation, and you should capture them by models of computation.
• 7:36
  So Turing's example was that there was competition in general and the model was Turing machines.
• 7:42
  But much of what we've been doing since in the algorithms area has been on the same tracks.
• 7:49
  So for example, okay, so, so that's a side comment.
• 7:56
  So I'm describing an analogy with physics and I'm just pointing out that there are other analogies other people use.
• 8:02
  So some people use the analogy that maybe unproved mathematical conjectures like P, not equal turn p, we should treat like physical laws.
• 8:12
  Okay, that the things people believe you can't prove, let's believe it until someone disproves it.
• 8:18
  So that's okay. I've got no quarrel with that.
• 8:20
  But I'm really drawing a different analogy which this kind of more general book I try to it's expansive, a bit more detail than here.
• 8:32
  So the physical laws are which are true. That's true, but not provable.
• 8:36
  It's like a model of computation. The claim that the model of computation is a valid for real, real phenomenon.
• 8:44
  Okay. And okay. So for example, another phenomenon is search.
• 8:50
  And in fact, the best description of this in words is the phrase mental search, which Turing used already in 1948.
• 8:58
  So the idea is that you're searching, searching for oil in the ground.
• 9:01
  You're searching for something in your head, and you're like searching for factors of a number.
• 9:07
  And the definition is called NP, where you're searching for solutions which are short compared to the input size.
• 9:17
  And also given this solution, you can easily verify it.
• 9:20
  So this is a formalisation of search and P is a rigorous is a formal statement of it.
• 9:26
  And we believe that NP is a real phenomenon and computation which lots of people find useful.
• 9:32
  And the model of computation is this non deterministic Turing machine.
• 9:36
  And again, the interest of this definition by itself is not so impressive,
• 9:41
  but the interest is that the powerful mathematical statements and an important one is that the hardest search problems and be complete problems.
• 9:51
  And there's a model of computation and some stunning mathematical statements which are surprising, which make it okay.
• 9:58
  So that's one. Okay. So as it happens, so by P, I mean roughly what we're sure we can compute efficiently.
• 10:08
  In this universe in polynomial time I should put it in randomisation, so it's kind of a stand in for that.
• 10:14
  So computer will captures everything that is during staying the most. General ANP is like a subclass.
• 10:21
  So in other subclasses as a Sharpie, some people here call it number three, which is the counting version.
• 10:28
  And again, this is like a. Okay.
• 10:35
  Okay. So yet another one is BQ P, which is quantum polynomial time.
• 10:41
  So this is our best effort at describing what a quantum machine what?
• 10:45
  How if you use quantum theory for computation, what you'd get.
• 10:50
  And so with each of these classes, again, I think it's the same story that there's a model of computation.
• 10:58
  This is bounded quantum polynomial time and again.
• 11:05
  Besides the suggestion that we should use quantum theory to compute, there are some mathematical consequences.
• 11:11
  And for example, a very powerful consequence is the second one.
• 11:14
  Powerful theorem is that the many ways you could try to use Quantum to compute, and it turns out that all equivalent.
• 11:22
  So that's a strong results.
• 11:23
  So by looking at this model of computation, you do arrive at the conclusion that there's a real competition phenomenon, which is right.
• 11:35
  And so another result is that BQ P is in fact reducible to sharpies.
• 11:40
  So the counting counting says problem. If good counting says problems are more powerful than the quantum class.
• 11:48
  Okay, so we've got this various classes with different.
• 11:53
  Uh. Power and there are more.
• 11:59
  So another phenomenon is a is for this captures the idea of games.
• 12:06
  So do I have a. No, I don't have that yet. So this is your kind of game theory.
• 12:12
  And again, you know, powerful results about that are that the compete problems the hardest members of the game theory class.
• 12:20
  So of course mathematically we don't know all these forces could collapse for all we know.
• 12:27
  And it may be that all these things well up to short P it may be that you can do everything in polynomial, polynomial time even efficiently.
• 12:36
  But even if that happens to the case, one can still discuss these.
• 12:40
  As far as for the phenomena, I think and others extended discussion of of that.
• 12:46
  But so it's with this background that I think I have approached topics.
• 12:53
  So if one looks into, into, into, into machine learning, then so I've tried to formalise the notion of, of supervised learning.
• 13:04
  So we discussed that more code back learning probably possible to correct learning, which is a subclass of of,
• 13:11
  of P which which roughly means that is believed to be a subclass is that there are many things
• 13:18
  that could write a program for by the program could fit into your computer or the universe.
• 13:24
  But this belief that most of this you can't learn from examples.
• 13:27
  So learning is harder than just computing. And okay, so the question of of where the speculating thing is within P, it's a.
• 13:39
  Uh, you know, some important stuff.
• 13:43
  For example, one observation is that cryptography lives out the difference.
• 13:48
  So, uh, so public key cryptography wouldn't exist if any function.
• 13:53
  You could easily learn from examples because then you could kind of learn all the secrets.
• 13:59
  So negative results about, uh, complexity are used every day, especially by cryptographers.
• 14:08
  Okay. Okay. So with the pack learning again, you have a model of computation.
• 14:16
  And so this model, which I describe in more detail, captures the notion of supervised learning,
• 14:22
  which is a well-known concept and widely practised, of course.
• 14:27
  And again, by following this, having a formalisation, uh, so obviously questions of robustness are important.
• 14:36
  If I define this class in different ways, do I get different classes?
• 14:39
  It's important that it's a robust many variants give you the same class and and so some consequences of
• 14:48
  for example that you can give a rigorous demonstration that learning algorithm does really generalise.
• 14:55
  You know, it's a generalisation which used to be some philosophical issue not many, many decades ago, of course, now is practised by machines.
• 15:04
  And you can also explain why some algorithms are, you know, predict in a certain sense, there's no nothing magical about them.
• 15:14
  Okay. So. Okay. So I want to describe actually a little bit because I won't I'll build later on on the on that.
• 15:24
  So it's a form of formalisation, of supervised learning and so supervised learning.
• 15:29
  So there's this term supervised learning, unsupervised learning, which I use in very general senses.
• 15:34
  And one reason for formalising it is that at least it defines what we're discussing.
• 15:42
  But generally it's, it's talk about learning where there's some feedback.
• 15:46
  So supervised learning doesn't mean that there's a supervisor necessarily.
• 15:51
  So, for example, I can look around the room and learn something about the average audience in a
• 15:56
  computer science lecture in Oxford is like the average age or something like that.
• 16:00
  There's no supervisor telling me things. I'm doing this because from other knowledge I can label people myself.
• 16:07
  I know roughly how old everyone is. Okay, so I can learn without an external label.
• 16:11
  So supervised learning doesn't mean that it has to be a supervisor.
• 16:15
  And so essentially it means there's any kind of feedback.
• 16:19
  It's, it's supervised learning. So unsupervised learning is where there's truly, truly no feedback.
• 16:24
  You know, it's some pattern. And somehow we're supposed to draw some conclusion.
• 16:30
  But certainly I think the or the impact of machine learning recently is all this feedback, the supervised ending.
• 16:36
  Okay, so what's this formalisation?
• 16:39
  So the idea is that there's some space through space where it's taught it is an example maybe of a flower and you're trying to classify flowers into,
• 16:49
  into what species they come from. They have types. And B there's a truth.
• 16:55
  F is a ground truth separates is s from the bees.
• 17:00
  And the learner also has a hypothesis which classifies examples.
• 17:06
  And in any rich enough world worth talking about, there will be errors.
• 17:11
  Okay. And we assume serious.
• 17:14
  It's a very rich world exponentially. Many different kinds of examples may maybe influenced infinitely many different examples.
• 17:21
  So it's we want to talk about something realistic.
• 17:26
  And so what's. What's this? Uh, what's the supervised learning phenomenon?
• 17:32
  It seems amazing. It works. People celebrate it, even even in the popular press.
• 17:37
  So what this formulation is first the three, three points.
• 17:41
  So one is that it's an efficiency criterion.
• 17:46
  So it says that there will always be errors.
• 17:50
  But the more examples you take and the more computation you apply, you should be able to reduce your error fairly fast.
• 17:59
  Okay. It should be rewarding to put more effort into into it, into learning.
• 18:05
  So if you double the amount of effort or the number of examples, you should see the increase decrease in the error.
• 18:14
  And so this is a quantify something quantified and the important thing is that it goes down algebraically.
• 18:23
  So if you have an examples there may be errors are good good at one over the square root of n maybe one over the 10th root of N,
• 18:31
  but it wouldn't or shouldn't be slower than that. Okay. So. And of course this is a realised realised gain that basically people in the last ten
• 18:42
  years have increased the budget in data and computation by a factor of maybe thousands.
• 18:50
  And this has brought really good rewards. So this is okay.
• 18:56
  And for some simple learning algorithms, you can prove that the thing learns and it learns so fast.
• 19:01
  Okay. And so this is just in pictures.
• 19:05
  So the quantitative aspect is that the more effort you put in, the error goes down as power of the effort.
• 19:14
  So it may be, you know, one over into the half.
• 19:19
  Okay. So if you want to reduce the error by a factor of two, maybe you should put in the fact a fixed factor like 100 or four.
• 19:27
  Okay. And some people actually experimentally verify this that for some task of predicting next words various deep learning.
• 19:39
  Uh, algorithms do have this linear this to go to this polynomial resulting in error.
• 19:48
  So this is a log log scale, so you straighten out the curve to a straight line.
• 19:52
  And so also speculating doesn't tell you what power law this should be.
• 19:58
  And in fact, there's evidence that different applications do have different vowels.
• 20:02
  So if you do some natural language data sets or a vision data set, they have different power.
• 20:08
  They have different power laws. Okay.
• 20:09
  So like, like this one is assumed to be a very slow power that's like point sort of fixed power, but it's .06 good enough.
• 20:18
  Okay. So this is an efficient efficiency that.
• 20:23
  Okay. So so we demand before we call a machine learning algorithm successfully,
• 20:29
  we demand this efficiency criterion because the two other aspects are one is that we want to be realistic.
• 20:36
  The world is complicated. So the last thing we want is to solve this, okay, is to make an assumption so we know that something's a something is a BS,
• 20:49
  but in different worlds, maybe the different probabilities of each, each kind.
• 20:54
  So if you come here to go to China, maybe the same flowers, but with different possibilities.
• 21:01
  So, so this requirement is that the second requirement is that this learning algorithm work for arbitrary distributions.
• 21:10
  So the secret, of course, is that you're going to learn on or learn on a distribution and you're going to have to perform on the same distribution.
• 21:18
  So for certain, the kind of flowers common here, you'll be tested on the on these common flowers which are common here,
• 21:23
  the different china, they'll be tested on something different.
• 21:27
  But so this basically says that in practice, the successful learning algorithms are very broad, broad spectrum.
• 21:33
  Okay. They don't just work for the uniform distribution. Okay.
• 21:38
  And then the third thing, which is a bit more sophisticated, is that so when you're learning, the learning algorithm gives you a hypothesis.
• 21:47
  There's a computational representation of everything else.
• 21:50
  But the classification, the teacher is just a function of you don't look inside the teacher.
• 21:57
  Okay. So it's just a behaviour. And so in practice it's, you know, so this learning algorithm is something you have in your hand,
• 22:07
  maybe a perception, a deep learning network, some sort of boosting.
• 22:14
  So something you have in your hand. But then the examples come and no one guarantees where it comes from.
• 22:21
  But often the use of this representation for learning and you've got no chance of learning everything you can represent.
• 22:28
  But you're still successful. And the reason usually is that the examples come from from a weaker world.
• 22:34
  So is there something something simple about the world are learning from. So the.
• 22:40
  Like this. So the. Okay.
• 22:43
  So the mystery of why certain the surest X work well in practice is often that the tasks they're given have some simplicity in them,
• 22:51
  which is often very hard to identify. Anyway.
• 22:55
  So so this is a specification of of a formal model of supervised learning.
• 23:03
  Okay. So. Okay. That was by way of introduction.
• 23:09
  Okay. So how. Okay. So.
• 23:15
  Okay. So I've got this model of inductive learning and we know that machine learning, which does roughly this kind of thing, is very successful.
• 23:25
  But the question is the type of problem is something about the intelligence.
• 23:29
  So is this all of intelligence? And so everyone would agree that the answer is kind of no or almost everyone agrees.
• 23:36
  So what? What more what more is there? And so what we want to do, if you follow this approach is to we need a model of computation.
• 23:45
  So back learning is a model of learning, but it's not enough because just learning we don't think is enough.
• 23:51
  So know what can we do more? What's what should we add?
• 23:56
  And I will add because I think inductive learning gets pretty powerful phenomenon and we need to add to it rather than start from scratch.
• 24:07
  What do we need to need to capture? And so the adage, which I've been going around for a long time and drawing in advertising is,
• 24:17
  is this line from Aristotle who said that all belief comes from syllogism or induction.
• 24:24
  And by which you mean something like, if you believe that, if you have a belief in your head,
• 24:29
  then you either deduced it's a selective syllogism or some sort of logical deduction.
• 24:36
  You use it for something else to you, or always by induction.
• 24:40
  So induction means that you somehow from on basic empirical evidence, you generalised it somehow.
• 24:46
  Okay. And so, of course, he spent, you know, 99% of his effort on syllogism and didn't say much about induction.
• 24:57
  But so what's happened since, of course, Syllogism has become this big field of mathematical logic and formulas.
• 25:03
  His reasoning? So induction became this mysterious sort of philosophical field.
• 25:09
  But I think it's the issues have been clarified by machine learning and machine learning theory.
• 25:15
  So as an example. So when I started.
• 25:19
  The question is question how come that, you know, children have seen different examples of of chairs,
• 25:25
  different parts of the world, and yet even a new chair, they agree on what's a chair and what's not.
• 25:30
  That was kind of a mystery. You know, there wasn't a good answer to that. But now machines can do this routinely.
• 25:35
  So asking this question when mystify anyone living now.
• 25:42
  And the reason is that that machine learning theory gives an answer on what it means to achieve this.
• 25:49
  You don't have to perform well on this distribution. You've seen it's probabilistic anyway.
• 25:55
  So we we do have a handle on on this. And before I go on, just to say that there are some technological aims here.
• 26:07
  So so what I'm discussing will be how you want to unify a view,
• 26:13
  have a unified view of of reasoning and of learning, because at the moment they're very different.
• 26:19
  You know, reasoning is a very classical reasoning.
• 26:23
  Classical logic is this very brittle kind of a mathematical theory.
• 26:27
  But as machine learning is of, it's this kind of robust thing of a different kind.
• 26:31
  So we do want to unify them. But kind of the grand goal, if you can do that as foundational technology,
• 26:38
  is kind of to approach the central problem of A.I., which I believe is, you know, how you put into a computer knowledge,
• 26:46
  which at the moment is very hard to acquire common sense knowledge and be able to use it in the computer to to reason,
• 26:53
  to make predictions, deductions, whatever. Okay.
• 26:58
  So to do the second, I can't imagine how you can do the second unless you take some unified view of what reasoning and learning are.
• 27:06
  It seems that if the two disparate things, it's a bit difficult.
• 27:12
  Now. So in modern terms, I suppose there's a debate.
• 27:19
  And so I'm basically I'll be saying that reasoning and learning are both important and we have to reconcile them.
• 27:25
  Not everyone has to agree. So, for example, at the moment,
• 27:30
  there are some people who are so enthusiastic about machine learning that they think that
• 27:34
  a single black box machine learning thing will do everything and we won't need reasoning.
• 27:39
  Okay, so that's as have you. And other people may, may put reasoning high on the pedestal.
• 27:50
  And says, but putting it more simply. The question is, are other people actually deny that reasoning is real?
• 27:57
  Other people who deny that learning is real. So certainly, I think 30, 40 years ago there were real learning deniers,
• 28:04
  people who thought that intelligence was all reasoning and putting learning facts and reasoning efficiently with them.
• 28:12
  They were suddenly learning deniers then and now there's some reasoning deniers around.
• 28:18
  But, you know, in this talk, I'll take a middle ground. Okay, so let's buy this one.
• 28:25
  That Aristotle on a cell phone. So.
• 28:30
  Okay, so. Okay.
• 28:33
  So so most people can answer this question without too much effort.
• 28:37
  But the question is, you know, did you use pure learning for this or did you use pure reasoning or did you use something else?
• 28:46
  Okay. So the main contrast I want to give is that at the moment one has to argue a bit
• 28:52
  against people who want to do everything by a single black box machine learning thing.
• 28:57
  So for example, the idea is that if you feed this black box, you know, a billion sentences from the Web,
• 29:04
  you know, maybe you can answer every question and the reasoning will go away.
• 29:08
  But I think kind of commonsense introspection suggests that to answer this question, you know,
• 29:16
  it's not that we've been exposed to thousands of sentences about Aristotle's property, okay?
• 29:24
  But we somehow knew some facts and we train together facts with you.
• 29:28
  So some some reasoning involved. So this is introspection.
• 29:33
  But can we ask the same question of learning versus reasoning a bit more scientifically?
• 29:39
  So. So we want some somebody to do some experiment to do which tests this kind of issue in a plausible way.
• 29:50
  And so the problem which came to us, which is, I think, very natural for this, is called the work completion problem.
• 30:00
  And this essentially is that I'll take a phrase from a website or a newspaper, usually a headline, and I delete a word.
• 30:09
  You have to guess what the missing word is, because this is kind of a test.
• 30:13
  It's quite a good IQ test because it's quite hard to do.
• 30:16
  These headlines are often quite succinct, just the minimum number of words to express what you want to say.
• 30:25
  And okay. And of course, it's important we took these headlines from is maybe from a world where you have no knowledge.
• 30:34
  So actually the ones are examples I have has happens to be from our English language Chinese newspaper.
• 30:41
  Okay. So let's have a some examples here. So.
• 30:46
  This was whatever the year of the dog holds in store, pet owners will be lavishing more attention than ever on their.
• 30:54
  So you have to guess where the missing word is. And question is, could your computer program do it?
• 31:02
  Any children. Okay.
• 31:07
  Okay. So the answer was Peaches, which is a in a bunch of scientists in early, early 20th century American origin word for dog issue.
• 31:18
  And so the question is, is this a hard problem, say, for a black box machine learning algorithm?
• 31:23
  And so my guess is that this is easy,
• 31:26
  because if you look up Google and you look for sentences with pet owners and their pooches in it, you've got tens of thousands.
• 31:33
  So this is an easy problem for a black box machine learning. Okay.
• 31:38
  Okay. So another one. China rises as a maritime powerhouse after snapping up profitable blank blank across the world.
• 31:47
  Fragrance and fragrance trade.
• 31:51
  So. Yeah. Okay. So that's the answer. The good seaport terminals.
• 31:55
  Okay, good. So I reckon this is slightly harder that you probably to do some reasoning.
• 32:01
  You just can't do it by some sort of word because, you know, I didn't have any examples of that.
• 32:11
  So the hardest examples for where? Inductive learning is disposable as well, where there's some kind of news.
• 32:18
  So to understand the headline, you have to know what happened yesterday. Okay.
• 32:22
  So, for example, one thing is retail sales are up 20.7% in second quarter.
• 32:28
  Okay. So so maybe again. So, okay, so the answer here is a cow.
• 32:33
  So maybe if you're an expert on conditions in the different parts of China, he could do it.
• 32:38
  But you need lots of knowledge and maybe recent news, that kind of stuff.
• 32:41
  But certainly, you know, if things depend on you news this morning, then, you know, having a billion sentences in your brain doesn't help you.
• 32:51
  Okay, so. Okay. So the interesting thing about this thing is that the idea is that with this problem,
• 33:01
  you can test your machine learning system how how well it solves this problem.
• 33:09
  And so I think this problem is not bad. So this is so it's kind of a stand in for the Turing test in certain ways.
• 33:17
  So in one sense, there's something for the Turing test has many aspects,
• 33:21
  but one aspect is that the measure something, you know, how well do you perform compared to something else?
• 33:28
  And of course, the other important thing about the Turing test is that he didn't say the intelligence depends on how well you play chess or how well,
• 33:34
  you know, chemistry, but it depends on general general knowledge of general stuff.
• 33:41
  So. So this missing word test is good on that.
• 33:46
  And so the learning theory, Pursell adds,
• 33:49
  is that certainly it emphasises that any kind of performance in any system like this is with respect to a particular distribution.
• 33:58
  So it's hard to be intelligent if, you know, if you go somewhere where your knowledge is irrelevant.
• 34:05
  And also it emphasises feasible computation that we're interested in efficient computation,
• 34:12
  infeasible computation and controlling the error of your prediction and things like that.
• 34:16
  Okay. So we'll come back to this problem.
• 34:20
  So I'm suggesting that if you tackle this problem of of common sense knowledge and learning and reasoning or whatever,
• 34:27
  this isn't a bad problem to test your system on because, you know, it's there's a ground truth.
• 34:33
  There's a ground truth. And it's about this general knowledge.
• 34:36
  Okay. Okay.
• 34:38
  So so what I'm really coming to is my content, which is okay, which is kind of my suggestions for having a model of computation which can do both,
• 34:51
  both inductive learning, which I think is important phenomenon, and you can add on reasoning to it.
• 34:58
  Okay. So and with this combined system, if you do it well, which we haven't yet,
• 35:04
  that you can test it on on this a problem like this with competition problem.
• 35:09
  Missing work problem. Okay. So the question is, how how do we add?
• 35:13
  Uh. Okay. So what is intelligent thinking?
• 35:18
  You know, what else do we do besides inductive learning? And how do we make this into a model of computation?
• 35:24
  So these models get, you know, kind of complicated.
• 35:29
  A story machine is very complicated. This is much more complicated.
• 35:34
  And it's justification is that you're capturing something important maybe, and that other ways of capturing it would boil down to the same thing.
• 35:42
  Okay. Anyway, so this is more a list of things you need to capture.
• 35:46
  And so the first feature of it is this idea which is borrowed from cognitive science of a working memory.
• 35:53
  So this amazing thing we have to cognition, which is that while we have this enormous storage of memories of each instance,
• 36:02
  somehow we've got this small mind's eye which directs our behaviour at each instant and of this little world in front of us,
• 36:10
  what we're aware of, and we use this awareness to plan our lives, what we do next, what we do after the lecture.
• 36:17
  And so now all our behave is channelled through the small window.
• 36:20
  And so what's going on here?
• 36:23
  So the explanation we're here will be is that we need to restrict the window for complexity reasons, for computational complexity reasons.
• 36:32
  And we as a model, we need to use it to get anywhere.
• 36:37
  Okay, so so roughly, this is how we formulated.
• 36:41
  So you wake up in the morning and your mind's eye is blank,
• 36:46
  but it's got some two free tokens and can fill it up during the day with what you're thinking about.
• 36:52
  Okay. So you fill it out with the scene.
• 36:55
  So you think of your dog. Okay, so you think of your dog, and then you want it.
• 37:00
  You want to pick your dog, you want to see what is the dog like.
• 37:04
  And then you have a rule in your head which tells you that, in fact, dog's like bones.
• 37:09
  Okay. Okay. So, so.
• 37:12
  So somehow with your background noise from your big, long term memory, you can fill up your your mind's eye, too, with missing information.
• 37:23
  So this is roughly what goes on. But but here we come to the first difference between logic and learning.
• 37:33
  And so the point is that this implication doesn't fit well with with like pack learning or any kind of learning because.
• 37:44
  I think when you do machine learning, you go got some target function and so you're learning to recognise an elephant.
• 37:52
  And so basically what you're recognising is in this is a, this is a sufficient condition of a picture to contain a reputation of an elephant.
• 38:01
  Okay. So what we are definitely learning, if you do if you do supervised learning is an equivalence.
• 38:06
  Okay. So you have a maybe of a big neural network, perceptron or decision tree.
• 38:14
  And it's you want to predict whether what's in front of you is a or not.
• 38:18
  And on the left hand side is some some very rich, complicated, incredibly complicated rule.
• 38:25
  Tens of thousands of bits. Best to try, possibly, but it does contain a useful criterion of whether you know what's in front of you is a bone or not.
• 38:36
  Like a it's a predictor. Hmm. Okay. So the proposal, the first step of the proposal is that are going to learn these things.
• 38:46
  So maybe for each word in the dictionary, you're going to learn a predictor in terms in terms of the other words.
• 38:53
  And this predictor can be any whatever you're machine learning out and can do depends on your computational resources about what you can do.
• 39:03
  And this is what learning rules are. They'll do equivalences, but the point is that you can use a equivalences to make.
• 39:11
  You can change these together to make predictions.
• 39:14
  Okay, so, so using something like this, if the conditions in your C predict that this is a bone, then this is a very good thing to predict as a bone.
• 39:24
  And you can once you predict it's a bone, you can make further predictions about your seen using these equivalences.
• 39:30
  Okay. So so that's that's basically the idea is that you're this is your mind works your mind is full of not one black box neural net,
• 39:40
  but tens of thousands. And somehow the predictions of these tens of thousands are can be used together in a principled way to make predictions,
• 39:50
  because that's the kind of the rough summary. Okay.
• 39:54
  So that's okay. First step.
• 39:57
  Okay. So so this robust logic is this the system for is this model of computation.
• 40:04
  And the first aspect of it is that it we're going to learn rules with equivalences which predict maybe every concept in the in the in the dictionary.
• 40:19
  Okay. Aspect one. Aspect to.
• 40:29
  Well that we will have quantifies. Okay.
• 40:32
  So this is we have sun exists and whatever's bit like in logic, but now they mean something much more grounded than they do in conventional logic.
• 40:43
  So in logic you learn things like you know or man or mortal.
• 40:47
  But then what does that mean? Has someone checked out meaning throughout the universe?
• 40:52
  Probably not. So this quantum quantifies a bit.
• 40:57
  Bit? Kind of embarrassing, almost. So in this logic, the quantifies only refer to this mind's eye.
• 41:04
  Okay, so you got what you're thinking about. Does something exist there in your mind's eye, or is something true for everything in your mind's eye?
• 41:11
  So it's a very, very local thing. And I suppose I should just point out already is that, you know,
• 41:18
  somehow this political calculus logic is hasn't worked out too well for I and there's almost something kind of embarrassing about it.
• 41:28
  So, you know, but certainly this rule well, obviously there are many things which your dog likes, not just the bone.
• 41:35
  So there's something, something very simplistic about its logical expressions and the something brittle.
• 41:42
  But the idea that, you know, the idea of predicting what a bone is or possible versions of it,
• 41:48
  you know, you know, there's nothing mysterious about that. That's what machine learning technology does for you.
• 41:55
  Okay. So, uh, so we have some quantify as etc., etc.
• 42:01
  Um, okay. So third thing is, uh, what about consistency?
• 42:07
  So as I said, so we're going to learn all these rules.
• 42:12
  Okay. So that's a rule we a lot of these rules. So what if somehow they if you train them together, you get inconsistencies.
• 42:21
  So logic is certainly hung up on inconsistency. Okay.
• 42:24
  So here we say, don't worry. You learn all these rules.
• 42:28
  We'll just live with the inconsistencies. And if you if the inconsistencies are important, you somehow learn your way out of it.
• 42:36
  Right. So I know the 1960s or seventies version of this problem used to be no longer is this what's called the Nixon Triangle.
• 42:46
  So you learn that, uh, Quakers are pacifists.
• 42:52
  Uh, that's a rule. You know, Republicans are not pacifists.
• 42:56
  So these are two rules you go round with. And then there's example of someone called Richard Nixon, and he was both a Quaker and not a pacifist.
• 43:04
  So then what do you do? Okay, so here the answer is, don't worry about it.
• 43:08
  Go around with your general rules. And if this counterexample is somewhat worrying, then, you know, your learning algorithm will say that,
• 43:15
  you know, all Quakers, Quakers except Richard Nixon are pacifists.
• 43:19
  Okay, so you you'll learn your way out of an inconsistency if it's important,
• 43:24
  but you've got no chance of maintaining consistency in a complicated world.
• 43:29
  Okay, so that's as easy. Okay.
• 43:34
  So rules will be learned. So instead of learning to recognise elephants, we're going to learn rules.
• 43:44
  And so these rules will predict in design the mind's eye.
• 43:49
  And then this probably possibly the correct sense. And we will look for rules which are highly reliable.
• 43:56
  Okay. So we've just learned rules.
• 44:00
  So as aspect for. So a more subtle issue, actually.
• 44:05
  There's a lot of discussion is was this distribution business.
• 44:09
  So here we do get rather kind of strange philosophical problems.
• 44:15
  So as I said, this question of how come we agree on other areas?
• 44:20
  Although we've seen different examples, that's kind of has some history, but in the end it's not so mysterious.
• 44:26
  Like we could believe it. There's one distribution here, but then it gets more mysterious.
• 44:31
  So we've learned that to know, uh, you know, Aristotle lived a long time ago, well, etc., etc.
• 44:38
  So how we, how we use that. He didn't have a cell phone.
• 44:42
  So when we learn these general facts, it's not quite clear what the distribution is, you know?
• 44:46
  So it's a bit lost. But but you have to take a stance on this.
• 44:49
  We have a model of computation. You have to kind of commit yourself. So the following is just a detailed version.
• 44:58
  But okay, but the short term version is, is, is this that maybe this is the central thing is that.
• 45:08
  So do we. Oh, yes. Okay.
• 45:12
  So the central model is this. The idea is that you've got this very long term memory of of lots of rules.
• 45:18
  You got very big brain, very complicated world outside.
• 45:23
  But what saves us makes cognition possible is that there's this kind of funnel
• 45:28
  in between your mind's eye and somehow the examples of the world you see,
• 45:34
  you summarise very as as a subset of sketch or caricature.
• 45:39
  And then within the simple scene, you see you you apply your rules.
• 45:45
  Okay. So if I look out and I'll probably see three groups of seats, I don't I can't see every individual.
• 45:53
  So we we apply these rules to simplified scenes. Okay.
• 45:58
  Like this. Okay. So. So what is this? Well, distribution.
• 46:02
  So, very roughly. Just to persuade you that there's a way of committing yourself.
• 46:07
  Although could persuade you. Persuade you that this is the right way.
• 46:11
  Isn't that would take more time. So the idea is that for each scene in your mind's eye, you think of something.
• 46:18
  There are all these features. So everything is true or false.
• 46:22
  But the whole essence is that the world in this game is in the description is incomplete.
• 46:28
  Okay. You think I want to go home? And then somehow you fill in this scene about how you're going to.
• 46:35
  How. What's a reasonable way of going home? So.
• 46:39
  So in this mind's eye, there's very little which is specified. So most of it are stars.
• 46:45
  So some are definitely. Yes, some are. Definitely knows about two knows infinite list of stars.
• 46:51
  Okay. And so this is that the world sometimes doesn't specify the value of a feature.
• 46:58
  Most of the time it doesn't. And again, so going back to the A.I. from a long time ago.
• 47:03
  So again, the famous paradox was that this is a bird called Tweety.
• 47:10
  I'll tell you, it's a bird. And I ask you, trees is a bird.
• 47:14
  Does it fly? And you say yes. And then I'll tell you the truth as a penguin.
• 47:20
  Then you change your mind. Okay, so this is some sort of paradox, if you think of it in any kind of reasonable logic.
• 47:27
  But again, in this formulation, uh, almost without doing anything, there is no paradox,
• 47:33
  because the idea is that if I tell you something as a bird and I don't comment on whether it's a penguin, then in fact it's probably not a penguin.
• 47:42
  If it was a penguin, I'd probably bother to tell you. So that is that there's a distribution of examples you've seen and the.
• 47:49
  And if something is mentioned, that may be useful information.
• 47:54
  Okay. So, so the incomplete specifications, uh, solve some, uh, paradoxes already.
• 48:04
  That's a comment. Okay. And then the game being played.
• 48:10
  Is that so? You've got your mind's eye.
• 48:14
  Uh, some things are, yes.
• 48:18
  Talking about your dog? No, it's not a cat. Most things are unspecified.
• 48:22
  And then there's one thing you want to predict or, you know, what is your dog like?
• 48:27
  So a question mark is like a force predictions that there's a ground truth.
• 48:31
  Maybe there's ground truth as a probability distribution, you have to reply.
• 48:35
  So this is there's a distribute, there's a distribution out there. Okay.
• 48:40
  Okay. So now. Okay. So another aspect which I think is a is a deep one is this idea that.
• 48:47
  So once you we have you're learning many things. At the same time, there's this notion of hierarchical learning.
• 48:53
  Okay. So you're going to learn this word in the dictionary, that little word in the dictionary.
• 48:57
  And so what happens if you only understand this word and you understand the second word in terms of the first?
• 49:03
  Or like if you go to a math, scores that are different concepts and someone.
• 49:10
  Okay. So so if you only have understand the concept, is it useful to label to tell you a new example where that's half understood concepts is in that.
• 49:21
  So all the evidence is that if you half understand, stuff is not very useful.
• 49:24
  Okay. So it's very hard to learn things.
• 49:28
  So it's very hard to learn a concept in terms of other concepts before you really understand other concepts very well.
• 49:34
  Okay. So this is one reason why if you just stare around in the universe, it's hard to learn a complicated concept like that.
• 49:44
  The planets go round in ellipses. But that's not so easy to spot from looking at the sky.
• 49:49
  Okay, so. So, in fact, the way the system works, which, you know, first I thought it's a weakness and embarrassment,
• 49:58
  but I think it's probably inevitable is that these examples do have to come with with kind of correct labels.
• 50:07
  So it's, you know, the value of of universities is that you go to lectures and someone,
• 50:14
  you know, meticulously gives the exact labelled examples which are kind of correct.
• 50:20
  Okay. You don't just skim the web and try to learn something complicated.
• 50:24
  Okay. So someone has to label the outputs correctly and also the features correctly.
• 50:30
  And if I say that's a yeah and a group or something, which is this is this then it has to be commutative.
• 50:38
  Not very helpful unless you've learned about commutative means at the beginning.
• 50:42
  Okay. So.
• 50:48
  Okay. So so that's an aspect of this of this model of computation and.
• 50:59
  Okay. And that's also a lost aspect of what I want to emphasise is that so this is different from making a public stock model, model of the world.
• 51:06
  This is something which is kind of avoid some of the complications.
• 51:10
  So idea of probably personally correct is that you're assuming that the things you're going to predict are close to probability, one of being correct.
• 51:20
  I'm not in the business of estimating probabilities point three and point five and point seven and computing with them.
• 51:25
  There's little, little evidence that humans are any good at that.
• 51:29
  And in trying to understand cognition, we somehow have to avoid that.
• 51:35
  Okay. Okay. So that's seven features.
• 51:40
  Okay. So we. Okay, so we're going to learn the rules using whatever it is you like.
• 51:48
  I mean, that's. That's the parameter.
• 51:54
  Okay. So I mentioned currently general features in a second, maybe one with one dictionary.
• 52:00
  And you. Okay. Okay. Good. Okay.
• 52:03
  So I mentioned this word missing word problem.
• 52:08
  So a long time ago. Well, a while ago with the law is this, Michael, which is an experiment.
• 52:13
  This was ten years ago. A simple experiment. Small data set, simple algorithms.
• 52:20
  And so the idea was that we took a natural language database from the Wall Street Journal.
• 52:25
  We use some standard stuff from machine learning, from natural language processing.
• 52:30
  We used online dictionaries, word net services,
• 52:34
  and and the exercise was that from record we were going to learn rules about the world from single sentences.
• 52:45
  So to do it properly, we should be learning from paragraphs or more.
• 52:51
  So the idea was we were trying to learn facts about the world,
• 52:55
  which are different from just syntactic features of which you can do by just by applying machine learning boxes.
• 53:05
  Okay. And the issue was so issue was testing my main hypothesis, which is that there's a there's even if you can learn very well,
• 53:13
  even if you can do black box learning well, is there added value in training and training these rules?
• 53:19
  Okay. So. Here we are testing this hypothesis, and this is an example of the kind of rules we learned.
• 53:29
  So this is The Wall Street Journal. It's about business. So a typical word is is so we call a missing word.
• 53:37
  And the question is, is the missing word price typical word you find in Wall Street Journal?
• 53:42
  Maybe. And so for each word, as I said, you have a predictor.
• 53:49
  Predicting some big, enormous, big mess.
• 53:52
  But suppose it is predicted for you, whether messing with its price and the machine learning algorithm we used was essentially close to perceptron.
• 54:03
  So we learning inequality, a linear inequality, but the features were these compounds features.
• 54:12
  And so the idea was that if somehow you find the structure in the sentence where there's one word X,
• 54:20
  which the word was bargain and the sentence was telling you that this bargain lowers something,
• 54:30
  then you should deduce that what it lowers is the price. Okay.
• 54:34
  So bargain lowering something is good evidence for the missing well being price and competition also known as price.
• 54:42
  So lots of independent evidence which could add up to decide whether you're missing, whether it was price.
• 54:48
  But anyway, so you could somehow you through this data set at your learning algorithm.
• 54:55
  And the aim was to learn the facts about the world. Okay. So this is facts about the world which were beyond just what things you could.
• 55:04
  Okay. So you learn facts about the world,
• 55:06
  which hopefully you could train together and then reach conclusions which would be beyond just a simple black box learning.
• 55:13
  Okay, so, so we did this and, you know, it's a kind of very small scale experiment.
• 55:21
  We got some results. So. So main thing was that.
• 55:25
  So we have this first about 260 words which were targets which are quite frequently enough.
• 55:32
  And I think already we. This is ordered from left to right.
• 55:37
  So the words were our methods were most favoured were the right. And I think red was what the.
• 55:44
  Okay. So blue was just machine learning, red was machine learning post reasoning.
• 55:50
  And so for some words we really did much better. And with some others kind of everything was hidden in the noise.
• 55:58
  And the general phenomenon here is that in machine learning, big data is very powerful.
• 56:05
  And once you start adding reasoning, you can introduce all kinds of noise.
• 56:10
  So pure machine learning is quite something to compete against, so it's quite hard to improve on it, but it's still possible.
• 56:18
  Okay. So. Okay. So I think I've got maybe two slides, which is a bit more technical.
• 56:24
  So this is what the robust logic thing is. So in our minds I.
• 56:32
  Okay. Because some objects to objects.
• 56:36
  So the right hand side of the rules on just particles like bone, they could be relations too, like above or byes or something like that.
• 56:46
  So the left hand side could be any the the hypothesis of any learning algorithm we used in the inequalities.
• 56:56
  Again, they could have compound features like this that, you know, so this thing is true.
• 57:04
  If, you know, there's an object in your mind's eye as if every other object in the mind's eye, various things are true.
• 57:11
  So having complicated features makes you learn better.
• 57:17
  But you've got enormous numbers of these. You generate them momentarily.
• 57:21
  So you got a trade off. And use any learning algorithm you like because it becomes propositional.
• 57:30
  You just plug in your learning algorithm. That's what you do.
• 57:33
  And then what you are guaranteed is that, well, these rules will be learnable by definition.
• 57:42
  And the main promise is that when you train together these rules, then.
• 57:48
  There's some some promises. And very roughly the main promise is that if you've turned together the two rules and each rule is accurate to 95%,
• 58:00
  then when you train them together, the conclusion will be correct with it's probably 90%.
• 58:05
  Okay, so you lose accuracy the deeper the training, but it gives you some principle, principled way of training together, even two things.
• 58:13
  So that's the kind of main promise.
• 58:16
  And the idea is that it seems that if you want to do logic on a learned on set knowledge in a principled way,
• 58:22
  in a big machine, in a, it seems hard to avoid such a requirement.
• 58:28
  Okay. And so everything will work in polynomial time.
• 58:35
  That's how things are defined.
• 58:37
  The only restriction you need is that the relations of constant parity so I can have above is above B, likes A, likes B.
• 58:47
  So there's a binary and but the costs will be go up exponentially with the arity of of the relations.
• 58:53
  So we have to divide straight up the world into relations of constant parity.
• 58:57
  So this doesn't worry people because it's a reasonable requirement.
• 59:02
  Otherwise everything is polynomials. In fact, the number of tokens you have in your mind's eye.
• 59:08
  So psychologists tell us it's what seven plus one is two.
• 59:14
  So things are polynomial in that. So we know it's not exponential.
• 59:18
  So maybe you can have 20, maybe 30. You don't have to worry too much about that.
• 59:23
  Okay. So. Okay.
• 59:28
  So the outcome was if you build a system on these principles,
• 59:31
  you would learn a lot is use lots of learning boxes, but they interact in a principled way.
• 59:39
  And what this would address, I think, is certainly acquiring knowledge, which is hard to acquire how to program.
• 59:48
  It has to be done by learning. And so hopefully we're building reasoning systems by programming failed because they're too brittle.
• 59:58
  Hopefully learning will get you out of the brittleness.
• 1:00:06
  Okay. So make a consistent. Okay.
• 1:00:13
  So so I think what the reasoning,
• 1:00:18
  the Aristotle example suggests is that will be when we reasonably often reason
• 1:00:23
  in cases when there are few direct examples and that's what this solves and.
• 1:00:30
  I may be a general comment. So this is a very general issue everyone discusses.
• 1:00:33
  In the end, machine learning is the idea of explanations.
• 1:00:37
  Okay, so people don't like blackbox machine learning because they know no explanations.
• 1:00:42
  So in this kind of system, it's kind of a half a solution because what we're saying is that we're going to we're going to have lots of black boxes,
• 1:00:51
  but each black box is going to predict something. You understand some word you want.
• 1:00:55
  You're going to choose the terms in which you want your problem. Understood.
• 1:00:58
  And once you've got a prediction, then you've got lots of black boxes there.
• 1:01:03
  But you understand which features you, you, you, you care about are being explained.
• 1:01:10
  And so this idea of explanations being kind of only half a level, I think is,
• 1:01:17
  is it's quite appropriate because I think this is similar with human explanation.
• 1:01:21
  So if if you ask me, you know, why, why do I bring my my umbrella?
• 1:01:27
  I'll say I, you know, I don't want to go. I thought it was going to rain.
• 1:01:31
  And you say, oh, well, so, well, I don't want to get wet.
• 1:01:34
  But at some point, if you keep asking me questions, at some point I'll say, I don't know the answer.
• 1:01:39
  Okay. So, okay, so our explanations are also only up to a certain level that we can explain what we think.
• 1:01:45
  So. Okay. So so computers won't be able to explain to you that they are soft give giving explanations also at some point.
• 1:01:54
  So this kind of system gives explanations in terms of what you request.
• 1:01:58
  And maybe beyond that is as is hopeless anyway.
• 1:02:03
  Okay. So. Okay, so I think. So by machine being educated rather than trained.
• 1:02:11
  I mean that you when the machine learns it doesn't know how it's learned nowadays it's going to be used.
• 1:02:19
  So when you're train with one single machine learning books, there's a lot of knowledge goes into it.
• 1:02:25
  But the only thing this knowledge will be able to do is to predict exactly what you had in mind when you were training it.
• 1:02:31
  Okay. Now we learn all kinds of stuff in college and elsewhere, and then we can apply it to the new situation.
• 1:02:38
  And this is very much like having, you know,
• 1:02:40
  many black boxes loading in parallel and having a principled way of of using them to make a prediction or an explanation and a new situation.
• 1:02:52
  Okay. So. Okay. Okay.
• 1:02:55
  So why me? Okay.
• 1:03:00
  Okay. So. Okay. Very quickly. And difficulties.
• 1:03:05
  Well, the main difficulty is getting good training sets. So in machine learning, obviously good training sets were very important.
• 1:03:14
  And the recent developments. This is a challenge. So.
• 1:03:18
  Okay, so where do I get trading material?
• 1:03:22
  I want to know the colour of an elephant. I put these different options into Google and I find this.
• 1:03:31
  So then I decide, well, this is better data than this. I want to go to Google Scholar.
• 1:03:36
  Okay, then I find this. Okay, so this is good.
• 1:03:42
  Okay. So, okay, so it seems that it's getting good data sets.
• 1:03:48
  Is is a problem. Okay. So let's forget about this.
• 1:03:52
  So I think what's needed is really big experiment, which basically needs big new data sets which can test something like this.
• 1:04:00
  So just like know the big vision data sets produced six, seven years ago.
• 1:04:04
  A very important influential. Um, what we need is,
• 1:04:09
  is big enough data sets with good with good information which kind of challenges this requirement of doing reasoning in a broad enough um,
• 1:04:22
  context context to be to be interesting. Okay.
• 1:04:27
  Um, okay. So certainly.
• 1:04:30
  So the general summary is that what we're good at is throwing computational power at something.
• 1:04:37
  And I'm suggesting that we should throw computational power at something where we know that there's a real phenomenon,
• 1:04:42
  that it's a supervised learning as one. But if we want to broaden it to intelligence,
• 1:04:46
  then we have to first decide what's the real phenomenon and then throw a computational power at it so that it.