Latest / Strachey Lectures / Strachey Lecture: How Are New Technologies Changing What We See?

Transcript

Strachey Lecture: How Are New Technologies Changing What We See? · Strachey Lectures

• 0:00
  It seems like.
• 0:14
  Welcome everybody to this term strategy lecture. This is a series of distinguished lectures named after Professor Christopher Street.
• 0:22
  He founded the Programming Research Group here in Oxford in 1965.
• 0:27
  And together with Dana Scott, he founded the fields of denotation semantics, but provided a firm mathematical foundation for programming languages.
• 0:35
  Before introducing today's speaker, which I'm about to do, I would like to thank Oxford Asset Management,
• 0:41
  who have generously supported this lecture series since 2015.
• 0:47
  So without their support, we wouldn't really be able to have this exciting lecture series full of distinguished speakers,
• 0:52
  so we're very grateful for that. I'm also reliably informed that they have brought Krispy Kreme doughnuts,
• 1:00
  and those of you that would like to speak to them during the reception are very
• 1:04
  welcome to do that and also to ask more questions to our distinguished speaker.
• 1:11
  OK, so it's a great pleasure to introduce Yvonne Rogers and Yvonne Rogers is the chair
• 1:18
  of Interaction Design and also the director of the London Interaction Centre,
• 1:24
  both at UCLA. Her research interests are ubiquitous computing, interaction, design and HCI.
• 1:33
  VON has won so many awards that I will not name all of them, but I'll name a few.
• 1:38
  So she won this year's Royal Society Milner Award, which is really great, and MRC Suffrage Science Award.
• 1:46
  She's the holder of an absolute dream fellowship. She was co-chair of the Intel Collaborative Research Institute on Sustainable Connected Cities,
• 1:55
  and she's also the recipient of a Microsoft Research Outstanding Collaborator Award.
• 2:00
  Yvonne has a whole host of academic publications, but in addition to that,
• 2:06
  she's the co-author of a manifesto on HCI in 2020 and also co-author of Really Lucky Textbook In the Area.
• 2:15
  So her talk today is going to be about how our new technology is changing what we see, and I'm really delighted to introduce Yvonne.
• 2:26
  Thank you very much. Firstly, it's a pleasure to hear real claps and seeing yellow hands popping up on in front of the screen.
• 2:39
  And secondly, it's a real pleasure to see a sea of faces, albeit mostly masks.
• 2:46
  It's nice to see eyes rather than just a blank screen. Thank you very much for coming along to to hear the lecture.
• 2:53
  I know it's, you know, we're still a bit concerned about whether we should be coming out,
• 2:59
  so I really appreciate you coming to my lecture in terms of awards.
• 3:04
  My very first award was when I was six years old and it was at a brook bond tea for best handwriting.
• 3:12
  So you should see my handwriting now.
• 3:15
  But anyway, I'm going to be talking today about some of my research,
• 3:21
  which is very much about how can we design technology to empower and augment people?
• 3:26
  And that's what I've done with all sorts of technologies really thought about what is it about this technology that can make our lives better,
• 3:33
  but also what we do? And my talk is going to be in two parts.
• 3:37
  Firstly, I'm going to talk about how we might extend the body physically.
• 3:43
  And this is Joy Melford, who I used to work with at Apple many years ago on the left with her cyber gloves.
• 3:48
  And the second part of my talk is going to be about how we extend the mine digitally with software.
• 3:54
  And this is Andy Clarke, who's a dear friend and colleague of mine.
• 4:00
  And this was him on the front of the New Yorker magazine. Many of you might know him from his work in the extended mind.
• 4:07
  He's a philosopher at Sussex University.
• 4:11
  So when we think about extending the body, how many of you have thought about what it would be like to have a third eye in the back of your head,
• 4:23
  so you might rather swivelling round to see who's behind you? You'd be able to see who is sitting behind you and what they're doing.
• 4:29
  Wouldn't that be great if you could have a 360 degree vision?
• 4:35
  And philosophers have thought a lot about what it might be to have other senses other than what we were evolved from.
• 4:43
  And when I was at Sussex with Andy Clarke,
• 4:48
  we spent time talking about this and then I put it into the HSC for a grant looking at how we might extend our senses.
• 4:56
  And we were lucky to get funded, and the project that we we came up with was called the Essence Project,
• 5:04
  and we wanted to think about how you would make a third eye and what it might mean.
• 5:10
  So this work was done with Andy Clarke, John Bird and Paul Marshall, and we set about thinking about where would you want to have a third eye?
• 5:21
  So it might be that you might not want it in the back of your head because it's stationary.
• 5:25
  You can only see that, but supposing you had a third eye that was on your hand as a ring,
• 5:30
  you could then move your arm around like this and and under and over and get you to be in control of what else you might see.
• 5:39
  So this got us thinking and prototyping with the technology of the day, thinking about having a tiny camera that you could move around like this.
• 5:51
  And it made us think, Well, where would you be the what the camera captures?
• 5:57
  Where would it go?
• 5:58
  So of course, you can see it on a display, but you wouldn't want to be walking along with the display whilst moving your hand around.
• 6:05
  It makes no sense. And so what we did was to try and think about how we might translate the images that are collected
• 6:13
  by the third eye into a pattern that can then be translated into a matrix of vibrating motors.
• 6:24
  And these motors would then be placed on part of your torso.
• 6:27
  It could be you, the front or the back or your shoulders.
• 6:30
  And the idea is that there would be a pattern of vibrations a bit like you have on your smartphone,
• 6:36
  and that you would learn to sense those patterns of vibrations and be able to transform them into this other sense.
• 6:46
  And this work follows on from some early work 50 years ago by a philosopher technician, Becky Rita.
• 6:56
  And so we started to sort of think about this, but then we realised that it didn't make sense with this eye to be moving around.
• 7:05
  It would go upside down the orientation. You just get all this fuzzy feeling on you.
• 7:09
  So we went back a few steps to think about what would be the basic principles of having some kind of sensory substitution.
• 7:19
  And we thought about, can you see through your tummy?
• 7:22
  That was what we were thinking about and this is one of the researchers working on it, and he can explain it quite eloquently.
• 7:30
  John Birch one piece of a really interesting experiment that was doing over 40 years ago by Paul back in.
• 7:39
  So what he did was he took a camera and as he controls the vibration on the back and blind people.
• 7:49
  And after 10 years of annual training, they learnt how to interpret that vibration and recognise objects in the world around them.
• 7:59
  And so what if you have this idea of how the mind is extended and how the body is extended by using technology?
• 8:05
  So we've got a beautiful starting place with features built around technology and sensory substitution devices, or tDCS.
• 8:12
  So we use cheap off-the-shelf components with the idea that a lot of people can take our designs and run with them.
• 8:21
  So what happens is we have a webcam that looks down on a table and as objects that blindfolded and we put headphones on as well.
• 8:31
  And all they can feel is an array of vibration on that very effectively.
• 8:37
  As the ball goes down the table, they feel the vibration both on and off.
• 8:42
  It's on the left hand side of the table after they come down here or what I thought it would come down right.
• 8:47
  And I think of that as glass and tight. So you shouldn't get any vibration from it.
• 8:52
  But now you get a feel of oppression, vibration and get a foothold in this corner and it's going to make this motor, which is that one there.
• 9:01
  But then so that the computer programme has worked out where the bullet is attractive and it's made that much of us and therefore move along more.
• 9:13
  Your feel that the next much about right, if I'm ever home alone before the next March to my right,
• 9:19
  and that should be the bottom corner so that if you move out of it, you can get used to higher level changes when you move the ball.
• 9:27
  We also track where the subject at hand is. We do that, by the way, a bicycle glove, which is very trivial expertise to track.
• 9:34
  So as they move their hands, they'll feel a slightly gentler vibration really,
• 9:39
  because basically their job is to try and get their hands vibration to coincide with the full vibration.
• 9:46
  They do that to catch the ball, and people are remarkably good at it.
• 9:50
  So I've done a few trials. Most people can do it by. So that was our attempt to sensory substitution,
• 10:03
  and we went round to many schools and exhibitions and science museums and thousands of children and families
• 10:10
  tried this and we were really struck at just how quickly they were able to learn to see through their tummy.
• 10:17
  So they're able through those vibrations to be able to know where the ball was and to catch it without being able to see it visually.
• 10:23
  That was a very early project, but it inspired, I think, others to think about this.
• 10:30
  We don't want you again.
• 10:31
  And some people at the University of Osnabrück started thinking about a magnetic belt, which was part of the field space project.
• 10:39
  And what they had was to try and give people a sense of north by having this belt,
• 10:46
  which comprised of 32 tactile vibration vibrators with one of the vibrating north.
• 10:54
  As you walked around, it would be north of that was north of it. I have no idea because I don't know Oxford very well.
• 10:59
  But if you go this way, which way's north that way? Yes.
• 11:03
  So that's why that one would go off. And then if I started walking that way would go that way.
• 11:08
  So it was again this idea that you could learn to have this extra sense of through vibration.
• 11:16
  And what they found was that when they asked someone to wear this belt constantly for nine weeks, which can you imagine it would be quite heavy?
• 11:24
  You could take it off during a lecture or when you were stationary. They found that it improved their spatial awareness so that they were able to know
• 11:32
  where North was and also their understanding of of the campus and the surrounds.
• 11:40
  So there have been developments along these lines, but not only just using vibration as a way of extending our senses,
• 11:49
  but others have thought about what other kinds of digits or ways in which we can extend and then include who is an artist.
• 11:56
  Engineer has been designing this third thumb and basically, as you can see here, you attach it to your hand and you control it with your big toes.
• 12:09
  And what it does is you can control it, so it moves a bit like one of the other digits.
• 12:14
  Again, I think it's good to see the people who have been researching explain this research say Tamar has been
• 12:22
  working as a new neuroscientist looking at whether or not it changes the neural structure of the brain.
• 12:30
  By using this third thumb, this robotic extra thumb can be controlled by moving your toes.
• 12:37
  A prolonged use may come at a cost of your brain being less certain about how your hands work.
• 12:43
  Researchers gave this six people a prosthetic limb, the wrapped around their wrist and sat underneath their little finger and their thumbs.
• 12:50
  Movement was controlled by sensors attached to the user's big toes and communications were sent using.
• 12:55
  One is technology affixed to the wrist and ankle by wiggling each time.
• 13:00
  The augmented humans could move the thumb in different directions,
• 13:03
  and 28 series participants were encouraged to use them both in laboratory settings and in the wide world.
• 13:10
  The additional thumb could cradle a cup of coffee, while the same hands or fingers held a spoon to stir in milk.
• 13:16
  While some hospitals could use the phone to peel banana blow bubbles or even play the guitar.
• 13:22
  Don't stand how we get this purple tick supersonic.
• 13:27
  The video goes on for quite a while to explain how it can change the structure of the brain when you've got this extra digit.
• 13:36
  But for the purpose of today, I just wanted to show you how researchers are innovative in thinking about how you can provide extra limbs.
• 13:47
  But I don't think we've got to this point yet where having forearms, anyone know who the painter is?
• 13:55
  It says in the corner there it's Marguerite's 51 resources, but can you imagine having forums at dinner and how you would use those?
• 14:06
  But I think there's a lot of interest in thinking about if we are going to extend our bodies, how can we do that?
• 14:14
  And there's been interest in those who have disabilities or have had strokes or have lost a limb,
• 14:21
  and how the technology is being designed to enable them to feel like they might have those limbs, but also to have more.
• 14:28
  What I want to do now in the second part of my talk is to talk about how we can extend the mind through using digital technologies.
• 14:37
  And one of the first, I think, attempts to do this was a project that was called the Sixth Sense, and this was at the Media Lab at MIT.
• 14:48
  A guy, a student at the time, Pranav Mistry and his supervisor, Patty Maze.
• 14:55
  And he was experimenting with different technologies of the time to think about what it would be like to be able to project data into the environment.
• 15:06
  This was before smartphones, which we've all got used to as our way of looking at data.
• 15:11
  So the way in which he created this was to have a projector that was on on the
• 15:18
  would be on the cap and a camera and then some colour markers that would be
• 15:24
  different for the four fingers and that these would combine into a wearable gestural
• 15:30
  interface and allow digital information to be projected into the environment.
• 15:36
  And the way in which it does that is it recognises and tracks the user's hands through these
• 15:41
  four coloured markers and is able through computer vision to decide what action to take.
• 15:49
  Is it that they're taking a photo? Or is it as in this case, acting as a digital keypad, which allows the user to make a phone call,
• 15:59
  which at the time was considered to be, you know, amazing and it went viral?
• 16:04
  And I think again,
• 16:05
  this was quite influential in getting people to think about what data would we want to be projected out there and how might we use it?
• 16:12
  And one of our projects that followed on from the essence was to think about the
• 16:18
  technologies that we might use to be able to provide information in the environment.
• 16:22
  So we didn't use projectors. We used very straightforward, simple LEDs that would light up to show a certain amount of information the person needed.
• 16:34
  And what we did was we were trying to nudge people into deciding what objects to buy when they're in a supermarket.
• 16:40
  So what? They want to buy this water or that water, and sometimes you look to see whether it's come from, this one has come from a can't see.
• 16:50
  So whereas this was just from Oxford, the tap water.
• 16:54
  And we wanted to provide people with information that would help them make that decision.
• 17:00
  So we designed what was called the lemon shopping handle.
• 17:03
  And I don't know if any of you have watched the gadget man, but Richard Odey was very taken by our design and featured it on one of his programmes.
• 17:15
  So I'm going to let the video speak for itself. So I want to have some of this delicious looking spaghetti bolognese, right?
• 17:25
  20 percent. Yeah, that's a bargain that thought. Handling that, that he's as good as it looks like you.
• 17:32
  I never thought from anything you want to know the recycling packaging people do like.
• 17:36
  I think those are the exact things. I well, if you look at the map here from 13 states have Union Jack Big, which is already taking too long.
• 17:46
  What do they say? They at take here.
• 17:50
  OK, we're going to do.
• 17:51
  OK, well, I'll tell you what, I'm going to use the Lambert shopping cart handle on this handle as an experiment in lot of psychology,
• 17:59
  aiming to influence buyers choices. The more unhealthy or further the food is terrible, the more lights come on, the three things that light up.
• 18:08
  If it was very thought you'd light up all the way to this 30 days, roughly Scotland, how long did that take effect this quick?
• 18:15
  Oh, right, and how long ago? That was interesting. I'm not going to dispute how fascinating you are.
• 18:21
  I'm just saying the time management situation you're at.
• 18:27
  No wonder he's such a good presenter. So I think from these examples I've shown you,
• 18:34
  there's lots of technology we can experiment with and we've seen, you know, tangible physical computing.
• 18:41
  We can design all sorts of ways in which we can sense, but also present information.
• 18:47
  And then in the last one, the use of mobile and ubiquitous technologies, there are lots of software tools that we can develop and augmented reality,
• 18:57
  virtual reality wearables, speech robots, chatbots, A.I. and machine learning, multi-sensory service and so on and so forth.
• 19:05
  Every year, there's a new technology that's coming along not so many in the last two years that provide people like myself and
• 19:12
  others in the field of interaction design with new tools by which to think about how we can augment and empower people.
• 19:20
  And I'm not going to go through all of these.
• 19:22
  What I'm going to do is just talk about three themes that run through my work where I think about how we are
• 19:28
  augmenting people and using three different technologies and these three kinds of technology augmentation.
• 19:36
  I'm going to give case studies for each one.
• 19:38
  The first one is what I'm calling cognitive offloading, where the technology, usually some smart phone device,
• 19:46
  allows us to perceive and see more in the world because we're offloading onto that device.
• 19:53
  But the question is at what expense, and I'm going to describe a study we did to explore this.
• 20:00
  The second theme that runs through my research is to think about how we can scaffold learning and thinking,
• 20:08
  how can we design technologies to enable you to guide you and to probe you into thinking, perhaps differently, change your direction of thought?
• 20:19
  And we've been designing virtual agents chat bots to do this.
• 20:24
  And the third one is called what we're calling engaged interaction.
• 20:29
  Sorry, engaged intelligence. And this is to make us think with cognitive software tools,
• 20:35
  and I'll present a case study there from some people who are in the audience today on their work.
• 20:43
  So the first one cognitive offloading we're all familiar with using our phones for
• 20:48
  all sorts of tasks where before we have to remember whether it's people's birthdays,
• 20:53
  telephone numbers and addresses and so on,
• 20:56
  we just put stories into our phones and we know we don't have to learn them and we rely on our phones to to do the, you know, remember for us.
• 21:06
  So how are they changing how we see and remember? And I think that's the really interesting question.
• 21:11
  There's been a lot of report in the media that is telling us now we're not using our brains,
• 21:15
  they're getting a trophy because we we're just not exercising our mental faculties.
• 21:23
  So the one I started looking at was the the way in which digital map apps have changed,
• 21:28
  how we find our way through cities and what we remember of them.
• 21:34
  And I a few years ago did a study with one of my MSA students show you some.
• 21:38
  You are looking at how this was changing, the way in which we interact and perceive the world as we use them.
• 21:47
  So I don't know how many of you remember datasets or even still have one in a cupboard somewhere.
• 21:54
  But this is the paper based maps that we used to rely on all the time to get us from A to B.
• 22:00
  And the way in which we would use them was to we would learn how to read them and then match up what we're seeing with landmarks,
• 22:06
  which might be names of streets or certain buildings if they are mentioned with smartphone apps.
• 22:12
  There are many different varieties of these maps.
• 22:16
  You can follow the dot, the line or the.
• 22:19
  Person on the map, and it moves dynamically as you move, and it shows where you currently are and indicates where you need to go.
• 22:27
  So it's dynamic and it leads you. And you don't have to do this learning and matching up.
• 22:32
  So as I said, the cognition is offloaded onto the map and it makes the navigation task much, much more easier.
• 22:39
  Many of us will probably couldn't even imagine, especially Generation Z, what it's like to use a map.
• 22:47
  But one of the questions in the research has been Does it make us have poorer mental maps of the area?
• 22:54
  So we couldn't if we were dumped in it without smartphones, find our way home.
• 23:00
  But the research we were looking at was, well, maybe because you're not spending all your time looking,
• 23:05
  you're actually looking around and enjoying the surroundings, and maybe you remember more of the surroundings.
• 23:12
  So we conducted and in the wild study, much of the research up until then had been experimental in the lab.
• 23:18
  We wanted to actually see how people look around and use these devices in situ.
• 23:23
  And so rather than just measuring performance time or errors, we thought about measuring something else,
• 23:29
  which was the strategies that people employ when they're using a phone or a paper based map.
• 23:36
  So as I said, for a paper based map, if you're looking for, you know, Downing Street as we would all be wanting to do now is you,
• 23:45
  you look on your map, you see it and then you try and match up with what you're seeing it outside in the environment or you do the other way round.
• 23:53
  Whereas with a smartphone, you follow the dot or moving icon to see if it's still on track at choice points.
• 24:00
  And those choice points might be where you're making a turn.
• 24:05
  So we conducted a study of having 28 participants split into two groups to prevent training transfer,
• 24:14
  and they were asked to walk a route through London that they weren't familiar with using either a paper based map or smart phone one.
• 24:23
  And my poor MSE student shadowed all of these 28 participants each time they did this and made notes of what they were doing.
• 24:32
  This was because at the time we didn't have any sophisticated technology that they might be able to measure what they were doing,
• 24:38
  but also we wanted to see what they were looking at, which is quite difficult to do.
• 24:43
  Using some kind of a camera after they they followed, they were then asked to come back a week later and to do a recognition task.
• 24:53
  And this involved saying whether they'd seen certain things, and they also asked to draw a map of the route they took.
• 25:01
  So we used at the time a Google map, and we tried to control this to be similar as possible.
• 25:06
  So the digital one was printed, but it was static and the digital one moved as the person moved.
• 25:14
  There are problems with this design. I think it's really difficult to get a control in the wild, but we can talk about that later.
• 25:22
  So as the recognition task, they were asked to see, say, whether they'd seen these images on their route.
• 25:29
  So this was, you know, recall I remember seeing those and we put in a number which weren't as part of the route.
• 25:37
  So what did we find? Well, we did find a difference between the two conditions.
• 25:42
  We found that the paper map participants in that condition looked more frequently at the map,
• 25:49
  especially when they were coming up to a junction to check that they were doing the right thing.
• 25:53
  We're smartphone map participants. They looked much less and they usually looked after.
• 26:00
  They'd crossed the road or taken a turn to see if they'd done the right thing.
• 26:05
  So quite different strategies, as we predicted in terms of recognition, what they remembered, the smartphone group remembered more street views.
• 26:14
  Those are the ones which you showed the images because they were looking around and they also grew grew through more detailed maps,
• 26:22
  whereas the paper based drew more accurate maps because they had memorised it.
• 26:27
  So here's my only graph I'm going to show you is a shows on the left.
• 26:34
  A significant difference between groups. These street views in terms of the number that was remember the blue is the smartphone,
• 26:42
  so you can see they remembered more for the other ones the street names, the shops and the landmarks.
• 26:47
  There was a slight difference, but not significant. What was interesting is when we asked them to draw what they remembered.
• 26:55
  The top ones were the paper based map condition, and they drew more crossroads because they remember those and they were spatially more accurate.
• 27:04
  So using a paper based map, you do have a cleaner, better mental model.
• 27:10
  Whereas the smartphone map condition was more linear, more just a straight line with descriptions.
• 27:17
  So on this one, you've got Nandos really important place in the Polish embassy and posh houses.
• 27:25
  That sort of thing that you notice, so in terms of comparison of the routes described the terminology, the words used,
• 27:35
  they were far more subjective emotional descriptions in the sketches for the smartphone than for the.
• 27:43
  The paper based map and also using first person, so here, I don't know if you can read this,
• 27:48
  but for the smart phone condition, some of the descriptions that we use for things like beautiful houses.
• 27:53
  Nice flowers. Most flowers. Nice car pausch.
• 27:58
  Nice bar, cool shops. Posh houses. Boring, confusing houses.
• 28:02
  And so on. And over in the paper map one, there were far few.
• 28:07
  They did mention the beautiful houses.
• 28:09
  If you know that part of London, you know why they described it and the strange cars so well, this was an initial study.
• 28:18
  And what I think it shows is that the the use of these smart phone apps lends itself to people adopting different strategies,
• 28:29
  which is keep going, then check to confirm if I get it wrong, I just go back and walk again.
• 28:34
  Whereas for the paper base, you've got to read, you've got to stop, look around, match up.
• 28:40
  So quite a different strategy. And I think more generally what it shows is that there are trade-offs when we use these smartphone apps because,
• 28:49
  you know, you can rely on the tech analogy and you know, you don't need to learn the skill and so you don't.
• 28:58
  But the trade-off with this is that you don't remember.
• 29:02
  But the benefits, as we've seen,
• 29:04
  is that you will use the time to look around and enjoy worrying rather than worrying too much about whether you're going the right way.
• 29:13
  So those were that was an early type of phone app.
• 29:17
  Supposing now, as you see, there are more sophisticated types of smartphone map apps,
• 29:25
  and this one here is Apple's look around you, which has the schematic map.
• 29:30
  But then you can click and it will show you a video or a 3D image of where it is that you're looking at.
• 29:38
  So you can combine these two together. This is in Cupertino, where I once upon a time used to work.
• 29:45
  And so you can zoom in and then you can look at particular places before you go.
• 29:50
  And then you can look at them side by side with a street map.
• 29:53
  And I think I haven't done a study of that because I haven't been in Cupertino for a while.
• 29:58
  But it's interesting to think about how you can combine these and how does that affect and extend our cognition, our cognition?
• 30:07
  Do we look? Do we think about navigation and what we're doing in this space differently with these different types of maps?
• 30:14
  OK, so that was cognitive offloading has a lot more I can talk about for that,
• 30:18
  but I think I want to just move on to think a bit more about one of my other themes that I'm interested in, which is how do we scaffold thinking?
• 30:28
  In a way that helps us to to deal with problem solving and decision making when it's hard.
• 30:35
  And virtual agents, chat bots have been around now for many years,
• 30:40
  and they may be designed to answer users questions or customers questions and trials due to a particular page on a website.
• 30:48
  But recently they have been designed for other uses. There's one called Réplica, which is acts like a therapist,
• 30:57
  and you can interact with this therapist chatting away, typing in how you're feeling and so on.
• 31:04
  We were more interested in how we might use them as prompts or ways in which to guide your thinking.
• 31:13
  And the project I'm to talk about today is called the voice phase project.
• 31:17
  And this is one of my Ph.D. students, Leon Reichardt's, and we were very interested in helping teams of clinicians make sense of complex health data.
• 31:28
  We would be working for with Great Ormond Street Hospital for many years,
• 31:32
  helping them to think about how technology can improve what it's like to be working in a hospital.
• 31:38
  And this is one particular project. And so we developed interface agent called Physi and Vishy.
• 31:46
  The idea was to probe these teams to to guide the where to look on this graph.
• 31:51
  Where's the where's the place you should be looking to think about what's happening? Why is that those too close?
• 31:56
  And to get them to think a bit more about what's the causes behind this?
• 32:03
  So our agent, Rossi was designed to prompt teams into hypothesising about a particular set of data.
• 32:13
  The data that we looked at was obesity data, and obesity is something that's on the rise and has been for the last 30 years.
• 32:22
  And there's lots of data there, and sometimes it's quite difficult to know how to make sense of it.
• 32:27
  So we design visitors do not interfere too much or get in the way,
• 32:33
  but just prompt occasionally when it appears that they might be stuck to ask open ended questions, things like Do you need a hint for analysing X?
• 32:43
  What do you think of this? Shall we move on? Did you consider the differences between Variable X and Y?
• 32:50
  Did you consider what might have caused that sudden spike?
• 32:54
  So here's just an example of a visualisation. The teams could ask Veasey to bring up data for particular parameters,
• 33:03
  and here is four different gender girls and boys and for developing and developed countries.
• 33:09
  And the idea here was to try and think, why is it that that graph is going straight, which there's a sudden spike here?
• 33:15
  What might have caused that?
• 33:17
  And they can do the same for mixes of the dimensions so women and girls and see that the graphs are quite different there.
• 33:26
  So building a a virtual agent can take a long time. Those of you who work in natural language processing.
• 33:32
  And so in human computer interaction, we cheat in that we use what's called a Wizard of Oz approach and Boy Wizard of Oz.
• 33:42
  We have someone here who's a human but supes pretending to be the agent.
• 33:48
  And this is one of my students, Ethan here, and he is there controlling the interface and sends the prompts at particular times and chooses which one.
• 34:01
  And this allows us to manipulate different criteria much more easily than if we are programming it from to begin with.
• 34:11
  And so in this particular study, he actually as busy and would send prompts not very often,
• 34:18
  but occasionally if a team had not discussed a pattern or trend being referred to,
• 34:24
  and there was a silence in the conversation for at least three seconds. And he was able to control the ones that he sent for different conditions.
• 34:32
  We ran various experiments looking at whether if he was spoke to the team or whether he just appeared as a text chat bot.
• 34:42
  But for today, I'm just going to mention some of the key findings about Veasey, per say.
• 34:48
  So this is how it would look. You'd have a team here. This would be the smart speaker here.
• 34:53
  And then they would ask is to change the visualisations and they would talk about them and then visit my prompt to say, Have you thought about this?
• 35:00
  What do you think about the sudden spike? So I'm just going to summarise the findings rather than go through them in any detailed way.
• 35:07
  And we found that it encouraged the team to generate more hypotheses about obesity trends,
• 35:14
  and it got them to focus on the causes behind the different ways the Lions increased in the graph.
• 35:23
  So here's a quote from one of the participants.
• 35:26
  I like that the questions are finding out more about the data by answering the question or even by looking at it.
• 35:33
  You would think about the consequences, and then you would ask yourself,
• 35:36
  why is this line more steady than the other, which wouldn't necessarily happen without this system?
• 35:41
  So this work is just been published. We are very happy and these are my other colleagues in Tokyo.
• 35:48
  But here's an example of a conversation after a visit prompt.
• 35:52
  So there's a silence as they're looking at this state over here.
• 35:56
  This visualisation, which is a change in prevalence of obese girls and boys for developing and developed countries and physics,
• 36:04
  says what might have caused a sudden spike. And then you see, for these team participants, they go through a number of different possibilities.
• 36:13
  Is it globalisation getting lots of different foods?
• 36:16
  I think obviously computers, PlayStations, yes, children play less outside and overweight, and it goes on and on and.
• 36:25
  And then once they've exhausted the possibilities, they think, well, maybe we should look at how it varies for men and boys.
• 36:34
  So we got a number of examples of conversations that happen like this when there was busy prompting the when visit wasn't.
• 36:43
  So I think here it's just early days thinking about how we can design agents
• 36:50
  to augment our cognition and to guide sense making like that and ideation.
• 36:56
  And as we were discussing what we might do next, we thought, Well,
• 37:01
  maybe we might have to think about, is it good to slow down cognition rather than speeding it up?
• 37:07
  Because much of what we do is we designed software to make things more fast, more efficient, quicker to do.
• 37:13
  And we thought, well, actually, sometimes it's better to slow people down.
• 37:18
  And last summer we did a. He carried out a study with my colleagues who hear about 10 minutes looking at phishing scams.
• 37:27
  That is to say, I knew you were. You quite often just tap on a button, sorry on an icon or a click here and you've been zapped.
• 37:40
  I got caught last week, by the way, but that's another story. And so what we thought was having this type of agent would disappear.
• 37:49
  So you sure you want to click on it? Have you looked at this or that might be sufficient to slow you down to stop making those mistakes?
• 37:58
  So I think it's quite interesting to think when and what context is it?
• 38:03
  Would it be useful and to slow people down and their cognition rather than always speeding them up?
• 38:11
  So we're looking at a bit more in other contexts.
• 38:15
  The final case study, I went to a theme that I want to look at is what we're calling engaged intelligence, which is designing cognitive toolsets.
• 38:24
  And this is work with Wendy Jefferson and Anna Leslie, who were at Nasdaq as part of the behavioural science team,
• 38:31
  but now have left to become part of a start-up called Let's Think.
• 38:35
  So they're very interested in how you can design technologies to get people to
• 38:40
  think rather than letting A.I. take over and do the thinking on behalf of us.
• 38:45
  And one of the things that they did was that mass that was to develop new
• 38:48
  software tools for supporting complex investigative work in the finance world.
• 38:54
  And why is that? Well, you're probably all familiar with some dodgy dealings that go on in insider trading and other types of bad behaviour.
• 39:07
  And so they're very interested in how you can design technology to help people whose job it is to discover market abuse of this type or other.
• 39:18
  And the people who do this are called compliance officers and Nasdaq,
• 39:23
  and banks and other financial companies all have people who try to detect whether this is going on,
• 39:31
  and there are many different types of market abuse. Spoofing is one.
• 39:36
  And this is where there's a type of market manipulation,
• 39:40
  where the investor tries to move the price of a financial instrument up or down by placing a very large order.
• 39:49
  You're nodding your head, Nigel. Have you tried it? So once the price has been shifted,
• 39:57
  the original order is cancelled and new orders are placed on the opposite side to take advantage of the favourable price.
• 40:04
  So this happens and then many other forms.
• 40:06
  And so these compliance officers, their job is to try and find out if this has occurred and who is being responsible or involved in it.
• 40:14
  So they have to create and collate lots of data from several sources to conduct their investigations.
• 40:22
  And it's complex work to do, very labour intensive to get to the bottom of some of these spoofs.
• 40:27
  What they do is they start by examining what are called trading alerts,
• 40:32
  and companies like Nasdaq have these systems that are rule based engine that will search
• 40:38
  for patterns of behaviour that are associated with market abuse from previous histories.
• 40:44
  And they will fire an alert, which is one of these things here.
• 40:49
  When something potential is detected and it might be buy or sell order that occurs close
• 40:55
  to a news story that might have influenced the price and that then once they've got that,
• 41:01
  they go through these lists and they have to determine which need further investigation.
• 41:05
  Most of them, they just closed down because they're just perfectly normal. But some of them aren't, and those are the ones they need to detect.
• 41:12
  But to do that, it involves lots of steps. And this is a task analysis that when did it and you start there?
• 41:20
  I'm not going to go through all of this and you go there and there and then and there.
• 41:24
  And as you can see, is quite complex. The green is worth the experts can do yellows for people who have got middle expertise.
• 41:34
  So you can see you need to be quite well trained up an expert in order to be able to do this range of investigative work.
• 41:44
  So how do they do it? How do the experts, the green ones do this?
• 41:49
  Well, a lot of it is done in their head and they might use notepads to jot down their thoughts.
• 41:55
  They will scan through thousands of these alerts.
• 41:58
  They'll sift through millions of combs, emails, texts, and they'll check various news feeds and try and put this all together in the head.
• 42:06
  And it's a huge cognitive effort and lots of demand on their attention, and they need to switch between various resources.
• 42:13
  And they may forget that they've looked at that and then they'll look again.
• 42:16
  So there's a big opportunity here to think about how we might design some cognitive tools to help them with this type of work.
• 42:27
  So what kinds might we develop?
• 42:29
  Well, we're all well aware that air is usually the first thing that comes to people's minds as an opportunity to automate some of the processes.
• 42:41
  And indeed, I think A.I. has a lot to offer for the time consuming and error prone parts of doing this type of investigative work.
• 42:49
  But when do you and Anna and their team did was to think differently.
• 42:54
  They had to think differently, to think out of the box and to develop a toolset which would be integrated set of tools that would enable them to do
• 43:03
  all the stuff they do in the head much more systematically and to externalise it so they could share it with others.
• 43:09
  And more importantly,
• 43:10
  they could bring together all of disparate parts of the information and to make more connexions and help to build up a visual picture that,
• 43:21
  as I say, can be shared. So they developed all of these tools. I'm not going to go through them.
• 43:25
  This is still very much a work in progress, but I'm just going to mention two or three of them.
• 43:31
  But the idea is that these tools would allow the. Compliance officers to externalise rather than doing it in the head,
• 43:39
  and this would mean the training would be much improved and more people in your organisation could see and share what they thought was going on.
• 43:50
  So here are just three tools that they originally came up with.
• 43:53
  They've come up with some other since then. But the main one was the canvas.
• 43:59
  This would be a bit like if those of you who've used mirror in the last two years,
• 44:03
  it's like a big digital canvas that you can place tools on you can sketch on.
• 44:10
  You can have visualised sessions appearing side by side, and one of these might be what's called the network,
• 44:16
  which has allows you to see information between different people in a visualisation so that you can compare, compare and combine those.
• 44:27
  And then there's a case builder that helps you to build up your case rather than,
• 44:31
  it just notes jotted down and can be easily shared amongst the different offices.
• 44:36
  So this is a very early design. It's gone through many iterations since then.
• 44:42
  But just to give you a sense of here's the case builder where you build up,
• 44:47
  and these are some of the tools that you might place open close to show who's connected with whom.
• 44:54
  So you can start to have this visualisation and be able to see side by side, which is very different from how they currently do it.
• 45:03
  So I see with the time is.
• 45:06
  Coming to the end of my talk, so I'm going to finish with just summarising what I think this type of way of designing tools can do.
• 45:16
  Certainly, we know that I can help reduce workload.
• 45:20
  It can automate lots of things. But there's an awful lot of work that we humans still do and that we should still do,
• 45:26
  I should say, using our brains and not just hand over to machines,
• 45:30
  but what we should do is to provide and design tools to help humans do this type of
• 45:37
  investigative work and to help them to externalise the picture and help to design these took.
• 45:44
  It's a bit like how we used to use diagrams to show the Connexions, but to make them more sophisticated and to allow more team working to take place.
• 45:54
  And I think, you know, we may be working in the finance world, but I think this type of approach is very valuable for other kinds of detective work.
• 46:04
  And I'm sure the police would be very interested in this way of helping them to to find out and solve crimes.
• 46:12
  So I'm going to finish now. I hope through those three case studies,
• 46:16
  these three themes I've shown how we can augment through digital means our
• 46:23
  minds and that we shouldn't be handing over to A.I. to do everything for us,
• 46:28
  but we should be empowering and augmenting how we do them.
• 46:31
  And there are different ways in which we can do this.
• 46:33
  We can use agents, we can use these software toolkits and we can use what was the first one we did with mobile smartphones.
• 46:44
  But certainly things are games and things are lost. And I think more generally, it's quite interesting to think about that.
• 46:50
  So the benefits of designing technology to augment us.
• 46:55
  First one, I would call cognitive versatility. Thank you, Eva.
• 47:02
  If you're one of which is to it,
• 47:06
  may free up mental capacity through the cognitive tools that we develop and the
• 47:12
  various apps that we use on smartphones such that we can engage in other activities.
• 47:17
  We don't know which we have to be spending time using them. And that's the one with the the mobile phone apps.
• 47:24
  Then there's this idea of cognitive slowdown rather than cognitive speed up, which is counterintuitive to to many software developers.
• 47:31
  But we think at certain times in certain contexts, it enhances sense making capabilities.
• 47:38
  And then this third theme, which I think is really important, is cognitive empowerment.
• 47:42
  How can we help extend our thinking to see new connexions in data?
• 47:48
  And I think that's really important for us to have to focus on that. But there are worries.
• 47:55
  And this is something I think for these students to be looking at, which is what happens is if we lose our ability to think for ourselves,
• 48:04
  if we rely increasingly on the technology to do it for us, I think that scenario is what some people have been worried about within the A.I. world.
• 48:14
  But I think also with some of these other technologies, we need to think, is this going to happen?
• 48:20
  Another one is we might trust the technology too much and don't notice when it makes errors.
• 48:26
  There's been some research on some of the explanations that have been developed for A.I. that people rely on them too much,
• 48:33
  that they don't notice when it makes errors.
• 48:36
  And so there's another problem that we need to consider, and we might be led astray and draw the wrong conclusions.
• 48:44
  So we need to be mindful of what these potential concerns are when developing these tools.
• 48:51
  I want to finish off now thinking about the future.
• 48:57
  So you've all heard or probably sick to death of metaverse, and I'm going to leave that to Microsoft and Metta to fight out and come up
• 49:07
  with something that we would all find useful for a few minutes occasionally.
• 49:14
  I think the future, actually, we should be more putting our efforts into is augmented reality and augmented reality.
• 49:24
  For those of you who played Pokemon Go a few years ago was fun.
• 49:28
  It was playful. You went out with your friends and your family, and you went searching for things in the and they popped up all over the place.
• 49:37
  And I think now we have not just mobile phones with augmented reality apps, but they're starting to see glasses.
• 49:45
  So Snap came out with Spectacles earlier this year, and these allow you to design things to appear and to allow people to be creative and imaginative.
• 49:55
  But as I said, there are dangers with it,
• 49:57
  and I'm going to finish off with showing you the dystopian view of what augmented reality can do and the utopian view.
• 50:06
  So this one is by a video that was created by Matt Suda.
• 50:28
  Grade ninth grade.
• 50:35
  What is it, because you don't know how people can view this means will create jobs,
• 50:41
  so that means we must do all of opportunities, but I was stood in the way cars because we must try this.
• 50:49
  I think that assessment is not okay for these people having all these different kinds.
• 51:03
  OK, that's enough of that. I think I who give you a headache where it's just overload, you get efforts blasting at you,
• 51:08
  you just have to take your glasses off versus what Snap are trying to promote, which is a tool for creativity.
• 51:17
  And. As an artist, I'm constantly thinking about the different ways that I can tell stories and share my vision of the world with people.
• 51:31
  I first saw them and I was like, How has all this technology fit into something so small?
• 51:36
  The new are going to allow you to overlay anything you want onto the album.
• 51:41
  I like using technology to delight people and immerse them in a world that they could experience in reality,
• 51:49
  because I really wanted to use air in a therapeutic sense.
• 51:53
  Looking at the ebb and flow of water and sea creatures, I think that really inspires my creation.
• 52:02
  As a black artist, I feel like a lot of young UP-AND-COMING artists are not represented.
• 52:06
  I wanted to showcase their work in augmented reality, gathering in 3D and showing them in a lake that is beautiful and just inspirational.
• 52:14
  Now I can actually walk around and see what I'm capturing and call shots for sure of my friends.
• 52:19
  I feel like this generation is going to finally push the new digital frontier.
• 52:25
  This is going to change the game forever.
• 52:28
  So I love New Mexico, and I really wanted to sort of create a lot of fun in Mexico and be an immersive history lesson.
• 52:36
  You were standing on the rim of the bay caldera. I love these historic red times.
• 52:41
  There's sort of this funny, archaic things, and I love the idea that I might bring new life to this old way that we used to bring people in.
• 52:55
  I think air is about having a conversation with the world.
• 52:58
  I always go back to try to ask really essential questions, what makes us human and what makes us happy?
• 53:03
  What does it mean to be alive and communicating? I love poetry of a language about words.
• 53:09
  What would it feel like to walk through a poem last studio and the Swiss Army knife of augmented reality?
• 53:16
  I think I want to finish there to think about what would be like to walk through a poem, so I hope you've enjoyed it as a thank you.
• 53:25
  I hope you've enjoyed listening to some of my stories and research on how technology can extend how we see.
• 53:35
  But I'd like to thank all of my colleagues and researchers, but especially these who have worked on the projects I mentioned.
• 53:44
  And for those of you interested to, heterochromatin is for those, particularly women who can see more colours and the rest of us.
• 53:51
  So thank you very much.