Latest / EDGE AI POD / Cows Chewed Our Sensors And Still Taught Us About Edge AI

Transcript

Cows Chewed Our Sensors And Still Taught Us About Edge AI · EDGE AI POD

• 0:09
  SPEAKER_00: So I work in two universities in Tiny ML, HAI, so I'm very passionate about code.
• 0:16
  SPEAKER_00: Sorry.
• 0:18
  SPEAKER_00: So I'm very passionate about the subject and I'm very active in a way to bring Tiny ML HAI to the local communities, to end users, especially in UK.
• 0:32
  SPEAKER_00: The reason why I wanted to present this talk, it was last minute confirmation of my presentation.
• 0:42
  SPEAKER_00: But the reason why I wanted to do this talk because I feel this is a very important topic.
• 0:49
  SPEAKER_00: It could be potentially the future of HAI or even AI, decentralized AI moving away from the cloud and servers or centralized infrastructure.
• 1:03
  SPEAKER_00: Over the years, I've been working in pervasive computing area, and the motivation behind this area is to bring intelligence and AI, smart sensors to everyday environment from retail to agriculture to parks and home environment.
• 1:31
  SPEAKER_00: It happens that we had a project funded by the government which aimed to utilize 5G network in a big park.
• 1:41
  SPEAKER_00: It's actually a forest, Robin Hood Forest, Sherwood Forest in north of Nottingham.
• 1:50
  SPEAKER_00: But what happened is that the whole project was about utilizing 5G and showing innovation with 5G.
• 1:58
  SPEAKER_00: And we had a few telecom companies.
• 2:01
  SPEAKER_00: But what happened is that many of these companies couldn't set up a proper 5G to cover the whole park because the park was very, very big and the forest as well.
• 2:12
  SPEAKER_00: So then we had, and um my team were aiming to embed little devices that can communicate and motivate people to move and be physically active and interact.
• 2:23
  SPEAKER_00: We use some gamification approaches and so on.
• 2:27
  SPEAKER_00: So we had to find another way to communicate between these devices using you know 5G wasn't uh viable.
• 2:34
  SPEAKER_00: We wanted even to you know find uh uh um uh internet access to download our apps, our uh services, and so on, but it wasn't possible.
• 2:43
  SPEAKER_00: So that's why at that stage I started to think more and more about edge AI, on how can we utilize decentralized uh approaches to be able to process things away from centralized infrastructure.
• 2:58
  SPEAKER_00: Um but also beside that, one device wouldn't be enough.
• 3:02
  SPEAKER_00: If you have a large number of people coming to a park, you need, you know, to utilize all these devices.
• 3:07
  SPEAKER_00: You need these devices to communicate.
• 3:09
  SPEAKER_00: Like, for example, if you want to allow people to play a game and you want a scoreboard, you need some kind of a system where you can exchange information between these devices.
• 3:18
  SPEAKER_00: So you need this kind of collaborative system that can process and exchange.
• 3:23
  SPEAKER_00: So you need wireless communication and all of these tools.
• 3:26
  SPEAKER_00: And that's why we started to think more with my team about the uh decentralized AI approach.
• 3:33
  SPEAKER_00: And there are many names for it.
• 3:34
  SPEAKER_00: So you'll see during my presentation I call it distributed, I call it federated, I call it uh decentralized because it's still a very much a new area in its emphasis, and we're still trying to establish it.
• 3:47
  SPEAKER_00: And I realized that um I need many, many more days to add all the components that can that contribute to the system.
• 3:56
  SPEAKER_00: So you might find some slides slightly missing here.
• 4:01
  SPEAKER_00: So okay.
• 4:02
  SPEAKER_00: Uh this is just like a little illustration showing you like the vision in the future.
• 4:08
  SPEAKER_00: Obviously, you will not see HAI written on sheep in the future, but but at the moment, if you go to a farm north of Nottingham, you will see our callers with HAI placed on cows who are not behaving at all.
• 4:22
  SPEAKER_00: They have been chewing on our devices all day long and on the solar uh panels that we've um printed.
• 4:29
  SPEAKER_00: But that but that's the vision that HAI should be everywhere, but without you seeing the devices.
• 4:35
  SPEAKER_00: They should be embedded.
• 4:36
  SPEAKER_00: I don't know if you know the concept of disappearing computing.
• 4:39
  SPEAKER_00: So computing is there, but you don't see it, it empowers people.
• 4:44
  SPEAKER_00: And I I don't need really to tell you how many devices are in the world.
• 4:47
  SPEAKER_00: And I asked the question yesterday about like how many of these devices are utilized around the world?
• 4:52
  SPEAKER_00: Um we still don't have, at least in the UK, we don't still don't have a working wearable system, for example, for healthcare.
• 5:01
  SPEAKER_00: We don't have even a working wearable system for anything other than you know just a personal device that can be used for you know um general lifestyle.
• 5:10
  SPEAKER_00: So there are many devices.
• 5:11
  SPEAKER_00: Hardware, of course, can be if can evolve, but it there's there are a lot of chips uh out there, but we're not utilizing them.
• 5:18
  SPEAKER_00: And these are potential resources that can be utilized.
• 5:23
  SPEAKER_00: Oops.
• 5:24
  SPEAKER_00: How do I go back?
• 5:29
  SPEAKER_00: Okay.
• 5:30
  SPEAKER_00: So what happens if we bring all of these devices and try to utilize their capabilities?
• 5:37
  SPEAKER_00: Let's say, for example, we in this room we have, let's say, 20 wearable devices, whatever.
• 5:42
  SPEAKER_00: Each device has one megabyte.
• 5:45
  SPEAKER_00: Um together, let's say we have about two uh 20 megabytes of processing power.
• 5:52
  SPEAKER_00: Exclude about two megabytes for the communication, then you have 18 megabytes, which is much, much better for running a model than one megabyte or half megabyte.
• 6:02
  SPEAKER_00: So that's the idea.
• 6:04
  SPEAKER_00: What if we bring all of these devices together as a computational resource for resource sharing, but not just the memory, not just the processing power, but also other things.
• 6:14
  SPEAKER_00: You know, this device, for example, might sense something, the other device might might sense something else.
• 6:19
  SPEAKER_00: So they can complement each other's.
• 6:23
  SPEAKER_00: And that's why, again, we I mean collaborative learning has been there for a long time.
• 6:29
  SPEAKER_00: Distributed learning has been there for a long time, like for example, using um desktop devices to run something.
• 6:36
  SPEAKER_00: Or federated learning is there as well.
• 6:38
  SPEAKER_00: But the idea here is that if we have these edge devices doing their own thing, but then contributing to the overall model or learning, um then we have a bigger resource.
• 6:51
  SPEAKER_00: It's still not as big as data center, but it's still powerful enough.
• 6:56
  SPEAKER_00: And imagine if you have thousand, or like for example, in the case of uh football match where you know you have so many um fans with a lot of wearables, then you really have a considerable resource that can help at least in managing the event or the or um coordinating chants or you know songs and things like that.
• 7:18
  SPEAKER_00: Um so yeah, so what happens if we distribute these tasks across all devices or multiple devices, um, multiple sensors, and then compress these features and make them float between these uh devices, and then you have all of these resources operating as one, reducing um memory per node, and then um you have a um a model, not necessarily bigger model.
• 7:44
  SPEAKER_00: So you have, I mean, I'll explain later that how you have um a local model and uh global model.
• 7:52
  SPEAKER_00: So again, uh just to put more emphasis on uh collaborative learning, because you could do distributed learning, each device can do its own thing, uh, and this device is identical to the other ones, or each device can do something different.
• 8:07
  SPEAKER_00: So take the case, for example, in the case of search uh uh and rescue scenario, if you have like a number of people trying to find a missing person, right?
• 8:17
  SPEAKER_00: So you could um ask um 20 people to go uh in the field and find um um a lost child or a person.
• 8:26
  SPEAKER_00: They all could have a camera, an audio exchanging information, they all could be looking for a specific object or face recognition.
• 8:33
  SPEAKER_00: But what if each device can look at um the scene from different um angles?
• 8:42
  SPEAKER_00: And they work like uh to build a jigsaw so each one can find, like for example, this part of the body might be occluded where the other part is visible to the other person.
• 8:52
  SPEAKER_00: So collaborative learning could be complementary, they could complement each other's, and then one angle is uh uh like the the body is um visible from one angle and then uh it might not be visible from the other side.
• 9:06
  SPEAKER_00: So that's the idea.
• 9:06
  SPEAKER_00: How can we do that in a way that each device can do something that is complementary to the other devices?
• 9:16
  SPEAKER_00: But then also we could do parallel processing not just on devices, we could do parallel processing on the device itself.
• 9:25
  SPEAKER_00: So, for example, if you are using uh dual-core processor um with TPU, let's say, for example, or MPU or extra uh processing units, you could uh distribute the model on all of these devices, on all of these processors, right?
• 9:41
  SPEAKER_00: So, for example, on ARM processor, you could use um the M4, M7, or in the case of the Google Coral, you could use the TPU.
• 9:49
  SPEAKER_00: And we've done that in multiple projects.
• 9:51
  SPEAKER_00: And this could save time.
• 9:52
  SPEAKER_00: So you don't want some of these uh processors to be idle while you are processing.
• 9:56
  SPEAKER_00: You want to utilize each resource as much as possible because you really need these extra few bytes or these extra uh processing powers.
• 10:10
  SPEAKER_00: And and there are many types uh for parallel processing.
• 10:13
  SPEAKER_00: I mean, you could, for example, do uh data parallel processing where each uh processor, for example, processing the same data, or um you could uh do a modular kind of processing where like each each processor is taking part of the model and processing, and then um then uh all the features uh uh and the out uh output of the uh initial model are fused to uh give uh an overall model.
• 10:41
  SPEAKER_00: Um this meant to be for later.
• 10:45
  SPEAKER_00: And then, of course, I mean we can't talk about this without talking about generative AI, right?
• 10:49
  SPEAKER_00: Because um everybody wants to do generative AI.
• 10:52
  SPEAKER_00: But again, you know, like generative AI is very much um interesting, but also power-hungry, and it needs uh you know um much much more uh processing capabilities.
• 11:03
  SPEAKER_00: But again, here we have a better chance at utilizing wearable and distributed devices to run generative AI.
• 11:09
  SPEAKER_00: I have many students at the moment working on this.
• 11:11
  SPEAKER_00: I mean, this is still work in progress, but you could uh imagine that you uh distribute, like for example, you have a local domain knowledge, so this device is collecting different types of um information than this one, so you have different domains interacting with each other's.
• 11:26
  SPEAKER_00: You might have different prompts.
• 11:28
  SPEAKER_00: So just imagine, like, for example, here with all your devices collaborating and giving me feedback, or like writing a story together, right?
• 11:36
  SPEAKER_00: So everybody will be using different prompts, but collectively we will build a story together.
• 11:42
  SPEAKER_00: And I think that's that's very interesting for collaborative uh interaction, and that could be generative.
• 11:48
  SPEAKER_00: Um I think I have a better slide for this.
• 11:51
  SPEAKER_00: So the output will be localized summaries, actionable insights, and natural commands.
• 11:59
  SPEAKER_00: But of course, I mean generative AI is one way around it, but you could do any other kind of models, you could do multi uh uh different modalities, you could use uh sensing, you you could do vision, and all sorts of things.
• 12:13
  SPEAKER_00: Doesn't need to be generative.
• 12:14
  SPEAKER_00: And there are many, many ways to do um decentralization.
• 12:19
  SPEAKER_00: Like, for example, in some of the projects we have heterogeneous systems where we have different edge devices that are either wearable, they are drones, there are robotics, or they could be the same.
• 12:29
  SPEAKER_00: Like, for example, when I mentioned the case of wearable in here, everybody has, for example, a photo or has a wearable device.
• 12:35
  SPEAKER_00: Uh so it could be the same type of device or multiple types of devices, they could be identical.
• 12:41
  SPEAKER_00: Each device can do something different or identical.
• 12:43
  SPEAKER_00: I mean, uh, and also doesn't need to be big.
• 12:45
  SPEAKER_00: It could be one-to-one, for example, collaborative system, like for example, two people um walking around the park, having a conversation, trying to explore a topic.
• 12:55
  SPEAKER_00: Um, but there's an important element to this that normally the HAI community doesn't think too much about it, which is communications.
• 13:04
  SPEAKER_00: Because if you want to do this, you really need to think about communications, wireless communication in particular.
• 13:09
  SPEAKER_00: So we're saying that we're not going to rely on the infrastructure, we're not going to connect to the cloud.
• 13:13
  SPEAKER_00: So, how do we communicate between these devices?
• 13:15
  SPEAKER_00: So we need some uh form of communication, wireless in particular, short range or mid-range, but it has to be low power and has to be uh robust enough to enable that communication.
• 13:27
  SPEAKER_00: And uh it it it also has to be available in a way that uh let's say uh uh robust, like uh I don't know if you know the concept of opportunistic networking, where you know uh not come to the network and disappear every now and then, or ad hoc networks uh or delay tolerant networks.
• 13:48
  SPEAKER_00: These are old concepts in communications, but they are very important for uh these types of scenarios.
• 13:54
  SPEAKER_00: Then when you start thinking about communication, you need to think about where to process, where to handle the communication, because you could run the models on the device itself, but then where do you handle communications?
• 14:09
  SPEAKER_00: Still, you need the the processor, the same processor, for example.
• 14:12
  SPEAKER_00: So some of the the things that we try to do, like to run a model on one processor and handle the communication on another processor, again you are using some kind of parallel processing, but for different um tasks.
• 14:28
  SPEAKER_00: And and beside the the physical wireless communication, we need to think about the communication protocols, what kind of protocol that we should use to enable this type of interaction.
• 14:40
  SPEAKER_00: On top of that, uh you have we we we think about continual learning, adaptive learning.
• 14:45
  SPEAKER_00: And and this is an important part of distributed systems.
• 14:50
  SPEAKER_00: Uh it's not like, for example, for other cases of wearable, you could use about personalization, you could choose to do uh continual learning.
• 14:57
  SPEAKER_00: But in the case of distributed learning, the decentralized AI, we have to do it.
• 15:01
  SPEAKER_00: And the reason for that is that uh we have to um manage or overcome uh catastrophic forgetting.
• 15:09
  SPEAKER_00: Uh as a concept where uh, for example, the the model has learned something, and then uh you update the model with new data and it might forget the previous uh learning.
• 15:23
  SPEAKER_00: So we need to make sure that the model can be updated but without forgetting the previous learning.
• 15:28
  SPEAKER_00: And in our case here, if you have a lot of nodes, um maybe I could I could show how this looks like just to explain the concept.
• 15:53
  SPEAKER_00: Yeah, so so you will have a local model to do the local data processing, but then you will have a global model.
• 16:03
  SPEAKER_00: The global model is really floating uh across all the devices.
• 16:06
  SPEAKER_00: And here, um in the global model where we do the continual learning, where we do we update the model, whether it's like um updating or retraining multiple layers or the five fine-tuning the final layer.
• 16:20
  SPEAKER_00: Um, and that's why continual learning is very important in this case.
• 16:24
  SPEAKER_00: I will go back to where I was very quickly.
• 16:29
  SPEAKER_00: So um because of the time limitation, I'm not going to be able to explain everything, but I will give you some examples of some of the things that we've been doing in my uh lab.
• 16:41
  SPEAKER_00: Um if I go, can I go back?
• 16:44
  unknown: Yeah, okay.
• 16:45
  SPEAKER_00: So, in terms of parallel processing, we've been using regular um uh microcontrollers to test if we can, like, for example, do multitasking.
• 16:55
  SPEAKER_00: Um, and we we have a paper, and we could read about our work in terms of uh parallel feature fusion um accepted recently for publication.
• 17:05
  SPEAKER_00: Um the idea here is that you have one processor um running um image uh model um CNN and another running audio CNN, and then you fuse everything and process the fusion on the M7, for example.
• 17:23
  SPEAKER_00: So this is quite interesting because you could really save some time and reduce latency.
• 17:28
  SPEAKER_00: Um but one of the key issues that we have faced, and that's why it's very important for us to work with um hardware designers, is that we've noticed that there's no shared memory between the processors.
• 17:39
  SPEAKER_00: So we had to find um um uh a way around it, you know, like we had to hack it uh using um uh procedure call, back call to be able to um communicate the parameters between the uh different processors.
• 17:54
  SPEAKER_00: But this is something very important.
• 17:55
  SPEAKER_00: If we want to do parallel processing, we need this type of shared bit memory, we need the hardware design to take into account that people will be able to process multiple things at the same time and maybe even have having more processors, maybe four processors or more on the regular um microcontrollers.
• 18:14
  SPEAKER_00: And in this in this particular paper, we saved around 48 milliseconds by just uh doing this.
• 18:21
  SPEAKER_00: So I will rush a little bit here.
• 18:23
  SPEAKER_00: Um we we applied similar things uh for for uh agriculture.
• 18:27
  SPEAKER_00: We have multi uh tasks on microcontrollers um in the farm, our devices under in the field, we have LoRaWAN, uh we use uh ST has very good um um uh microcontroller with LoRaWAN interface.
• 18:42
  SPEAKER_00: So sometimes our choice of microcontrollers, not the baby or processor, it's the um wireless interface or the other sensors that are available on the device.
• 18:51
  SPEAKER_00: And another big problem, I mean, we talked about this, the data sets issue, where to find the data sets for the students to work on.
• 18:58
  SPEAKER_00: I mean, this is this is completely a different problem, and we struggle with this all the time.
• 19:02
  SPEAKER_00: And especially with distributed systems, like where do you get the data?
• 19:05
  SPEAKER_00: How do you test this system uh and and where do you test it?
• 19:10
  SPEAKER_00: Um also we we we did some experiments to see you know the running time on different processors, and then going to the concept of distributed system very briefly.
• 19:21
  SPEAKER_00: This is the typical uh pipeline for HAI.
• 19:25
  SPEAKER_00: Um and then I explained this briefly before.
• 19:29
  SPEAKER_00: Um again, we have a paper on the archive in this area to explain the concept, but uh more or less you have um model that tries to converge on multiple devices, and in this case, we assume we have hundreds of devices, many clusters, intra-clusters, interclusters, and we average uh the so we use uh federated average uh to uh inform uh the different clusters.
• 19:55
  SPEAKER_00: Um and the initial results showed very, very promising uh progress, but obviously it depends on the data set.
• 20:05
  SPEAKER_00: I'm just rushing here, and we applied this in in many, many um similar scenarios, like what I mentioned before in the park.
• 20:12
  SPEAKER_00: This is the park I mentioned earlier.
• 20:14
  SPEAKER_00: Uh, we've been working with the Met Police because this is very important to be able, like for example, to find um uh someone in high street, you know, by by uh the the policing services, or for example, for um uh traffics or uh vehicular uh networks.
• 20:34
  SPEAKER_00: Um again we have another uh strand of work.
• 20:41
  SPEAKER_00: And let's see the last.
• 20:44
  SPEAKER_00: Sorry, this is yes, just just to uh um finish this with to say this is quite interesting, um, but it's it requires a lot of work.
• 20:52
  SPEAKER_00: That's why we try to uh tackle many, many issues, whether it's the continual learning, whether it's the data sets, whether it's the um, you know, like where where do you run this model and um how do you uh exchange the information, how do you deal with the gossip uh protocol or do you use different protocol for the communication?
• 21:13
  SPEAKER_00: But our biggest challenge, and I I mentioned this uh to many of you, is the test bit.
• 21:19
  SPEAKER_00: How do you test this?
• 21:20
  SPEAKER_00: How do you evaluate it?
• 21:21
  SPEAKER_00: How do you um run the model?
• 21:24
  SPEAKER_00: Because you need hundreds of devices to be able to do this.
• 21:26
  SPEAKER_00: So sometimes we do simulation, but sometimes we use a small number of devices to run this.
• 21:31
  SPEAKER_00: So this is quite difficult, so it's it requires more attention from the community.
• 21:36
  SPEAKER_00: Very briefly before I finish, um we have some upcoming events, uh, the retail event in July, we have um wearable, AI unwearable, uh digital health in UK.
• 21:52
  SPEAKER_00: Um I'm sharing this event, and then we have HAI Japan, Tokyo in October.
• 21:59
  SPEAKER_00: If you are interested, please let me know.
• 22:02
  SPEAKER_00: And finally, we have a new uh funded network uh called TinyML UK, uh where we will have a network manager to complement what HAI Foundation is doing to bring the uh HAI community in UK to collaborate with the wider community.
• 22:18
  unknown: But uh that's all.
• 22:19
  unknown: Thank you.