Latest / EDGE AI POD / Deploying TinyML Models at Scale: Insights on Monitoring and Automation with Alessandro Grande of Edge Impulse

Transcript

Deploying TinyML Models at Scale: Insights on Monitoring and Automation with Alessandro Grande of Edge Impulse · EDGE AI POD

• 0:05
  Speaker 1: Cool, awesome guys.
• 0:07
  Speaker 1: I'm excited to be here and if you don't know me, I'm Alessandro Grande.
• 0:12
  Speaker 1: I'm heading product at Gimpulse.
• 0:14
  Speaker 1: As Andrea mentioned, I was chair of this great community and this great event last year.
• 0:20
  Speaker 1: It's great to be back and to have a chance to tell you more about what we're doing at Edge Impulse and some of my thoughts on how we should look at evolving this field in the near future.
• 0:32
  Speaker 1: You see on the left here a slide of me showing, actually, one of the first demos of TinyML while I was still at Arm.
• 0:40
  Speaker 1: It took me a long time to get that demo working.
• 0:41
  Speaker 1: It was not easy at the time.
• 0:42
  Speaker 1: Took me a long time to get that demo working.
• 0:44
  Speaker 1: It was not easy at the time.
• 0:46
  Speaker 1: It was a few years ago and this community didn't exist yet, or maybe it just started at that point and this stuff was hard.
• 0:55
  Speaker 1: I think we've gone a long way and I'll try to tell you a bit more about that in a second, but I still hear a lot of this, right?
• 1:01
  Speaker 1: I think yesterday a lot of the presentations were around focusing on what models are we building?
• 1:08
  Speaker 1: You know, how do we optimize these models?
• 1:09
  Speaker 1: How do we make them smaller, better?
• 1:11
  Speaker 1: It's great, right?
• 1:13
  Speaker 1: I mean, we're doing a lot of this still, I think we should all do a lot of this, but is this the whole tiny ML workflow?
• 1:19
  Speaker 1: Is that the only thing we should think about, or is there more, right?
• 1:22
  Speaker 1: I mean, I'm curious.
• 1:31
  Speaker 1: Hands up, if you think this could be the end.
• 1:32
  Speaker 1: Okay, I'm glad to hear that we're all on the same page.
• 1:34
  Speaker 1: Right, there is more, but not many people are talking about this, or not many of us are thinking about this, and it's something that fortunately starts coming up right in conversation.
• 1:44
  Speaker 1: So it's good to address this as a group, I think.
• 1:48
  Speaker 1: Why is this important?
• 1:49
  Speaker 1: Because the real world is messy, it's fuzzy, it's complex.
• 1:53
  Speaker 1: We have a lot of sensors, a lot of data, a lot of things coming in the system, right, and we need to analyze it, and a lot of times, it's not as easy as saying, okay, I'm going to build a model, you know, in my test lab and get it off and it's going to work.
• 2:10
  Speaker 1: It doesn't always work like that.
• 2:11
  Speaker 1: What could possibly go wrong?
• 2:12
  Speaker 1: There's a lot of things that could go wrong.
• 2:14
  Speaker 1: It could explode in your face, right?
• 2:15
  Speaker 1: It could completely stop working or, even worse, it could do things that you're not expecting it to do.
• 2:23
  Speaker 1: Think if you are putting a sensor with smarts in a car that is meant to brake when you press the brake or when it sees even worse when it sees someone in front of the car autonomously.
• 2:36
  Speaker 1: If it doesn't work because the sensor changed, the data changed, the person that you're in front was different from what the model was trained on it could go really wrong, and I would hate if we were in a position where we would start seeing news like this coming out.
• 2:57
  Speaker 1: This is obviously generated by ChatGPT.
• 2:59
  Speaker 1: It's not real, but I think we have a responsibility as a group to think about how we make sure that what we build is not a one-off thing, right?
• 3:09
  Speaker 1: It's not something that just live and dies with the system we're building, because we don't actually know how long this stuff will be in the field for right and where it will actually end up being.
• 3:21
  Speaker 1: And so I really think that as a group and I think myself, I have a responsibility, and I think we all do to make sure that what we build is actually built for scale.
• 3:33
  Speaker 1: We need to think about the whole flow, right, and actually the whole flow is a continuous flow.
• 3:39
  Speaker 1: There is no end to this flow.
• 3:41
  Speaker 1: Unfortunately, bad news, it's not easy.
• 3:43
  Speaker 1: It doesn't come as a single shot.
• 3:46
  Speaker 1: Ml is not something that you do once and then you forget.
• 3:50
  Speaker 1: It's something that you need to improve over time, and sometimes it's not your fault.
• 3:56
  Speaker 1: The model might be perfect at first, but then it stops working, and so you really need to think about this as a cycle, and this cycle is forever until the product lasts.
• 4:08
  Speaker 1: Fortunately, we're starting to see customers appreciate this.
• 4:13
  Speaker 1: Partners that we work with tell us that this is the direction they're going for, and, as you can see here, one of the main things that the CEO of Hive actually told us is that they appreciated the fact that they can now deploy in more settings, in different environments, and they could continue this loop over time.
• 4:41
  Speaker 1: How does this look like in the real world?
• 4:43
  Speaker 1: What are the solutions?
• 4:44
  Speaker 1: What are the solutions?
• 4:45
  Speaker 1: What are the approaches?
• 4:48
  Speaker 1: I'll start by kind of just plotting again this cycle with this additional block, right called model monitoring.
• 4:55
  Speaker 1: We call it model monitoring.
• 4:57
  Speaker 1: It's this idea of not only having the ability to change what you're building or what you built, but actually monitor what you built in the first place.
• 5:06
  Speaker 1: You can only change and adapt if you know that something is wrong, something has gone wrong or something is diverging from your expectations.
• 5:13
  Speaker 1: So it's really important to have this concept of monitoring the lifecycle of a device, of a model, and being able to adapt as things change.
• 5:22
  Speaker 1: So, yeah, I'll focus on this first, because I think this is one of the main aspects of a model, and being able to adapt as things change.
• 5:25
  Speaker 1: So, yeah, I'll focus on this first, because I think this is one of the main aspects, and then I'll focus on another aspect later that I also think is important in this workflow.
• 5:33
  Speaker 1: So how does it look like in the real world?
• 5:35
  Speaker 1: It's actually much more complex than that, right?
• 5:38
  Speaker 1: Usually, we deal with deployments, where things get deployed through an IoT platform of some kind.
• 5:45
  Speaker 1: They get deployed to multiple devices in the field, fleets of devices, and all these devices are different.
• 5:52
  Speaker 1: So, actually, what's really hard is not just one model.
• 5:55
  Speaker 1: In a lot of cases, it's personalized models per each device, right, or per classes of devices, and so the problem is really a problem of scale, right, and you need to make sure that whatever you're building, whatever you're developing today, you have scale in mind from the start, because otherwise the challenge is that you might get stuck later.
• 6:17
  Speaker 1: And so, focusing on this aspect, I wanted to show a use case.
• 6:22
  Speaker 1: I wanted to focus on a use case specific to this flow.
• 6:27
  Speaker 1: Okay, we have a customer, a user that is developing, is doing production of thermostatic valves, and they're looking for defects in the production line.
• 6:41
  Speaker 1: We built this new FOMO AD architecture that is doing visual anomaly detection.
• 6:47
  Speaker 1: The cool thing is it can do it on microcontrollers, so TinyML, and it's super interesting.
• 6:53
  Speaker 1: It works right.
• 6:55
  Speaker 1: But what does it look like in production?
• 6:57
  Speaker 1: Well, someone has a production line.
• 6:59
  Speaker 1: They see a fault, right.
• 7:01
  Speaker 1: So what happens is faults get detected, right, and they all get labeled, as you see in the orange, as anomalies, right?
• 7:05
  Speaker 1: So what happens is faults get detected, right, and they all get labeled, as you see in the orange, as anomalies, right?
• 7:07
  Speaker 1: So you get anomalies back in the system and you can decide what to do with those anomalies.
• 7:13
  Speaker 1: You can look at the data.
• 7:15
  Speaker 1: Here is a section of what our data explorer looks like.
• 7:19
  Speaker 1: It's basically embeddings of the model and it shows the clustering of the model, and the different clusters mean different things.
• 7:28
  Speaker 1: So you see, there is a nominal set of data and then there's all these anomalies that have been detected in the field.
• 7:33
  Speaker 1: Ok, what do you do next?
• 7:36
  Speaker 1: So what we've done, for example, in this case, is actually we know that these, or we can look at this data and actually identify these different anomalies.
• 7:45
  Speaker 1: So what you can then do is you can change the model, you can improve the model to actually now be able to detect these anomalies.
• 7:51
  Speaker 1: Not only say it's an anomaly, but actually say you know it's anomaly, there's a crack or there's a missing cap or there's a missing screw in the valve, and this is just an example of you.
• 8:05
  Speaker 1: You know in practice what it looks like, right, you gather new data, you put it into the system, you understand the problem better and then you redeploy.
• 8:13
  Speaker 1: What's really cool is that we're working on on a set of new features to actually enable this in production more easily.
• 8:19
  Speaker 1: One of these is is the ability to look at stream of data coming in from a device and inspect effectively the data and be able to see these anomalies in real time.
• 8:29
  Speaker 1: Another thing that is super important that customers and users are looking for is this ability to look back in time.
• 8:39
  Speaker 1: Right, because imagine you're at the valve production line and something gets flagged as an anomaly, but now that anomaly has gone somewhere, right, like it's on the production line and it's just passed.
• 8:51
  Speaker 1: How do you know what happened?
• 8:53
  Speaker 1: Why that flag?
• 8:54
  Speaker 1: You know why that thing was flagged as an anomaly?
• 8:57
  Speaker 1: You need to look back in time at things that happened and you need to be able to inspect and decide what to do.
• 9:05
  Speaker 1: Sometimes anomalies are your model not performing well.
• 9:08
  Speaker 1: Sometimes they're actual anomalies.
• 9:09
  Speaker 1: You need to be able to inspect that.
• 9:13
  Speaker 1: And then what's really important is having this ability to package these models and deploy them for different devices.
• 9:19
  Speaker 1: Sometimes what happens in the field I mentioned fleets of devices.
• 9:22
  Speaker 1: These fleets of devices might be different in form factor, in type, in architecture, and so you need to be able to scale your deployments in a seamless way.
• 9:36
  Speaker 1: Obviously, there's a whole OTA piece right.
• 9:40
  Speaker 1: Edge Impulse does not do OTA.
• 9:42
  Speaker 1: I really believe in the power of the ecosystem here and I think there is a lot of companies out there that are doing OTA, and so we're partnering with other companies, like some of them are in this room, to actually scale the ability to do these deployments on fleets of devices in real time, on different types of devices in real time, on different types of devices.
• 10:04
  Speaker 1: You can see some of the names here.
• 10:06
  Speaker 1: It's really an ongoing process because actually, as you know, it's a very fragmented world and so we're really trying to build the infrastructure to make this happen.
• 10:15
  Speaker 1: And then it's about like partnering and making it happen together.
• 10:19
  Speaker 1: It happen together.
• 10:25
  Speaker 1: Again, another example of a quote from one of our users it's interesting that part of this effort is on optimizing these models, on making them smaller, more optimal for the hardware, and this is possible thanks to all the technology that we're building together.
• 10:43
  Speaker 1: So it's just another proof point that we're going in the right direction.
• 10:47
  Speaker 1: Cool.
• 10:50
  Speaker 1: Next piece I want to focus on is actually the data.
• 10:52
  Speaker 1: Right?
• 10:52
  Speaker 1: I think this was mentioned a few times Data is king.
• 10:56
  Speaker 1: Models only work if you have the right data, if you have enough data, if you have the right diversity of data.
• 11:02
  Speaker 1: And this is a you know, it's not only a challenge to build the first model, but it's also and it stays a challenge across your you know again the lifecycle of the model, because, as there is data drift, right, you need to collect more data and you need to be able to adapt the model.
• 11:18
  Speaker 1: So it really remains an issue throughout the lifecycle of that product.
• 11:26
  Speaker 1: This is another kind of use case, another class of use cases.
• 11:29
  Speaker 1: Here I'm showing the kind of workflow for health use cases where actually one of the big aspects is really that ability to collect data from different participants, different users, at scale, so that you have a diversity of data, but also the ability to personalize.
• 11:48
  Speaker 1: And so you know, here I'm gonna focus mostly on on this piece, right?
• 11:54
  Speaker 1: So the data collection and preparation and and there are really actually before getting into that, yeah, the use case specifically is HIFE, hife AI.
• 12:04
  Speaker 1: They do cough monitoring, they're able to detect different types of coughs and, again, they're able to do this at scale on multiple types of devices.
• 12:14
  Speaker 1: You see, here they're actually deploying to a suite of devices and they're looking to do more of this type of deployment to different kinds of devices in the future, and so it's really important that they are able to collect a diverse set of data to be able to then deploy these applications in the field.
• 12:34
  Speaker 1: Again, another quote here, so interesting here they're mentioning that effectively, you know it's a complex use case, right.
• 12:46
  Speaker 1: Again, this goes back to like the fact that data is the world we live in, is a complex world and really sometimes like data is hard to collect and it's hard to manage and so on.
• 12:57
  Speaker 1: So this is again showing that, effectively, these applications are happening right.
• 13:02
  Speaker 1: Like the reason I'm showing these quotes is that this is again showing that, effectively, these applications are happening right.
• 13:03
  Speaker 1: Like the reason I'm showing these quotes is that this is real, right.
• 13:05
  Speaker 1: Like it's not, these are customers in the field with real products.
• 13:11
  Speaker 1: The three things that I always think about when I think about data are effectively preparing the data well, identifying outliers or anomalies or any kind of things that should be looked at, and then labeling the data.
• 13:26
  Speaker 1: And what's interesting again, this is another loop, not only because you want to do it across the lifecycle of the product, but actually because it's not a one-off itself and you have to really think about scale in this loop as well.
• 13:39
  Speaker 1: First thing, data preparation.
• 13:43
  Speaker 1: We've done a lot of work on trying to identify patterns and it comes down to effectively checks Based on the use case.
• 13:52
  Speaker 1: Here we're showing a few different checks for some health customers.
• 13:55
  Speaker 1: For example, are the files present?
• 13:58
  Speaker 1: Are the labels present?
• 14:02
  Speaker 1: Did all the files start?
• 14:06
  Speaker 1: Dates or times are within the first 10 minutes?
• 14:08
  Speaker 1: Just simple things that are dependent on the use case.
• 14:11
  Speaker 1: So, unfortunately, there's no generalization, but it's all stuff that I think we're all familiar with.
• 14:15
  Speaker 1: The important thing is that this needs to be done, because what ends up happening is, if you're not careful, let's see if it was yeah, it's the next slide.
• 14:23
  Speaker 1: If you're not careful, you could be in a position where there is no more correlation, for example, between two different data, two different sensors, and you might get in a position where you're capturing data for hours, days, months and actually your sensor is broken or it's not working.
• 14:41
  Speaker 1: So it's really important to have these checks again live in real time make sure that the data you're collecting is actually being collected in the right way.
• 14:49
  Speaker 1: Just as an example, we have one customer doing days in the lab effectively and one day in a lab is $10,000.
• 15:01
  Speaker 1: And they've done thousands of days in the lab and obviously collecting that amount of data is Is paid, is costly, and then if you're not careful, it can it can really be difficult to fix or kind of Find additional money to collect more data.
• 15:16
  Speaker 1: So, again, it's important to think about being able to understand whether the data you're correcting, collecting, is correct.
• 15:27
  Speaker 1: The next piece is identifying outliers.
• 15:29
  Speaker 1: This is all about trying to identify.
• 15:32
  Speaker 1: You know there's always this question of is my data diverse enough, right, or do I actually have you know?
• 15:41
  Speaker 1: Do I need to collect more of this outlier right, because actually it's another cluster of data that I've never seen, another type of data, another class of data that I've never seen before.
• 15:49
  Speaker 1: Or is this actually a random piece of data that I should not be seeing in my system and so I should throw away?
• 15:55
  Speaker 1: It's a question that we should ask ourselves.
• 15:58
  Speaker 1: Usually it requires an understanding of the problem.
• 16:02
  Speaker 1: So really, I think as a group here we can solve a lot of the building blocks, or we can address a lot of the building blocks, but actually it really comes down to domain expertise.
• 16:12
  Speaker 1: We heard before from Luxottica to build an eyeglass, to build smart eyewear, you need to have all that domain expertise that goes into the problem.
• 16:22
  Speaker 1: It's not just about math and ML.
• 16:27
  Speaker 1: The last piece.
• 16:28
  Speaker 1: I mean I couldn't leave the stage without not talking about ChatGPT and GenAI, but I purposely left it at the end because I'm pretty opinionated about the use of GenAI in our tool.
• 16:42
  Speaker 1: Obviously, we all know that it's a big hype, but we're building tiny stuff right and sometimes Gen AI is not needed, and so you know, if it makes sense, let's do it right.
• 16:55
  Speaker 1: But actually what we're seeing is the use of Gen AI within our tool to enhance productivity of developers, right, really, that's.
• 17:02
  Speaker 1: The main focus for us right now is how can we use more gen AI to actually automate some of the tasks that are more painful, more tedious, more time consuming?
• 17:12
  Speaker 1: And so one of the things I'm showing here is actually something we just released is the ability to do automatic labeling based on ChatGPT 4.0.
• 17:21
  Speaker 1: Simply, you write a prompt and you get the model to understand your labeling or your data and label it for you.
• 17:32
  Speaker 1: What's interesting is that we've done some experimentation with this and it really allows you to make a model right very fast based on this ability to label data in real time at scale.
• 17:48
  Speaker 1: So it's a really interesting use case.
• 17:50
  Speaker 1: We're going to be building more of this, but again, I think let's all think about how we can use this technology to enhance our products, the productivity of people using our product, and perhaps not always think about the ability of these models to understand voice or text, because that's not always useful.
• 18:14
  Speaker 1: And then, last, I wanted to just show an example of what actually happens in production.
• 18:20
  Speaker 1: All this stuff that I presented, it all needs to happen in automated ways.
• 18:27
  Speaker 1: So what we build is pipelines, right, I mean, you know I hate to say this, but my job is pretty boring.
• 18:34
  Speaker 1: We're actually an infrastructure company.
• 18:36
  Speaker 1: That's what we build, right?
• 18:38
  Speaker 1: We build infrastructure to make stuff like this work at scale.
• 18:43
  Speaker 1: And really what it boils down to is you need to know what the pieces are and then you need to piece them together.
• 18:48
  Speaker 1: What comes to mind is CI CD.
• 18:51
  Speaker 1: We've done that for many years.
• 18:52
  Speaker 1: It's automated code development.
• 18:56
  Speaker 1: This is automated ML development, but really it boils down to having these pipelines in production that just run continuously so that you can not only relabel new data that comes in, analyze it, relabel it, throw away data if you need to throw it away and then retrain the model based on the new data and redeploy it in the field.
• 19:16
  Speaker 1: And what we do, for example, here I'm showing just an example of you know an email that gets sent when your model is ready and it's been redeployed, and so you get updates.
• 19:24
  Speaker 1: You know, over time gets sent when your model is ready and it's been redeployed, and so you get updates over time.
• 19:28
  Speaker 1: But the point is really this needs to happen, no matter how you do it I don't care if you do it using our tools or other tools but it's really important that we all think about this to make sure that we don't go into the news in the wrong way and we end up building a lot of stuff that just ends up not working anymore after a bit of time, and so, really, the end message is build for scale from the start, Don't leave it as an afterthought.
• 19:57
  Speaker 1: I always think about security, right.
• 19:59
  Speaker 1: I was working for ARM for many years.
• 20:01
  Speaker 1: Security was an afterthought, and then people started hacking IoT systems right, and then it became a big thing for everyone.
• 20:07
  Speaker 1: But it's it's.
• 20:09
  Speaker 1: You know, it's not always there.
• 20:10
  Speaker 1: We shouldn't always focus just on the fun stuff, right?
• 20:13
  Speaker 1: Sometimes there is some important stuff that needs to be tackled, and I think, as a group, we need to make sure that we tackle it together and we tackle it well.
• 20:21
  Speaker 1: Thank you, thank you.