Latest / EDGE AI POD / Aptos: Creating ML models that fit your edge device like a glove

Transcript

Aptos: Creating ML models that fit your edge device like a glove · EDGE AI POD

• 0:06
  SPEAKER_00: Hi, thanks.
• 0:10
  SPEAKER_00: So today I'd like to entertain you with uh my talk about uh the tool we're developing at Eta Compute.
• 0:16
  SPEAKER_00: Uh just one word about the company.
• 0:18
  SPEAKER_00: Uh we are a startup based in uh uh Sunnyvale, California.
• 0:23
  SPEAKER_00: Uh and we are in the business of building uh AI models.
• 0:31
  SPEAKER_00: Okay, yeah, there it goes.
• 0:32
  SPEAKER_00: And let me just check the laser.
• 0:35
  SPEAKER_00: As a laser, perfect.
• 0:36
  SPEAKER_00: I'm all set here.
• 0:38
  SPEAKER_00: Uh well I mean I think it goes without saying that it's challenging to to develop and deploy uh successful uh uh applications.
• 0:48
  SPEAKER_00: Uh gaps are many.
• 0:50
  SPEAKER_00: Uh the gap between the actually the embedded world and and uh uh AI world is is is big.
• 0:57
  SPEAKER_00: There are two different animals that move at different pace and they have different histories and so on.
• 1:02
  SPEAKER_00: Uh surely one gap is talent.
• 1:04
  SPEAKER_00: Um there are not so many people that actually have competence in both worlds.
• 1:09
  SPEAKER_00: Uh I think we we know that very well.
• 1:11
  SPEAKER_00: Probably most of the people who are actually can call themselves experts in edge AI are in this room uh today.
• 1:17
  SPEAKER_00: There are not all that many of us uh that understand both both both worlds of embedded and and uh ML well.
• 1:25
  SPEAKER_00: Uh targeting uh uh you know a ML model onto a device, uh embedded device with resource constraints and so on is challenging for a number of reasons I'm gonna cover in this talk.
• 1:35
  SPEAKER_00: Uh even choosing a silicon today where you want to build your application on is not that trivial.
• 1:42
  SPEAKER_00: I think it has accelerated a lot in the last uh last years.
• 1:46
  SPEAKER_00: Probably have like at least one uh uh chip release per month right now with some uh NPU or some uh accelerator.
• 1:53
  SPEAKER_00: So actually being able to keep up with that, choosing a good chip for your project or moving to a newer chip is uh is is challenging enough.
• 2:01
  SPEAKER_00: And this eventually leads up to long development cycles uh and a number of problems.
• 2:06
  SPEAKER_00: Uh so today um I would say that I identify three types of uh uh design patterns that offer a problematic.
• 2:14
  SPEAKER_00: So first one is uh many people when they want to build an uh AI, edge AI application, uh they would start with uh see the model zoo that comes often from silicon vendors.
• 2:25
  SPEAKER_00: So that you know that's just often the path that is preferred by more embedded aligned teams.
• 2:31
  SPEAKER_00: It's uh you know typically a very easy deployment.
• 2:33
  SPEAKER_00: It's it works great on hardware, so you are able to deploy a model you know within uh a day or so.
• 2:39
  SPEAKER_00: Uh but then you just start to realize that oh, you know, this this model probably doesn't exactly fit your application, right?
• 2:44
  SPEAKER_00: Because maybe it runs too slow, maybe maybe it runs fast but it's not accurate enough, maybe it just doesn't really represent your use case because maybe it's uh model was trained specifically to detect people in the rooms, like uh VW type of model, but then you're actually looking for something that detects uh say animals or something different.
• 3:00
  SPEAKER_00: And then you start to have problems, okay.
• 3:03
  SPEAKER_00: How do I actually adjust?
• 3:04
  SPEAKER_00: How do I retrain?
• 3:05
  SPEAKER_00: How fine-tune this model?
• 3:06
  SPEAKER_00: Um, many of the models that are uh given in in the uh model zoos are are quite uh uh kind of binary, compiled already, not really trainable.
• 3:16
  SPEAKER_00: I think there's some activity from the in the community here that is trying to address those issues as well, but as of today it's it's a difficult path.
• 3:24
  SPEAKER_00: Uh another path that's typically preferred by data science-oriented teams who maybe have less experience in uh in embedded is to just go for foundation models.
• 3:32
  SPEAKER_00: Right?
• 3:33
  SPEAKER_00: And out of the box, often they get very good accuracy and they are very happy, okay, it works, and they say, you know, now we're gonna put the model on our device.
• 3:42
  SPEAKER_00: And suddenly, oh, we need to export it and maybe into quantize it because it doesn't uh really uh fit or the chip doesn't support the non-quantized models.
• 3:50
  SPEAKER_00: Uh maybe when quantifies we lose some accuracy.
• 3:53
  SPEAKER_00: Uh okay, we eventually got through this process, we jump uh we thread it on the board, and then oh, some ops are not supported, and the inferences uh time is really long, doesn't fit the application after all, and this is where some steps will uh some some stairs will uh or problems will start.
• 4:08
  SPEAKER_00: Um and the third path is is probably kind of in the between if you have maybe balanced teams or two teams that are independently working between embedded and uh uh and uh data science world, that you will say, okay, let's just craft our own model for specific application, right?
• 4:22
  SPEAKER_00: So the so you'll build a model, improve it, uh deploy, try to deploy it, it doesn't work, you optimize, you come back and keep uh uh working in this very interactive uh mode.
• 4:32
  SPEAKER_00: So this one, uh, you know, often it gets out uh like the budget is is overblown, the time is is timelines are not met, it's it's maybe not as good as you initially hoped for.
• 4:42
  SPEAKER_00: So I think whichever path you follow of those three, it's it's very common that you end up in some sort of feedback loop of try and error, right?
• 4:51
  SPEAKER_00: And and well, if your organization is big enough, uh maybe you have two teams, embedded one and the and uh uh some data science one on the left and embedded one on the right.
• 5:01
  SPEAKER_00: And the first guys will propose the model, uh select some candidates retrain, you try deployment.
• 5:06
  SPEAKER_00: Uh you realize there are some constraints that are maybe not met after deployment doesn't work as well, you you close the feedback loop and keeps coming and spinning back and forth.
• 5:14
  SPEAKER_00: And uh weirdly enough, we as a company we already went from through this process and it took more or less 12 to 18 months.
• 5:21
  SPEAKER_00: Uh uh today, I think on the uh on the panel uh Avjit, if I remember correctly, he mentioned uh a use case from his company where uh a project from his company where actually some 20 engineers worked for six months to get the model deployed on the board and kind of get it to work.
• 5:36
  SPEAKER_00: So it actually, from my perspective, it's really this is how how long it takes uh in many cases to get something to that is a product ready uh feature.
• 5:44
  SPEAKER_00: And uh I would say this is how most embedded projects, uh GI projects are being uh developed today.
• 5:51
  SPEAKER_00: Uh so I would say majority of them never really pass through the you know experimentation or proof of concept phase, and they don't end up being in a um in a product just for the sake of of the the the you know the time and money it takes to actually get it to work.
• 6:07
  SPEAKER_00: Um you know typically the the inputs to this whole system is what you're trying to achieve, right?
• 6:11
  SPEAKER_00: It's it's uh uh objectives.
• 6:13
  SPEAKER_00: You have some sort of KPI or metric of success for your application as a product as a whole.
• 6:17
  SPEAKER_00: Uh you know, you have your data that you will want to train your model on and so on, and you have your constraints, right?
• 6:22
  SPEAKER_00: So this is where okay, I have to have uh this many milliseconds of runtime, or I have to fit in this memory because I need to have some extra memory for my application, and so on.
• 6:30
  SPEAKER_00: And what you expect as the output uh is you know is your ideally your opt-optimized uh model that does the job.
• 6:38
  SPEAKER_00: Um so at the compute, we thought I think it's impossible to really continue in this this loop.
• 6:44
  SPEAKER_00: Let's invent you know how this process could be automated.
• 6:47
  SPEAKER_00: And we thought, okay, how about we just work with uh data in model out approach where there's no people doing the the loop and teams glueing the loop uh in between, right?
• 6:56
  SPEAKER_00: So we thought, okay, this is kind of uh we let's build a black box system.
• 6:59
  SPEAKER_00: Uh and this is what Aptos basically is.
• 7:02
  SPEAKER_00: It's a system where you provide your input, uh, objectives, data, constraints, uh, you have the black box that does the magic, and you get eventually your model is optimized for the hardware.
• 7:12
  SPEAKER_00: And we you know what we aim for is that you get this thing done in in one to two weeks.
• 7:17
  SPEAKER_00: And on top of that, we give a little bit of a knob for for your team uh uh to actually fiddle and tune this machinery of this black box, right?
• 7:26
  SPEAKER_00: So so if you're somebody's more embedded, he can tune some embedded parameters of how this machinery should work.
• 7:31
  SPEAKER_00: If somebody's more data science oriented, he'll work more on the how to tune the data with kind of data science part of this machinery.
• 7:39
  SPEAKER_00: Uh overall it's quite intelligent by itself, so so you don't have to fiddle a lot uh to get good results.
• 7:44
  SPEAKER_00: Um what's what's in the box, right?
• 7:47
  SPEAKER_00: Uh so this is our general architecture.
• 7:50
  SPEAKER_00: The the main uh theme, let's say, or or the main uh idea we follow here is that we want to build AI that builds AI.
• 8:00
  SPEAKER_00: I think this this uh word has been already said today on on one of the uh um in their panel presentations.
• 8:07
  SPEAKER_00: Um so we have a system that uh has two feedback loops.
• 8:13
  SPEAKER_00: The first one is basically a model builder, and this is independent of what kind of user wants.
• 8:18
  SPEAKER_00: So basically we kind of keep building different architectures, uh, generate different models, uh uh that this can be a combination of neural architecture search and can be also uh our proprietary, edge-friendly architecture um recipes.
• 8:35
  SPEAKER_00: And when I say recipe, I don't want to say architecture simple architectures because those those architectures are often very parameterized, and we can actually fiddle and squish and squish them and modify how they will actually uh uh work when they we when we instantiate them.
• 8:48
  SPEAKER_00: So there's this one part of the system that is uh doing this this loop of building models.
• 8:53
  SPEAKER_00: And the second part uh is actually selecting some of those models from our pool and training them and optimizing and deploying on target.
• 9:02
  SPEAKER_00: Um so the this part is is you know uh from the uh from the the whole knowledge store of of all the models that were generated and uh put in our kind of knowledge database, we'll select one that we believe fit the best to the user request.
• 9:18
  SPEAKER_00: So this is where the user comes with a certain objective, it comes with his certain data and with certain constraints.
• 9:24
  SPEAKER_00: And if we are able to uh meet uh uh from from our pool of models uh uh through those constraints the best models and train them, this is where you're gonna get close to the optimal result.
• 9:35
  SPEAKER_00: Right?
• 9:35
  SPEAKER_00: So whenever a customer would run a certain job on our platform, like you know, okay, build me a model that detects uh people in pictures or uh you know classifies uh images and so on, we're gonna spin this kind of loop.
• 9:48
  SPEAKER_00: Everything that we do is get eventually to the hardware.
• 9:51
  SPEAKER_00: So we have a uh app farm.
• 9:54
  SPEAKER_00: This is an actual server racks with uh SDKs, sorry, not SDKs, but development kits or evaluation kits of different chip manufacturers being uh uh instantiated uh there.
• 10:06
  SPEAKER_00: So every time we actually pick a certain model with a certain instantiation of an architecture, so actually proper model with weights and so on, uh we're gonna validate it on our on our um on this farm.
• 10:17
  SPEAKER_00: We're gonna get feedback whether this has run and how well it ran and so on and so forth.
• 10:22
  SPEAKER_00: Uh this feedback gets uh this merch gets sent back to our core, which is the model tide, everything gets stored in the in the knowledge store.
• 10:30
  SPEAKER_00: So knowledge store is where we have our recipes for architectures, where we have our performance of how well the the models performed after being trained and how well they perform on different chips.
• 10:41
  SPEAKER_00: This the kind of triad of of knowledge.
• 10:44
  SPEAKER_00: Uh now, interesting part is that you know why there's a brain here in the center, right?
• 10:49
  SPEAKER_00: Because we uh if you do it on scales, you actually start to collect a lot of data here.
• 10:54
  SPEAKER_00: So you know, the more you spin those two loops, the more data points you're gonna realize.
• 10:58
  SPEAKER_00: You know, the you have some hundreds to thousands of architecture recipes, uh, you're gonna have uh tens of thousands of actually instantiations of architecture with trained weights, and you're gonna have hundreds of thousands of profilings coming from from boards because you have you can test the same model on multiple, multiple boards with different uh runtime parameters, different settings, SDK, and so on.
• 11:17
  SPEAKER_00: Um so having all that, what we do is we train meta models.
• 11:21
  SPEAKER_00: Right?
• 11:21
  SPEAKER_00: So meta models, uh this is the kind of intelligence of our platform.
• 11:24
  SPEAKER_00: So meta models is our machine learning models, uh sometimes it can be deep learning models, that will uh understand what has worked, how and why.
• 11:35
  SPEAKER_00: So next time we asked, uh when next time the customer asks us, okay, I want to train and you know a model on a given chip of this data set, we can say, okay, you know, let's predict what would be the best model that fits his requirement, right?
• 11:48
  SPEAKER_00: So this model, Metamos, for example, and there are many different types that we use, can uh predict the runtime on the board before we actually even have to test it.
• 11:55
  SPEAKER_00: So we can pick uh uh uh pick or discard uh models very quickly based on his uh specific requirements on runtime parameter.
• 12:02
  SPEAKER_00: Or they can understand where is the bottleneck of in the of the memory in the model and based on that reject the model because it's not gonna fit on specific uh a specific chip and so on and so forth.
• 12:12
  SPEAKER_00: Or which hyperparameters to use, actually, because you know that hyperparameters are very important.
• 12:16
  SPEAKER_00: If you go for very slow learning rate, you're gonna lose a lot of time training.
• 12:19
  SPEAKER_00: If you go for too high, you're not gonna converge or have divergence, and so on.
• 12:23
  SPEAKER_00: Uh so actually every model will have a certain uh uh range of hyperparameters which are quite optimal to it.
• 12:29
  SPEAKER_00: Right?
• 12:29
  SPEAKER_00: So we're basically trying to infer of what those parameters uh and the architecture and so on would be in there.
• 12:35
  SPEAKER_00: And the more jobs are coming from our system, the more of data samples we collect, the more we can learn.
• 12:42
  SPEAKER_00: Uh one more important part of the system is the little academic hat here.
• 12:46
  SPEAKER_00: Uh so um even if some of our customers can be fairly big companies, they rarely have really large data science teams that are dedicated to edge.
• 12:55
  SPEAKER_00: So, you know, maybe five, ten people really most from in my experience.
• 13:00
  SPEAKER_00: Uh it's not that easy that those teams they stay uh in touch with all the newest research that happens.
• 13:06
  SPEAKER_00: And there's so much research being published every, basically every day, you can say, on optimization, on quantization, on new architectures and so on.
• 13:13
  SPEAKER_00: It's it's really a lot.
• 13:15
  SPEAKER_00: So if you want to get the best outcomes, you would have to really you know have people follow all the all the academic work that happens.
• 13:22
  SPEAKER_00: And you know, it's it's really not realistic uh scenario.
• 13:25
  SPEAKER_00: So often they just say, Oh, I heard about YOLO, let's pick YOLO and let's go for it.
• 13:29
  SPEAKER_00: Uh this is what we do, right?
• 13:31
  SPEAKER_00: So here we actually do try to keep track of what happens, uh, even a conference like here, of you know what are the new techniques for optimization, what are the new the techniques for pruning and so on, and we do uh inject those uh this knowledge into our model building engine.
• 13:47
  SPEAKER_00: We actually also do it in the training optimization, right?
• 13:50
  SPEAKER_00: So, what can we do to make the models better after they are trained?
• 13:54
  SPEAKER_00: Um so here is uh an example of uh example result of a job that we run on Aptos.
• 14:02
  SPEAKER_00: Um so user inputs here, data set.
• 14:05
  SPEAKER_00: I used the you know the very standard well-known data set for everybody, just uh for an academic example.
• 14:11
  SPEAKER_00: So a visual wake word.
• 14:13
  SPEAKER_00: Um there is the device we targeted was was uh uh NXP uh IMX8 Plus.
• 14:18
  SPEAKER_00: We gave a constraint inference time, less than five milliseconds.
• 14:22
  SPEAKER_00: Uh we selected some input uh uh size range, you can specify a very defined one, or you can just say leave it open, whatever.
• 14:30
  SPEAKER_00: Uh and aspect ratio for input.
• 14:32
  SPEAKER_00: The aspect ratio I put it here because if you actually look at the foundation models or model zoos, almost all the vision models will have a square input.
• 14:40
  SPEAKER_00: But it doesn't mean that your application actually would be optimal if you have a square input.
• 14:44
  SPEAKER_00: If you look at some of the automotive applications, you often have like a very wide angle, uh uh uh for example four to one or three to one aspect ratio.
• 14:52
  SPEAKER_00: So actually, Aptos can also give you models with uh non-square aspect ratio as well.
• 14:58
  SPEAKER_00: This means you're just more optimal because you're not losing information when you're deforming the images to fit them into the model.
• 15:04
  SPEAKER_00: Um so every one of those dots here uh is an actual model with its own weights and so on.
• 15:12
  SPEAKER_00: The the four orange ones are I just took what was available in the publicly available model zoos.
• 15:17
  SPEAKER_00: So uh you know the mobile netv1 is the original VW benchmark model that everybody uses, and uh I took a few models from the uh ethos uh no ethos, uh ARM, I think ARM uh model zoo here.
• 15:29
  SPEAKER_00: So I mapped them.
• 15:29
  SPEAKER_00: You can see that you know the what is the inference time, what is the accuracy here, right?
• 15:33
  SPEAKER_00: So, first of all, you what stands out is that what aptos are generated throughout maybe a day of a job uh are models that dominate this this part of so the part of frontier in terms of the runtime versus accuracy is is here gently dominating those models.
• 15:50
  SPEAKER_00: We also have many more models of higher accuracy.
• 15:53
  SPEAKER_00: And also today, we've said, you know, nobody wants to actually pay for models with uh uh with 70 or 80 percent accuracy.
• 16:00
  SPEAKER_00: You often you want to go higher.
• 16:02
  SPEAKER_00: Uh if you go for model zoo, often you in this specific case, it's not uh the only case that can happen, but here you're probably you know way under your limit of application, which is five milliseconds, or maybe one or two, and so on.
• 16:12
  SPEAKER_00: If your application can run five milliseconds, well, why not have a much more accurate model?
• 16:17
  SPEAKER_00: So um so you can see that here it's really a point cloud, and this is where you can pick up the model that meet meets best your uh your uh your uh application.
• 16:29
  SPEAKER_00: Uh accuracy and time is not the only uh it's not the only compromise that you can take because also there's memory, usage, and so on.
• 16:36
  SPEAKER_00: So this all these models will fit in a certain constraint that was defined, but uh you can also pick you know whichever will be matching best your your expectation here.
• 16:46
  SPEAKER_00: So you have much more models to work with than from a model zoom.
• 16:50
  SPEAKER_00: Uh yeah, every model, as I mentioned, we have this farm.
• 16:54
  SPEAKER_00: Every model gets validated fully.
• 16:55
  SPEAKER_00: So actually you get a full access to each one of those dots that you saw before.
• 16:59
  SPEAKER_00: You're gonna see whether it has run on the board, is runnable or not.
• 17:02
  SPEAKER_00: I mean, obviously the ones were on the plot were runnable.
• 17:04
  SPEAKER_00: Uh what is what is the inference time and the uh per layer uh timing?
• 17:09
  SPEAKER_00: Uh you get a full memory profiling, so how much actually those models uh take, obviously, under your constraint that you have defined, but maybe they're you know some of them consume less or more.
• 17:18
  SPEAKER_00: You will actually get a proper power measurements.
• 17:20
  SPEAKER_00: We have instrumentation that's connected to all those boards, and we'll measure uh in this inference window you know what what was the actual uh power consumption of a given uh given board.
• 17:31
  SPEAKER_00: This is, I think, very few people, if anybody does it.
• 17:34
  SPEAKER_00: Uh and we also look at the on-device accuracy.
• 17:37
  SPEAKER_00: That's the that's the part that also is funny because you know most people say, oh, after quantization lost some lost some accuracy, and that's true.
• 17:43
  SPEAKER_00: You typically do.
• 17:45
  SPEAKER_00: But after deployment, you sometimes lose accuracy that you don't even measure.
• 17:48
  SPEAKER_00: Uh because uh you know there's uh some ops in some uh NPUs will actually maybe do some rounding errors and so on.
• 17:57
  SPEAKER_00: And if that's a very deep model, maybe you're gonna accumulate and actually you could can get uh a fairly bad result after deployment.
• 18:03
  SPEAKER_00: So it's not just enough to say, oh, my my TF quantas TF Lite file is having good accuracy.
• 18:09
  SPEAKER_00: It's you have to actually measure it after it's deployed.
• 18:11
  SPEAKER_00: So actually, Aptos also measures uh uh those parameters as well.
• 18:15
  SPEAKER_00: Um yeah, so maybe I'm probably running out of time, so I'm just gonna jump quickly through a few points.
• 18:20
  SPEAKER_00: So you know, I I think that Aptos does address many of the points that that today's uh uh typical development flow for a GI has.
• 18:29
  SPEAKER_00: You know, which architecture should I use?
• 18:30
  SPEAKER_00: You don't have to worry because you just provide data and we create architectures or select architectures based on the uh uh on your requirements.
• 18:38
  SPEAKER_00: Uh does my silicon support uh those ML operations and so on?
• 18:42
  SPEAKER_00: That's being taken care of.
• 18:44
  SPEAKER_00: Enough memory?
• 18:44
  SPEAKER_00: Yes, you can define constraints or you can optimize against memory.
• 18:48
  SPEAKER_00: Can I get uh more out of your NPU CPU GPU?
• 18:51
  SPEAKER_00: Yes, we do test a lot of architectures and a lot of model instantiations in order to find the one that will you know fit best.
• 18:58
  SPEAKER_00: Uh what if I choose a different hardware?
• 19:00
  SPEAKER_00: That's interesting.
• 19:01
  SPEAKER_00: So you provide data to Aptos, you just select a different chip from your drop-down, uh, and you're gonna have a model uh done for that specific hardware.
• 19:09
  SPEAKER_00: So migrating to newer hardware as long as it is in our system, it's gonna be you know fairly seamless experience.
• 19:16
  SPEAKER_00: Uh training training hyperparameters, you don't have to worry about this one anymore.
• 19:20
  SPEAKER_00: Uh training takes too long, we actually have a uh fairly large uh cluster uh uh uh uh infrastructure that can do the training efficiently.
• 19:30
  SPEAKER_00: Um yeah, can I make my model go faster without losing accuracy?
• 19:35
  SPEAKER_00: Yes, another level of the optimization that Aptos takes care of.
• 19:38
  SPEAKER_00: So you don't have to really code.
• 19:39
  SPEAKER_00: There's no coding involved uh in getting your model done.
• 19:43
  SPEAKER_00: Uh as mentioned, we also measure battery or measure power consumption so you you know can estimate what is your expected battery life in case you're building a battery-powered device.
• 19:54
  SPEAKER_00: Um yeah, so basically, long story short, we try to bridge the gap between embedded and ML world.
• 20:00
  SPEAKER_00: Um and you know, I think we build some automation tools that for many companies can be uh a savior because you don't have to have a whole data science team and embedded team.
• 20:11
  SPEAKER_00: You may have one or two engineers, and uh this will be enough for you to launch a successful project that is uh you know the mods that are validated and and working properly.
• 20:21
  SPEAKER_00: So yeah, that's that's it, thanks a lot.