Latest / EDGE AI POD / Revolutionizing TinyML: Integrating Large Language Models for Enhanced Efficiency

Transcript

Revolutionizing TinyML: Integrating Large Language Models for Enhanced Efficiency · EDGE AI POD

• 0:23
  Speaker 1: Hi and thanks to accept the invitation and to present, so consolidating TinyML's lifecycle with large language models, reality, illusion or opportunity.
• 0:36
  Speaker 1: To you the floor, roberto.
• 0:37
  Speaker 1: Thanks.
• 0:38
  Speaker 2: Thanks, thanks a lot, danilo, and thanks to the Tiny ML Foundation for organizing such a nice event.
• 0:47
  Speaker 2: This is pretty much the outline for my presentation.
• 0:51
  Speaker 2: What we will try to do is trying to essentially answer to this question and understand to some extent whether the usage of large language models can somehow help in the management and automation of the lifecycle management of tiny ml.
• 1:09
  Speaker 2: So the core of the presentation will be, of course, the introduction of this framework that we have developed a lot between eurocom and university of elsinki, and we will also show a very short proof of concept and demonstration before also providing some empirical insight on what's the type of resource consumption and the possible bottlenecks that we have observed in this first phase of the work.
• 1:41
  Speaker 2: And what I would like to also emphasize is that we really embrace the great opportunity that we are envisioning and this event is definitely like.
• 1:53
  Speaker 2: We've seen the great momentum that there is increasing towards more and more interesting works in the area of generative AI at the edge, and we have seen during the first series of this event and there is a lot of work both in industry and academia that kind of show how there is more and more effort from the research and development point of view on trying to push language models and generative AI towards the edge, but what we want to actually do with this kind of work is essentially trying to take a step back and trying to understand if there was any possibility of taking advantage of a large language model.
• 2:43
  Speaker 2: In the intersection with Tania and Al, and a little bit also considering the theme of this event, we really wanted to go beyond the use of chatbot.
• 2:57
  Speaker 2: When we think about large language model, the idea is obviously taking consideration the code generation capabilities in particular and, looking at the tiny ML perspective, some of the essential stage of its lifecycle One essential aspect also and this was also briefly discussed in the previous discussion something that we really don't want to forget is the cost, the actual cost from time and monetary perspective, that the usage of API, large language models, api can eventually occur and we have been trying to put in relation to the development of our framework.
• 3:45
  Speaker 2: And the reason why we wanted to focus on the comparison between the traditional machine learning lifecycle and the Italian machine learning lifecycle is because this latter one includes obviously some additional stage that it depends, uh, it's to some extent a little bit more complex than the traditional machine learning life cycle and also it involves the, the hardware perspective, the constraint of the hardware especially, which is also reflected on the software and so on and so forth.
• 4:20
  Speaker 2: And, as I said, mentioned, as I mentioned earlier, the main idea was trying to understand, uh, what understand, what are the different trade-offs from this kind of work are essentially three main questions.
• 4:49
  Speaker 2: Obviously, to some extent, because we are kind of at the early stage of the prototyping and the work, the research work that we are trying to do, but the main questions are essentially like the wish to understand what kind of stage, what type of stages of the Latania-Metalab cycle can actually be automated using large language models.
• 5:12
  Speaker 2: But also, more interestingly perhaps, is trying to understand what are the key elements of the large language models.
• 5:20
  Speaker 2: Here.
• 5:20
  Speaker 2: I'm referring to and we will see shortly For example, the prompt templates, the prompting of the model itself for tailoring and optimizing this automation and, of course, as a kind of takeaway of this first two questions, trying to understand what are the possible bottlenecks, the trade-offs, the challenges, but also the opportunities.
• 5:46
  Speaker 2: Yeah, I essentially went through these, but this can give a better overview, I guess, than what's been shared.
• 5:56
  Speaker 2: But just to remark these three main components that characterize the framework and, as mentioned earlier, the four surrounding supporting aspects of the framework and, as mentioned earlier, the four surrounding supporting aspects of the framework the human in the loop, the large language model, the end device and then whatever is the outcome of this processing.
• 6:17
  Speaker 2: And I would like to spend some time on the third component that I've been mentioning, that is, the prompt template.
• 6:26
  Speaker 2: And these are essentially the structure that is used to, as I mentioned, to guide a large language model to produce the desired output.
• 6:33
  Speaker 2: And here there is, of course, a lot of momentum and activities from R&D point of view and trying to understand what's the best technique to be used for this kind of engineering aspect of the large language model.
• 6:52
  Speaker 2: In our particular case, I have to be honest, we have been essentially learning by doing and trying different approaches and trying to find the most suitable.
• 7:06
  Speaker 2: Of course there might be some further optimization that can be done and this is also part of our future work.
• 7:12
  Speaker 2: But one of the lessons learned here is one essential part is trying to make the format and the structure of the prompt very precise in order to kind of have a consistency and accuracy.
• 7:27
  Speaker 2: And it's well known, and this was also proved in our case, that the structured output like a JSON helped to get better outcome from the processing.
• 7:41
  Speaker 2: And also different techniques like delimiters or indicators and we will see shortly an example can help the large language model to produce a better output.
• 7:54
  Speaker 2: And in our particular case we decided to structure the prompt in a way that we can essentially define in a modular way the context, so trying to explain the large language model, the role and the kind of expertise that should be used for producing the output.
• 8:26
  Speaker 2: And then there's quite, of course, quite important trying to describe as much as possible that particular task that is going to perform, referring to the life cycle management of TinyML could be the model conversion, the generation of the sketch for the Arduino and so on and so forth.
• 8:48
  Speaker 2: Obviously, the error handling is part of the prompt template as well, and you will see shortly in the demo that this is very important because it kind of allowed the large language model to iterate over itself together with the framework in order to fix possible errors.
• 9:09
  Speaker 2: And then a specification that essentially gives additional guidelines to the large language model, essentially some background context, additional information that can help the model to produce a better output.
• 9:25
  Speaker 2: And this is a practical example of what I just mentioned here.
• 9:30
  Speaker 2: In this case, this is the prompt template script for the sketch generation in the Arduino.
• 9:38
  Speaker 2: You can see the context of the task.
• 9:40
  Speaker 2: In context, you're essentially telling to the large language model that you're an expert in tiny machine learning, etc.
• 9:49
  Speaker 2: Etc.
• 9:49
  Speaker 2: While the task is obviously more technical and oriented and also perhaps trying to involve other components either of the framework or the lifecycle, for example, like the data set or other component of the framework.
• 10:09
  Speaker 2: This is just an example there is a it's quite long code here.
• 10:14
  Speaker 2: There is a obviously very important error handling that plays a key role in this context, and the current proof of concept involves three main aspects of the lifecycle the data processing but the main emphasis that I'll try to give today during the talk is actually on the sketch generation.
• 10:41
  Speaker 2: But we have also worked on the model conversion, which is obviously very important and key in this particular case, because we need to build our machine learning model in such a way that it can be executed, in this particular case, in an Arduino board, and the test case and the demo that we are going to see shortly refer to a CNN-based vision model.
• 11:08
  Speaker 2: So we tested the automation with this very simple machine learning model of fluid identification using Arduino and TensorFlow.
• 11:21
  Speaker 2: And before showing the demo, I would like to quickly go to the explanation of what is actually in practice happening in the lifecycle.
• 11:34
  Speaker 2: So we have obviously the user input, but this is, of course, like only at the first stage.
• 11:40
  Speaker 2: Then everything is automated through the framework and the larger language model.
• 11:45
  Speaker 2: But essentially what is happening and that's the content of the demo shortly is that once that input from the user is acquired, the framework will try to cooperate with the larger language model in order to be using all the background information in order to generate the code, the compiled code, to be used in the device.
• 12:07
  Speaker 2: Essentially, that's the final outcome that we are wishing to see.
• 12:15
  Speaker 2: So here comes now the demo.
• 12:18
  Speaker 2: I actually ask if it's possible to switch to the PowerPoint once once again, because otherwise I guess we'll be able to to watch the video.
• 12:30
  Speaker 2: All right, if it's possible.
• 12:33
  Speaker 2: Obviously, okay, seems that it's not possible to run the video here, not even with a PowerPoint.
• 12:52
  Speaker 2: All right, but yeah, perhaps.
• 12:53
  Speaker 2: Yeah, I would keep pointer.
• 12:58
  Speaker 2: There is a pointer, some pointer at the end of the presentation and you will get the chance to look at the demo itself.
• 13:06
  Speaker 2: But yeah, perhaps it can be shown at some point.
• 13:14
  Speaker 2: But anyway, I will move forward.
• 13:17
  Speaker 2: I don't want to run out of time, but one key aspect that we wanted to understand also is the closed source model.
• 13:25
  Speaker 2: Like the OpenAI ones are drastically changed, probably weekly if not daily, so it's kind of hard to get reproducibility of these results.
• 13:54
  Speaker 2: But there is pretty much consistency in what we have been able to observe from our analysis and the type of cost that we wanted to analyze.
• 14:10
  Speaker 2: It sort of includes the perspective both of the token and the time.
• 14:15
  Speaker 2: So the token is this essentially unit of text that would model the process.
• 14:20
  Speaker 2: And what is important to clarify is that both the input token, so all the context, and this kind of context, the error rendering the task itself, description of the task, of course affected both the processing time, the monetary and also the computational resources used on the other end, from the model point of view, and what you can observe here is essentially a distribution of time and token consumption for the different tasks for the model conversion, the data processing and the sketch generation.
• 15:02
  Speaker 2: What you can immediately notice, both from the graph on the left-hand side but as well from the table on the right, is that the success rate and this was limited.
• 15:13
  Speaker 2: The number of attempts for kind of a query in the large language model was limited to five in this particular case.
• 15:22
  Speaker 2: So you don't keep invoking the large language model endlessly.
• 15:26
  Speaker 2: It was quite good the success rate for the data processing and the model quantization and conversion, but you can see also that the sketch generation, which is quite relevant right for the lifecycle management.
• 15:41
  Speaker 2: In this particular case it was actually uh pretty low and this was all also reflected uh to some extent on the average time and number of token uh that were kind of uh used for uh for generating uh that output.
• 15:59
  Speaker 2: Just to give an example.
• 16:00
  Speaker 2: That output, just to give an example, in the case of even past compilation, the sketch generation was lasting almost two minutes and the number of tokens was not as high as the data processing, but fairly high.
• 16:19
  Speaker 2: And here comes one interesting perspective here, which is the total cost on average for automating and deploying this kind of different steps in an automated way.
• 16:35
  Speaker 2: And obviously this doesn't look much if we look at the actual cost.
• 16:42
  Speaker 2: In this case, for example, for the sketch generation is like $0.14.
• 16:46
  Speaker 2: It's not much in the worst case, but obviously we need to take into consideration that this could potentially be on a very high scale.
• 16:56
  Speaker 2: You might need to manage a bunch of devices and you might have a much more complex model so to to automate.
• 17:04
  Speaker 2: So on high scale this can become a relevant, relevant bottleneck.
• 17:11
  Speaker 2: So, approaching the conclusion, trying to answer to this question, of course it's not a definite answer.
• 17:18
  Speaker 2: There's a lot of work.
• 17:20
  Speaker 2: Of course it's not a definite answer.
• 17:21
  Speaker 2: There is a lot of work to be done, but the reality is that there is some potential of using a large language model for automating the tiny ML workflow, especially for some of the stages that we have been analyzing, like data processing and model conversion.
• 17:38
  Speaker 2: But there are still quite many challenges.
• 17:42
  Speaker 2: For example, key stage like the sketch generation proved to be a big bottleneck, and also our proof of concept is essentially limited to a simple model, one type of board, so obviously the complexity can become much, much bigger, and so the average time and all the different trade-offs that we have been considering.
• 18:09
  Speaker 2: And here comes the opportunity right.
• 18:16
  Speaker 2: There is, of course, opportunity from the perspective of first trying to support the large language model with either with the external tools and there is a lot of work ongoing in this respect but also kind of trying to augment the knowledge of the large language model for trying to optimize and get more qualified and efficient output.
• 18:45
  Speaker 2: But last but not least and somehow this kind of connected, I guess, what we are going to see later today and tomorrow connected to the fact that we really do believe that a key role will be played also by the end device themselves, whether they are very constrained, like Arduino, where of course there won't be the possibility to run small language model or something a little bit more powerful like a single board computer.
• 19:14
  Speaker 2: We really believe that the device can become a key, essential part in somehow becoming external tool, uh, the self, external tool, import for uh, optimizing uh, the, the, the life cycle, because you can essentially abstract device specific information, application specific formation that can reduce the cost and the amount of time required for processing uh, for processing uh, the amount of time required for processing the lifecycle stage at the large language model site.
• 19:54
  Speaker 2: And, yes, soon to be published an overview about this work.
• 19:58
  Speaker 2: We will make sure to add also a link to the demo, perhaps sooner.
• 20:03
  Speaker 2: But this is pretty much concluding my work and I would like to thank especially that from the University of Helsinki who has put the theory in practice in this work.
• 20:17
  Speaker 2: Thanks a lot.
• 20:35
  Speaker 1: We have still some time for a few questions.
• 20:40
  Speaker 1: I have some, roberto, more complex to deal with, especially if you want to generalize the workflow on different tiny target hardware.
• 20:57
  Speaker 2: Yeah, as I mentioned in my last slide, the kind of illusion here comes from, and we can definitely say that this work generalizes over different type of models, because it's quite complex.
• 21:16
  Speaker 2: Definitely, what we have been observing and we have been testing, largely with different configuration, even with a very simple model like the one that we have been using for our prototyping, the sketch generation that perhaps in this particular context of tiny ML devices, along with the model conversion, that represents the key part, right, if you are not able to generate the correctly, the Arduino sketch, it's pretty much useless what you're asking to do, right.
• 21:46
  Speaker 2: So definitely that part is still quite challenging.
• 21:51
  Speaker 2: Obviously, it's been proved that the code generation capabilities on, for example, languages like C, c++, are less performing than code generated, for example, for Python.
• 22:07
  Speaker 2: So that could be one possible reason that it's kind of worth to investigate.
• 22:12
  Speaker 2: But definitely the sketch generation in this particular case was the main bottleneck and I expect to be even a little bit more complex if we take in consideration a more complex model, for example, or different types of devices.
• 22:30
  Speaker 2: So definitely that was the trickiest part.
• 22:35
  Speaker 1: For example, also considering hardware with neural processing unit embedded.
• 22:41
  Speaker 1: Is that true as well?
• 22:43
  Speaker 2: Yeah, at this stage, one of the actually the one wish that we have next is trying to look at devices like single board computer that have this AI acceleration capabilities, whether it's GPU or TPU.
• 22:58
  Speaker 2: But it would be definitely interesting to look at those devices as well because in that particular case I expect also the model conversion to become a possible bottleneck because there are additional requirements in terms of libraries that they really can can make the data processing a little bit more complex.
• 23:21
  Speaker 2: So a quite fair success rate it was pretty much 90% with the model conversion.
• 23:27
  Speaker 2: But I expect in the moment that we are going to use devices with AI accelerator, embedding AI accelerator also, the kind of success rate for those particular stages could actually go down.
• 23:43
  Speaker 2: And that's the reason why we truly believe that the kind of possible cooperation or anyway the usage of either small language model on edge devices on this type of devices, of abstracting in a smart way information from the end devices, can make this sort of use a little bit more effective and also from additional perspective like privacy etc.
• 24:15
  Speaker 1: And maybe one last question, which somehow is inspired also to a question I see in the comments how often should I reapply to the workflow in the real life?
• 24:33
  Speaker 1: In your opinion, Are there conditions to be set or to be respected?
• 24:39
  Speaker 1: Any consideration on that?
• 24:41
  Speaker 2: Well, essentially it really depends At the end of the day how often you wish to update your model, right?
• 24:50
  Speaker 2: I guess that's the main thing that drives the decision on how often you would like to involve the large language model for automating and updating.
• 25:07
  Speaker 2: I guess it depends on the application.
• 25:11
  Speaker 2: So in the moment that you have like a big data drift and you really need to upload and retrain and convert the model more often, those kind of application will certainly require a larger employment of this type of automation.
• 25:30
  Speaker 2: Otherwise, if it's a very simple application like the one we have been showing, for example, today, in that particular case perhaps I don't see an urgent need to record this type of solution.
• 25:45
  Speaker 2: But there might be cases, this type of solution, but there might be cases in which this could actually be the case.
• 25:52
  Speaker 2: I'm thinking, for example, sort of time series forecasting models where you really in specific industrial setting where you need to update the model more frequently.
• 26:05
  Speaker 2: So those particular cases, I do see kind of an increasing need of automation and iteration over the stage.
• 26:16
  Speaker 1: Thanks a lot, Roberto.
• 26:17
  Speaker 2: There are also further considerations but we needed to uh stop here.
• 26:21
  Speaker 1: Thanks a lot for your contribution.
• 26:36
  Speaker 1: It was much appreciated.
• 26:38
  Speaker 2: Thanks a lot for the opportunity.
• 26:40
  Speaker 2: Thank you so much.
• 26:41
  Speaker 2: Thank you.