Latest / EDGE AI POD / Revolutionizing Edge Devices with Energy-Efficient Generative AI Techniques

Transcript

Revolutionizing Edge Devices with Energy-Efficient Generative AI Techniques · EDGE AI POD

• 0:36
  Speaker 1: Thank you All right, here we are back with Edge AI Talks.
• 0:59
  Speaker 2: Hello, Ciao Pete.
• 1:01
  Speaker 1: Yes, good to see you.
• 1:02
  Speaker 1: So where are you?
• 1:04
  Speaker 1: You're dialing in from Milan.
• 1:06
  Speaker 1: Yeah, awesome.
• 1:08
  Speaker 1: At STMicroelectronics I'm actually on site.
• 1:12
  Speaker 1: I'm in Santa Clara, california, this morning, if you can see the sign behind me at Amazon over there so presenting at their Silicon and Hardware Summit about the future of Edge AI.
• 1:25
  Speaker 1: So I will be ducking out once we have Victor rolling here, but it's great to see you again and it's great to you know.
• 1:35
  Speaker 1: Bring Victor back on.
• 1:36
  Speaker 1: We had him talk at our Generative AI at the Edge forum, I think, in October.
• 1:43
  Speaker 1: Right, a huge, a huge thing.
• 1:45
  Speaker 1: We had some challenges in terms of recording the live stream over two days.
• 1:50
  Speaker 1: We want to have Victor come back and talk more about but his stuff.
• 1:55
  Speaker 1: So, by the way, peter, you pay.
• 2:02
  Speaker 2: Your meeting in Amazon will be.
• 2:04
  Speaker 2: Your talk will be exciting.
• 2:08
  Speaker 2: We need really to to share the beauty of a GI with the companies like Amazon and others, and so yeah wish the best for you for sure.
• 2:19
  Speaker 1: Yeah, no, I'm looking forward to it.
• 2:20
  Speaker 1: I mean I'm sure they're working on some cool stuff.
• 2:21
  Speaker 1: Yeah, yeah, lots.
• 2:22
  Speaker 1: Don't know if they're gonna let me see anything, but uh, yeah, yeah, lots of cool stuff.
• 2:27
  Speaker 1: I mean we all have not all of us, but most of us have alexis and things like that, so they really pioneered a lot of some of the, some of the edge devices and wake vision type things, so cool.
• 2:37
  Speaker 1: Well, why don't we bring victor on?
• 2:38
  Speaker 1: Let's see, here he is.
• 2:41
  Speaker 3: Hi everyone.
• 2:43
  Speaker 2: Hi, Victor Hi.
• 2:45
  Speaker 1: Welcome.
• 2:45
  Speaker 1: Welcome back, I should say yes, and where are you dialing in from?
• 2:52
  Speaker 3: I'm dialing from Zurich right now, and thanks for hosting me a second time here.
• 2:56
  Speaker 3: That's always a pleasure.
• 2:59
  Speaker 1: Awesome, that's fantastic.
• 3:01
  Speaker 1: Ok, great.
• 3:02
  Speaker 1: And then just a reminder folks that are out there that are just joining us, this is a live Q&A live stream show, so you can send in your questions here and Danilo and I will gather them up as Victor talks and hopefully we have a good conversation going here for people who have any questions about what's going on, about what's going on, cool.
• 3:22
  Speaker 1: So let's see.
• 3:23
  Speaker 1: Why don't we uh get you into into your mode here?
• 3:27
  Speaker 1: Uh, are you ready to to get started, danilo, any opening?
• 3:35
  Speaker 2: comments, questions or thoughts for victor before he dives in.
• 3:37
  Speaker 2: Yeah, um, I would like to introduce victor reading his his.
• 3:42
  Speaker 2: I promise it's short, but I think we need to know better.
• 3:47
  Speaker 2: Victor Context, sure.
• 3:49
  Speaker 2: So Victor received his Bachelor in Computer Science and Engineering Physics from Uniata College and a Master in Computer Science from IZIN LIL in 2022.
• 4:06
  Speaker 2: Then, after three months as a research intern in KU Leuven, supervised by Marian Verhulst, he started his PhD with Professor Luca Benini in the Integrated System Laboratory, with the professor luca benigni in the integrated system laboratory, and his current interest include efficient deployment of machine learning model on microcontrollers, tiny transformers, scheduling and quantization.
• 4:36
  Speaker 1: So that's, that's a very great topic to talk, awesome yes go ahead, richard.
• 4:43
  Speaker 3: Okay, great.
• 4:43
  Speaker 3: Thank youilo for the great introduction and yes, I can say this work is a bit the quintessence and the perfect intersection between all those research interests that also form all the community around Edge, ai and TinyML.
• 4:56
  Speaker 3: So today the talk is about energy efficient generative AI deployment on the-source RISC-V heterogeneous SOCs so the new generation of SOCs we see emerging to tackle those very challenging workloads of transformers.
• 5:12
  Speaker 3: That runs generative AI.
• 5:14
  Speaker 3: So let's quickly dive in.
• 5:16
  Speaker 3: What are we first talking about when we mean GenAI and foundation model for TinyML?
• 5:22
  Speaker 3: So those models are the very famous ones, such as the ChartGPT models, the one that everyone used basically, even if you don't work in tech, or also the Meta Lama models, the one that are a bit more niche but also very known in the tech industry.
• 5:38
  Speaker 3: So the advantages of those foundation models here are definitely that you can use one backbone for many different tasks and you can also support mixed modalities such as images and voice.
• 5:51
  Speaker 3: So that's very important here and that's groundbreaking capabilities for many applications and some embedded application of this model already exists.
• 6:01
  Speaker 3: So, instance, with the AR glasses of Meta, for instance, the Ray-Ban here they have a very small budget in terms of power and in terms of also area here and they need to perform very complex tasks.
• 6:14
  Speaker 3: So that's where you want to have a really well-compressed and embedded foundation model to perform all these tasks.
• 6:21
  Speaker 3: That takes different modalities, like the camera on the glasses, also a mic and different other possible sensors that you can plug in here.
• 6:30
  Speaker 3: Also, you have, of course, the application with the nanodrones, where you want to have the most autonomous engine possible that also can fuse different modalities to build a representation of the world around and being able to navigate here.
• 6:45
  Speaker 3: So those generative and foundation model at the edge.
• 6:49
  Speaker 3: This is a trend that is increasingly getting getting bigger and bigger.
• 6:55
  Speaker 3: So there is this new trend of small language model and the release frequency of this model is definitely increasing.
• 7:01
  Speaker 3: So here is a quick timeline from a recent survey and you can see that in the year 2024 only there has been many small language models that have been released, and so this keeps growing and growing.
• 7:13
  Speaker 3: However, most of those small language models, they are still too large for tinyML applications because they range from 1 billion to 125 million parameters.
• 7:24
  Speaker 3: So it is still orders of magnitude higher than the requirements of tiny ML, of course, or small embedded system.
• 7:31
  Speaker 3: But the tiny language models are coming and we need as a community to be ready for this.
• 7:37
  Speaker 3: So let's quickly come back and give some context on the generative inference process so that we know we are all aligned on what it means.
• 7:46
  Speaker 3: So if you take the sentence here what does BULP mean?
• 7:50
  Speaker 3: And you break down this sentence in different tokens, so basically one token is one word and the sentence here is named the prompt.
• 7:59
  Speaker 3: It's the prompt that you give to your model.
• 8:01
  Speaker 3: So you give this prompt directly to your foundation model, here the llama, for instance.
• 8:07
  Speaker 3: Then it will generate, it will predict basically which token should be next or which word should be next.
• 8:14
  Speaker 3: Here it's parallel for the p of pulp.
• 8:16
  Speaker 3: Then you can append this new token at the front of your prompt.
• 8:21
  Speaker 3: This becomes your new prompt.
• 8:23
  Speaker 3: You send it again in the language model, you get the next token predicted, etc.
• 8:28
  Speaker 3: Etc.
• 8:29
  Speaker 3: And in the end you end up with the full answer basically of the language model to the question what does pulp mean?
• 8:36
  Speaker 3: It means parallel, ultra low power and it's also the name of the chip that we developed here at IIS.
• 8:44
  Speaker 3: So this generative model, they have two different modes.
• 8:47
  Speaker 3: They can work for two different modes for two very different purposes.
• 8:50
  Speaker 3: So it's important to quickly introduce them before we continue.
• 8:54
  Speaker 3: The first mode is the parallel mode that is used to process the prompt.
• 8:58
  Speaker 3: So when you give the prompt it processes all the tokens in parallel of this prompt and build up the context.
• 9:04
  Speaker 3: And this mode here is basically composed of matrix multiplication, so gems.
• 9:09
  Speaker 3: Then there is the autoregressive mod, which is the mod used to generate the answer Each time your model is looping to generate a new token.
• 9:19
  Speaker 3: That's in the mod you are and it stores the context here in the KV cache, and it's mostly composed of matrix to vector multiplication, so gemv, and uh, we'll not know that.
• 9:36
  Speaker 3: We know more or less what the workload looks like.
• 9:38
  Speaker 3: Let's quickly look at the constraint for the tiny ml heterogeneous soc.
• 9:39
  Speaker 3: So what kind of hardware are we talking about here?
• 9:41
  Speaker 3: We talk about the hardware with limited off-chip capacity, usually a maximum of around 64 megabytes for the MCU, and since it's off-chip, the read and write energy and latency are quite high.
• 9:54
  Speaker 3: So it's a high cost to access, sometimes necessary, but we try to minimize these transfers here.
• 10:00
  Speaker 3: Then we also have many smaller on-chip memory, typically 1 to 4 megabytes, and we finally have different memory hierarchies.
• 10:11
  Speaker 3: So it's represented with the green links on the right-hand side figure and it's made of software-managed caches, so it's basically scratch pads.
• 10:20
  Speaker 3: It doesn't have any hardware feature to prefetch and stuff like this.
• 10:24
  Speaker 3: So this is very suitable for workload that we know at compile time because we can predict everything and it increased the energy efficiency compared to regular caches.
• 10:34
  Speaker 3: However, this completely offloads a lot of complexity to the compiler because then the compiler has to perform static memory allocation.
• 10:44
  Speaker 3: It has to do also to plan all the memory transfer and to do some scheduling to decide when and until when do all the memory transfer from which memory to to another memory.
• 10:56
  Speaker 3: So that's, that's a lot of added complexity here, but it comes.
• 11:00
  Speaker 3: It's necessary if you want to reach the best energy efficiency you can get.
• 11:06
  Speaker 3: And talking about those specific heterogeneous SoCs there is a real Cambrian explosion in the number of heterogeneous SoCs that are proposed because each of these SoCs have their own quirks basically and so many flavors are developed and you need to evaluate how good is your hardware, your new SOC that you developed.
• 11:25
  Speaker 3: You need to run end-to-end workloads here.
• 11:27
  Speaker 3: You cannot only do micro-benchmarking.
• 11:30
  Speaker 3: That's not a good enough metric anymore.
• 11:32
  Speaker 3: So those heterogeneous SOCs they can contain some cores with different kinds of ISAs, so this most likely often changes.
• 11:41
  Speaker 3: Then it also has different neural processing engines with different capabilities and additionally, on the top of all that, there are different memory systems and hierarchy.
• 11:51
  Speaker 3: So if you take into account all these three different characteristics, where you think that there are many different possible choices for each of those characteristics, you always almost end up with a unique SOC, like each SOC is unique in its own way.
• 12:08
  Speaker 3: And then you need a flexible tool to deploy on those different SOCs without having to make a tool from scratch from the beginning.
• 12:16
  Speaker 3: So one example here is the Syracuse microcontroller.
• 12:20
  Speaker 3: We'll talk again about this specific MCU here, so it's good to give to everyone a quick glimpse of it first.
• 12:26
  Speaker 3: So this one has, for instance, a convolution or gem accelerator.
• 12:30
  Speaker 3: It also has a cluster of eight RISC-V 32-bit cores and it has a specific neural memory subsystem that is basically a weight memory attached to the NPU.
• 12:41
  Speaker 3: So what are the actual tools to deploy on those heterogeneous SOCs that exist in the state of the art?
• 12:48
  Speaker 3: So there are many, but they each focus on one specific SOC.
• 12:53
  Speaker 3: For instance, if you want to deploy on the STM32 MCUs, you will use the STM provided the QAI tool for this.
• 13:03
  Speaker 3: The STM provided the QAI tool for this.
• 13:05
  Speaker 3: If you want to deploy on the gap processors, you will use a tool given by the company GreenWave and so on and so on.
• 13:12
  Speaker 3: So this makes sense for industry.
• 13:14
  Speaker 3: Of course, you are proposing one specific class of SOC and you don't really care about supporting deployment for other SOC, which makes complete sense.
• 13:23
  Speaker 3: But it's very inconvenient for researchers Because you don't want in a lab where you develop, like ours, when you develop many different chips and you try different combinations, and where you open source all the hardware so other people can also take some of your chips and make some new ones.
• 13:39
  Speaker 3: You don't want everyone to rebuild a full deployment stack every time they need to deploy, to try a new architecture here.
• 13:49
  Speaker 3: Additionally, these tools here from vendors.
• 13:52
  Speaker 3: To the best of my knowledge, they don't support the specific Gen AI pattern yet, such as KV caching.
• 13:59
  Speaker 3: So our deployment stack what does it look like?
• 14:02
  Speaker 3: It always starts with the very famous PyTorch TensorFlow training framework.
• 14:07
  Speaker 3: Then we export the neural network into a graph representation named ONNX, which is a command representation to share neural network, and then this representation is fed into our tool deploy.
• 14:18
  Speaker 3: That then generates some code that can be run or compiled with your favorite compiler for your platform like ARM or R live.
• 14:26
  Speaker 3: So in this talk we will focus on deploy here and deploy here.
• 14:32
  Speaker 3: To be just to be clear, it generates a bare metal code that requires a minimal runtime support.
• 14:38
  Speaker 3: Just to you know, make have a development cycle that is faster.
• 14:41
  Speaker 3: You don't have to build a super fancy runtime for each of your platform, you just need the bare minimal and the general structure of deploy makes it easy to add new platform and complex network.
• 14:52
  Speaker 3: I'm stressing that a lot in this talk is because that's really one of the requirements of this tool is try to unify a bit this deployment tools, especially for researchers that do architecture research.
• 15:04
  Speaker 3: This tool Deploy has been published at ES Week 2024.
• 15:10
  Speaker 3: And so if, after the talk, you're curious to learn more about the tool, you can of course check the open source repo, but if you want more technical detail and research insight, you can of course check the paper.
• 15:22
  Speaker 3: So let's first dive into the front end.
• 15:24
  Speaker 3: What is this doing?
• 15:25
  Speaker 3: So what we get as an input of deploy is a graph that represents the neural network and there we first perform a graph lowering.
• 15:34
  Speaker 3: That is platform dependent.
• 15:36
  Speaker 3: So depending on which platform you want to deploy in, you will perform different operations here.
• 15:40
  Speaker 3: So you will fuse or insert operators, for instance, and the graph still remains fully ONNX compliant.
• 15:47
  Speaker 3: This is very useful, for instance, if you have a requantization that follows a matrix multiplication and you have a specific kernel that fuses those two operands.
• 15:57
  Speaker 3: Then in the lowering stage you want to fuse them in the graph in order to have one node for one kernel.
• 16:03
  Speaker 3: And this is the next step the nodes are matched with a specific low-level kernel that supports this operator.
• 16:10
  Speaker 3: So, for instance, we use libraries that are very famous like CMCisNN or PulpNN.
• 16:16
  Speaker 3: I mean, you can always add your own custom kernel, but the goal in the end is to have one specific kernel that has been made by experts to execute a specific operator of the graph.
• 16:29
  Speaker 3: And then, once we have this graph where each node is binded basically with one specific kernel, we need to handle the on-chip memory management like how do we move data around and stuff like this.
• 16:42
  Speaker 3: And this is handled in the mid-end and that's the deep part of Deploy.
• 16:47
  Speaker 3: Basically, because this mid-end here is quite complex, it's divided into two different stages.
• 16:51
  Speaker 3: The first stage performs the tiling and the second stage performs the memory allocation.
• 16:57
  Speaker 3: So during the first stage we tile by constraining our tensors here in order to slice them, and we divide those constraints into two different categories.
• 17:09
  Speaker 3: The first category is the operator constraint, which basically makes sure that you produce valid output for a certain operator.
• 17:17
  Speaker 3: Then the second category, the target constraint, which just makes sure it works well with the specific hardware target.
• 17:24
  Speaker 3: For instance, you may have a processor that supports SIMD operations and you have a kernel that uses this and it's very efficient for this.
• 17:33
  Speaker 3: But for instance, if you're in Integer 8 and you pack 4 Integer 8 in your world, then you may have a big performance boost if you can have tiles that are multiples of four in certain dimensions, for instance.
• 17:49
  Speaker 3: So you can enforce this just to give a performance hint to the tiler in this kind of constraint.
• 17:55
  Speaker 3: So from this constraint programming representation you get the symbolic buffer sizes that will be used later for the memory allocation.
• 18:04
  Speaker 3: And these symbolic buffer sizes, just to give a hint for the next stage, they represent the height of the block in the Tetris memory scheduling that we will see in the slide right after.
• 18:18
  Speaker 3: So now let's go to the second stage of the mid end of deploy, which is the memory allocation.
• 18:23
  Speaker 3: How do we do the memory allocation in deploy?
• 18:25
  Speaker 3: Well, it's not so simple but also not so complex to understand visually.
• 18:30
  Speaker 3: The we start from the graph here on the left hand side where each node is represents one operator.
• 18:37
  Speaker 3: Then from this graph we convert it into a tensor graph where each node this time represents a tensor.
• 18:44
  Speaker 3: And from this tensor operator we generate an adjacency matrix and we use a cost vector c where each element of this cost vector c it's the symbolic buffer size that you get from tiling stage.
• 18:57
  Speaker 3: And from this, from this a and c, a and C matrix and vector, you can get one Tetris memory allocation scheme.
• 19:08
  Speaker 3: So remember that this A matrix here it encodes the order of the block placement when you basically play Tetris to do your memory allocation.
• 19:19
  Speaker 3: So the core idea here is to solve this integer linear programming problem with jointly for tiling and the memory allocation solution to get a good solution, both, that that both gives a good tiling and also that respects all the memory constraint of our specific hardware.
• 19:38
  Speaker 3: But you can see here that if you use the topological order of the graph and to do the Tetris scheduling it leads to very suboptimal solution.
• 19:47
  Speaker 3: It basically creates some kind of a staircase allocation where you have very high defragmentation.
• 19:54
  Speaker 3: That's very suboptimal.
• 19:55
  Speaker 3: So we really don't want to do that here.
• 19:57
  Speaker 3: How can we fix this?
• 19:59
  Speaker 3: It's quite simple.
• 20:00
  Speaker 3: You can use permutation matrices, just generating a valid permutation matrices that respects the following property so every element has to be a Boolean, of course, and the sum of the rows and the columns have to be one.
• 20:13
  Speaker 3: Once you respect those properties, you have a valid permutation matrix, p, and by this quick linear algebra trick you can generate a new permutation matrix, a, prime, prime and a new associated cost vector, c prime.
• 20:27
  Speaker 3: Basically, it's like shuffling the order in which you place the block in this Tetris scheme here.
• 20:34
  Speaker 3: So you can see this example here with the coordinate transform we basically change the order with which we are placing the blocks and it leads to a better design point.
• 20:48
  Speaker 3: Here the solver co-optimizes the tiling solution with the memory load schedule in order to find a solution that fits all the requirements, so the specific on-chip memory, but also the specific tiling requirements to validate to keep the correctness of the operations.
• 21:07
  Speaker 3: Now let's quickly go through one example.
• 21:09
  Speaker 3: So the tiny language model on Syracuse.
• 21:13
  Speaker 3: So we took a TLM based on Lama2.c from Carpati and we trained it on the tiny story dataset, just to get an example of a tiny language model.
• 21:25
  Speaker 3: So it's very small, it's only 4.6 million parameters, but it's using the Lama2 architecture and it's a good demonstrator for all case here.
• 21:34
  Speaker 3: So we completely quantize this network in integer eight using Quantlib, which is an open source quantization tool that we use and we developed here at the lab at IIS.
• 21:45
  Speaker 3: And for the quantization of the nonlinear operator, such as Softmax or the RMS normalization, we use polynomial approximation.
• 21:57
  Speaker 3: So let's quickly come back to the specific platform that we are targeting here, which is Syracuse.
• 22:01
  Speaker 3: Quickly introduced before In this platform we have again this CNN accelerator named N-Eureka.
• 22:08
  Speaker 3: We also have a lot of on-chip memory here 4 megabytes of SRAM weight memory, 2 megabytes of L2 SRAM and 256 kilobytes of L1 TCDM.
• 22:19
  Speaker 3: Then we also have a high bandwidth link to the NPU, both from the neural memory subsystems to stream the weights and from the L1 to stream the activations.
• 22:31
  Speaker 3: Finally the L1 to stream deactivations.
• 22:34
  Speaker 3: Finally, we have an Octa-Core RISC-V cluster here that supports core with the DSP extension, integer SIMD support and also post-increment and hardware loop extensions.
• 22:43
  Speaker 3: So all of these things are extensions to make them very efficient at crunching big numbers, such as performing matrix multiplication or other dsp tasks.
• 23:00
  Speaker 3: So our benchmark setup, before we talk about numbers siracusa is silicon proven, so it has been typed out and we perform measurements on board directly.
• 23:05
  Speaker 3: So we benchmark three different configurations here.
• 23:07
  Speaker 3: The first configuration is with the Octa-Core cluster only.
• 23:12
  Speaker 3: Then the second configuration is with the NPU but without the neural memory subsystem.
• 23:18
  Speaker 3: And the final configuration is the complete SoC with the NPU, the neural memory subsystem and, of course, the Octa-Core cluster.
• 23:26
  Speaker 3: So first we benchmarked the first benchmark we performed Basically.
• 23:31
  Speaker 3: The goal was to see if it was beneficial to use the KV caching and, if yes, by how much.
• 23:37
  Speaker 3: So we performed the cumulative latency and energy of a 256-step inference and we looked at what it's like with and without KV caching.
• 23:47
  Speaker 3: So we know that if we don't use KV caching we do a lot of redundant operation.
• 23:51
  Speaker 3: But we wanted to quantify how it was translating in terms of latency and energy.
• 23:56
  Speaker 3: So we can see that it's definitely non-negligible and not supporting the KV caching here is a big loss and it's 23x of speed up in terms of latency and 26x in terms of energy.
• 24:07
  Speaker 3: So that's definitely super costly to ignore the KV caching and this should not be done.
• 24:13
  Speaker 3: So you need the deployment tool that supports this specific paradigm.
• 24:18
  Speaker 3: Then we looked at the breakdown of where do we spend time, depending on the specific category of layer, for the parallel and the autoregressive mode in an end-to-end inference scenario.
• 24:28
  Speaker 3: So we looked at this for the three different configurations I introduced earlier, so the cluster, the cluster plus the accelerator and the complete SOC.
• 24:38
  Speaker 3: So for the softmax, the RMS norm and the activation it's all the same, which makes sense because they are always executed on the cluster, whatever the configuration.
• 24:49
  Speaker 3: But now when we look at the gem, we see there is a big difference between the three configurations.
• 24:54
  Speaker 3: You clearly see that when we use the NPU we can compress the time we spend in the gem.
• 25:00
  Speaker 3: Consequently, we actually gain 59% of the runtime compared to the octa-core regime.
• 25:06
  Speaker 3: Also, it's very important to clarify that we offload only linear layers to the NPU.
• 25:12
  Speaker 3: So we offload matrix multiplication between wait and activation, not between activation and activation.
• 25:17
  Speaker 3: Finally, we have a little bit of data movement and it's quite homogeneous across the different configuration.
• 25:25
  Speaker 3: So from this breakdown here we can clearly pinpoint some of the properties of the parallel mode.
• 25:31
  Speaker 3: Here we can clearly pinpoint some of the properties of the parallel mode.
• 25:33
  Speaker 3: Here we clearly see that we are in a compute bound regime and that we benefit maximally from using the NPU to relieve a bit this compute boundness.
• 25:40
  Speaker 3: Then we did the exact same thing, but with the autoregressive mode here.
• 25:45
  Speaker 3: So again same story for the softmax and for the activation.
• 25:50
  Speaker 3: It is the same because it's executed on the cluster.
• 25:54
  Speaker 3: But now when we look at the matrix to vector multiplications, the gem V, you see it's a very different trend than with the parallel mode.
• 26:03
  Speaker 3: So the NPU here still provides 16% of speedup on the gem V, but way less compared to the parallel mode.
• 26:10
  Speaker 3: That's clearly because the gem V has a way lower computational intensity compared to the Gem and so we are way more memory bound, so memory bound that the accelerator is way less beneficial here.
• 26:22
  Speaker 3: Now if we, if we look at the data movement, we see that it's a consequent part of the of the runtime here, but that when we use the neural memory subsystem, when we add these quirks basically to the configuration this allows basically if you remember in the schematic before it was the neural memory subsystem basically gives more bandwidth towards the accelerator.
• 26:47
  Speaker 3: So it relaxes the NPU memory boundness by giving more bandwidth, but it still stays in a memory bound regime.
• 26:53
  Speaker 3: And that's the two properties we can extract from the two insights we get from this.
• 26:59
  Speaker 3: Benchmark here is that the autoregressive mode is definitely used in a memory bound regime and leverages mostly the octa-core cluster.
• 27:09
  Speaker 3: So see, we have this new kind of workload that can be run in two different modes and you need the two different modes, otherwise you lose a lot, as we saw in the previous benchmark where we tried not using at all the kvc caching.
• 27:22
  Speaker 3: So you need these two mode.
• 27:23
  Speaker 3: So you have an heterogeneous workload and an heterogeneous workload need heterogeneous hardware, fundamentally if you want to be able to execute all the mods efficiently.
• 27:33
  Speaker 3: So the text of this talk here is that we demonstrated the end-to-end inference of a tiny language model on an heterogeneous SoC.
• 27:43
  Speaker 3: With Deploy.
• 27:45
  Speaker 3: We also benchmarked different configurations of the SoC for the two modes of the network to isolate different trade-offs and collect insights, and we finally reached a throughput of 340 tokens per second at 490 microjoules per token with this specific configuration measured on chip.
• 28:02
  Speaker 3: And the best part of it is that it's all open source, both in the paper, of course, but also the tool, deploy and additionally Quantlib, which is the tool we used to quantize the network.
• 28:14
  Speaker 3: So with this, that's the end of the talk and I think we can go through the Q&A now.
• 28:25
  Speaker 2: Thanks, victor.
• 28:26
  Speaker 2: Excellent presentation and well taken topic.
• 28:34
  Speaker 2: Topic In the meanwhile we wait for questions from attendees.
• 28:49
  Speaker 2: I would like you to address some of mine.
• 28:52
  Speaker 2: So I took my notes and clearly ethereogenius was a key word in your presentation and you addressed this from the main perspective of both hardware and software.
• 29:05
  Speaker 2: So let me try to ask you something around this keyword, in no specific order.
• 29:23
  Speaker 2: Ethereogeneous means you pointed the different deep learning frameworks, and different deep learning frameworks means many operators, hundreds of operators.
• 29:37
  Speaker 2: You know operators that do conv convolutions, different flowers of convolutions, and you know also mathematics, mathematical operators, tensor operators, different meanings of this operator across the deep learning framework, and and and so on.
• 30:01
  Speaker 2: So how do you address this, this complexity, through your, your tool chain?
• 30:06
  Speaker 2: What are the principle or or the, the thought or the advices you you think to share?
• 30:14
  Speaker 3: well, that.
• 30:14
  Speaker 3: That's a very good point, danilo, because, yes, the yes, the different, like there are many like deep learning framework that we take the network from.
• 30:23
  Speaker 3: I think if I quickly come back to this slide describing our stack here, people will also have an overview of what I'm talking about A lot of animations, but here we are.
• 30:33
  Speaker 3: So here, yes, I mean there are many different framework upstream.
• 30:37
  Speaker 3: So here, yes, I mean there are many different frameworks upstream.
• 30:41
  Speaker 3: And the way we, when we built the framework, the way, the reasoning that we, the reason why we have the ONNX in the model export, is to have this kind of harmonization step between all the different frameworks.
• 30:51
  Speaker 3: So all these frameworks, they support an export in the ONNX format, which has a specific frozen dialect, an export in the ONNX format, which has a specific frozen dialect.
• 31:01
  Speaker 3: And if we support the operators from this dialect, then we support the operators from every deep learning framework that supports export in ONNX.
• 31:10
  Speaker 3: So that's why we have this layer here to harmonize all the operators into a dialect that we know exactly, instead of trying to directly support, for each different tool, their own dialect.
• 31:24
  Speaker 2: OK, thank you.
• 31:25
  Speaker 2: And staying on this slide and on this processing, do you use ONX as a representation during the manipulations that are performed by this stack?
• 31:42
  Speaker 3: Yes, exactly.
• 31:42
  Speaker 3: So here I didn't introduce the backend.
• 31:45
  Speaker 3: I have some slides in backup for this, but this is really just code generation, so it's not so interesting.
• 31:51
  Speaker 3: But during the frontend especially, and also during the midend, we keep this ONNX graph and during the frontend we modify it heavily.
• 31:58
  Speaker 3: We keep this ONNX graph and during the front-end we modify it heavily, but we keep it ONNX compliant.
• 32:02
  Speaker 3: So it allows us basically to re-export the ONNX graph or like to just export it or save it at different point of the flow.
• 32:10
  Speaker 3: To clearly visualize what we are doing within the tool, like how do we transform the graph across the execution of the tool?
• 32:17
  Speaker 2: And this help also verification, because you could cross check against the onyx runtime exactly the outcomes of your transformations yes, yes, yes, that was that.
• 32:29
  Speaker 3: That was one of the big point also of uh, why we want to keep the graph on the next compliant.
• 32:35
  Speaker 2: It's because of and yeah, you said you didn't touch the backend, but there is also the possibility to generate C code Now, which maybe is not interesting.
• 32:53
  Speaker 2: You said but let me staying on these slides, ask you another thing.
• 32:58
  Speaker 2: Say but let me staying on these slides, ask you another thing.
• 33:03
  Speaker 2: You know that there are tools for automated code generation, for example, pyjit.
• 33:07
  Speaker 2: Um, what do you think about, uh, this possibility to automate C code using a tool like PyGIT?
• 33:25
  Speaker 2: Is that effective or maybe less optimal in your experience?
• 33:28
  Speaker 3: Usually it's a bit complicated because most of the tools they assume a traditional architecture that uses caches, so they don't have to do explicit data transfer and they don't have to do static memory allocation.
• 33:43
  Speaker 3: So that's why we could not really connect or extend an existing tool here.
• 33:49
  Speaker 3: But there is still room for improvement because basically with Deploy we still use kernel libraries and so if you have a tool that, for instance, automatically fine-tune your kernel based on the specific argument to call this kernel, then you could use in a collaborative aspect deploy with another tool that, once deployed, generated the C code.
• 34:14
  Speaker 3: The tool will basically fine-tune the specific kernels based on the argument of this kernel.
• 34:24
  Speaker 2: Thanks.
• 34:24
  Speaker 2: When you speak about these low-level kernels, it means that you took the load of optimizing operators on the hardware architecture right.
• 34:36
  Speaker 3: Yes, architecture right.
• 34:38
  Speaker 3: Yes, because how it goes when we have a new class of cluster, for instance with new ISA extension.
• 34:49
  Speaker 3: In the lab, or in most of the labs, I think, usually people start by programming, doing micro-benchmarking, so just looking at how efficient is a matrix multiplication here, just a very simple kernels, and they usually build a simple kernel library to validate the concept and once this is done, we basically reuse those kernels because they have been made by experts already and we just enable the execution of a more complex workload that also takes into account all the data transfers and all the movement, with the whole memory hierarchy instead of just the L1, for instance.
• 35:26
  Speaker 2: Sure, sure good, if you are available.
• 35:30
  Speaker 2: I would continue with some questions.
• 35:32
  Speaker 2: You know, supporting Keras, there is the possibility and sometimes is not said that all the kernels are explicitly known ahead, because maybe these are the intellectual property of the developer and the Keras custom layers are a great asset to capture specificity and proprietary implementation.
• 36:05
  Speaker 2: So how would you, your methodology, your workflow, deal with this special purpose kernel, for example, the Keras custom kernels.
• 36:18
  Speaker 3: For example, the Keras custom kernels.
• 36:19
  Speaker 3: The usual how we go usually is that we have always fallback kernel that are less tailored to a specific hardware but functionally correct.
• 36:29
  Speaker 3: To validate the correctness first and then, depending of which kernel we have access or which specific operator we want to optimize because we see it takes a big chunk of the runtime.
• 36:40
  Speaker 3: For instance, we just write it ourselves or we ask the people is it possible to use it in an open source manner?
• 36:49
  Speaker 3: For instance, we did this with GreenWave Technologies.
• 36:53
  Speaker 3: They have a very well-engineered and super specialized library for convolution layer that is way better than ours.
• 37:01
  Speaker 3: So we discussed with them and we in the end are currently integrating it into Deploy such that we can use also their own kernel.
• 37:10
  Speaker 3: So that's usually how it goes.
• 37:13
  Speaker 3: But if the kernels are proprietary and we cannot have access to them, we fall back on the less optimized but still functionally correct functionally correct yeah, yeah, I think that's a great way to support uh, unknown, unknown kernel and at least keep the application running yeah, yes, that's all thanks to the onyx dialect, that is, you know, the the source of truth that we are both.
• 37:37
  Speaker 3: We are both talking the same dialect, so we know exactly how to fall back on which kernel we need to fall back for a specific operator.
• 37:46
  Speaker 2: So, from your point of view, nx is a de facto standard in terms of representation of ML models.
• 37:56
  Speaker 3: I think it became a bit of a standard standard, or at least what we use, what I see other researchers using.
• 38:01
  Speaker 3: I see a lot of researchers using onnx to share to visualize the model.
• 38:05
  Speaker 3: I'd say it's less and less true because I'm more and more tempted to directly support PyTorch operators, because I think that PyTorch is getting a lot of traction in the research world.
• 38:19
  Speaker 3: But still PEN software is very used also in industry, so I don't want to tailor it to a specific framework yet.
• 38:26
  Speaker 3: So yes, I think ONX is still the standard here.
• 38:32
  Speaker 2: I tended to also agree with you, but it happened at some time that you know conversion between formats still happen because there are some converters on GitHub that helps from Nix, keras, tf Lite.
• 38:48
  Speaker 2: You know this type of converter and sometimes they mix the channel location within the pipeline.
• 38:58
  Speaker 2: So you would expect everything channel first or everything channel last, but sometimes it happens that channel are messed in between, and so how do you deal with this intricacy?
• 39:10
  Speaker 2: I mean, it's really, it breaks the regularity you know.
• 39:18
  Speaker 3: Yes, definitely.
• 39:19
  Speaker 3: We saw this thing happening and we had to hack a bit the converter to make sure it was returning the thing in the specific layout that we like, because if the layout of the input is not assumed correctly there is nothing you can do.
• 39:34
  Speaker 3: But what we do in Deploy it is that we automatically we say which layout by default you want to use for each operator and based on this we insert the transposition to make sure that the specific kernel that use, for instance channel last or channel first, will get the correct data.
• 39:53
  Speaker 3: So that's also one transformation performed in the front end here.
• 40:02
  Speaker 2: Sounds great.
• 40:02
  Speaker 2: Thanks, Victor, and I'm taking advantage for a few minutes still.
• 40:06
  Speaker 2: You know, quantization integer 8 is well known, but sometimes there are also other possibilities.
• 40:16
  Speaker 2: I'm thinking about this experimental 16x8 provided by TF Lite or FP16 and sometime through QKeras.
• 40:30
  Speaker 2: You have all the possibility in fractional or integer between 1 and 32 bits.
• 40:37
  Speaker 2: What's your experience on dealing with these ethereal geniuses still standing on the keywords?
• 40:45
  Speaker 2: That was driving your presentation.
• 40:47
  Speaker 2: How do you stand in front of these many possibilities in terms of bit depth and representations about quantization depth and representations about quantization.
• 40:59
  Speaker 3: In the deployment part.
• 41:00
  Speaker 3: It's actually not so hard in the deployment part to support.
• 41:05
  Speaker 3: I think the real complexity lies in the hardware and in the quantization tool themselves.
• 41:11
  Speaker 3: Because from the point of view of the deployment tool it's just a new operator and you need to.
• 41:19
  Speaker 3: The annoying thing is that you most likely will have to build a specific kernel to perform if you want to perform efficiently those operations.
• 41:27
  Speaker 3: For instance, for the gem to fuse the quantization with, or like to fuse the way they perform the quantization with the gem.
• 41:36
  Speaker 3: And for less important operator, you may have a very default kernel that is not so optimized but since it is a small part of the runtime it's not so bad.
• 41:46
  Speaker 3: But I think the real difficulty here is to have a runtime to execute those fancy quantization scheme.
• 41:55
  Speaker 3: So like you need to have a hardware that support this specific format.
• 41:58
  Speaker 3: I'm also thinking of the recently released, you know, mx format, the micro scaling format.
• 42:04
  Speaker 3: Um, that is very interesting.
• 42:07
  Speaker 3: It's a way to do low precision floating point and to basically have a small block that share the same scale for the quantization and you need to have the platform that support this.
• 42:18
  Speaker 3: This is new type, basically if you want to be able to simulate it and to do the quantization, and that's currently, I think, one of the bottleneck for quantization.
• 42:32
  Speaker 2: Thank you, so much, thank you so much.
• 42:34
  Speaker 2: And you say the short SLM are Getting getting smaller, but not enough and so, in general, we need at the edge generative model uh, smaller and smaller, but how much?
• 42:57
  Speaker 2: Smaller and smaller, but how much?
• 43:04
  Speaker 2: What's your ballpark, in your experience or in your vision, even in other cases, not only SLM that we could achieve to make this, this new wave of edge gen?
• 43:14
  Speaker 2: I happen.
• 43:16
  Speaker 3: I think the ballpark would be around like a million, like a hundred million parameters, a little bit below hopefully, like maybe 50 million parameters there, with aggressive quantization, so going a bit lower than 8-bit.
• 43:30
  Speaker 3: You could start to fit this into, for instance, like you know, the L3 memory of those tiny platforms here and you can start to have very meaningful applications.
• 43:43
  Speaker 3: One application I think about it's not generative AI, but it's still foundation model it's using the network named DenoV2, which is basically a backbone for any visual task.
• 43:54
  Speaker 3: So from this backbone you can do sentiment analysis, you can do image classification and you can do segmentation of images, for instance, with only one backbone.
• 44:03
  Speaker 3: And this backbone is 21 million parameter.
• 44:05
  Speaker 3: So it would clearly fit in this kind of use cases, where it would be embedded on a very small processor at the very edge and where you could have a lot of applications that would benefit from the fact of Deno being a foundation model, where you can use one backbone to do many different tasks right after.
• 44:27
  Speaker 2: So you name it the backbone.
• 44:29
  Speaker 2: It sounds like the more the technology advances we are able to find, as you said, a common denominator the backbone.
• 44:42
  Speaker 2: So here the it's an opportunity.
• 44:45
  Speaker 2: I mean to really do a very optimal mapping about it.
• 44:49
  Speaker 2: Do you see this backbone becoming as well a de facto standard type of machine learning graph?
• 44:59
  Speaker 3: yes, yes, yes, definitely like this.
• 45:01
  Speaker 3: The clearly, if you, if you really think about a full product, like, for instance, what the the the ir glasses I showed before, the one from the collaboration between rayban and meta, this is the only way to go if you are reasonable, because this is, you cannot afford anymore to have a specific network for every task, considering how much space it takes, like having a common backbone for those tasks will lead to a way.
• 45:28
  Speaker 3: Basically, you will be way more efficient in terms of parameter per accuracy for each task, if you prefer and yes, just taking, starting from what you said about their glasses even the nano drone.
• 45:46
  Speaker 2: Clearly, this system must be very energy efficient.
• 45:50
  Speaker 2: So you were commenting 490 microjoules per token.
• 46:00
  Speaker 2: So let me elaborate a little bit the question.
• 46:05
  Speaker 2: So, from another aspect, the architecture that you spoke and the tool that are able to perform the deployment on on the architecture, use memory.
• 46:17
  Speaker 2: There is a lot of memory in the device, but it's not used as a compute asset.
• 46:23
  Speaker 2: So so the question is how do you see, maybe in five years from now or even less?
• 46:32
  Speaker 2: as you judge going on this energy efficiency with or without new opportunities, implying memory as a compute asset and not only as a storage asset memory as a compute assets and not only as a storage assets.
• 46:54
  Speaker 3: Definitely, I mean, I think we it's easier.
• 46:56
  Speaker 3: Either we have this kind of, you know, new 3d stacking technologies that allow us to drastically improve the number of bytes per centimeter square or millimeter square that we can store.
• 47:11
  Speaker 3: So it would increase the density of the memory by a lot and would allow us to keep up with our current way to do the things.
• 47:19
  Speaker 3: But if this is not happening, of course, the way to go is the in-memory computing or the closed-memory computing, where you move some of the compute units very, very close to the memory there, and I think this is a this is definitely a trend.
• 47:34
  Speaker 3: It's a huge research field and it has already proved uh to be, to be efficient.
• 47:40
  Speaker 3: And, um, I expect that if those uh 3d scaling technologies are not coming soon, uh up to speed for small microcontrollers, for instance, or, you know, edge devices, then that the computing memory trend will keep growing and keep growing and start being implemented here.
• 48:01
  Speaker 2: And, yes, I also really hope that researchers will put their best energies in helping to make the system even more and more energy efficient, because that was we need about generative ai at the edge.
• 48:18
  Speaker 2: I've been played with some of the shelf hardware and definitely there is a huge room of of improvement, which I think is a is a huge opportunity for researchers, for people like you and and others, and I hope this will be addressed also in the interest of the community, of the aj community.
• 48:39
  Speaker 2: So, victor, I would like really thank you so much for your passion.
• 48:47
  Speaker 2: You spent 20 minutes with my questions.
• 48:51
  Speaker 2: I hope I didn't torture you, it was not, uh, wanted by by myself, but really I I enjoyed this interaction with you and the topic that you uh that you exposed.
• 49:05
  Speaker 2: So thanks so much for all the passion and all the answers that you provided were really much much helpful, and I hope the the community will also uh take advantage of of your talk.
• 49:22
  Speaker 2: Thanks so much and let's meet at the next possible event why not in Austin?
• 49:32
  Speaker 3: Sure, sure, maybe, maybe I'll discuss with my professor about that, but thanks for inviting again.
• 49:37
  Speaker 3: That's always amazing to be here in these talks, so thanks a lot, danilo.
• 49:43
  Speaker 2: Thanks to you, victor, and, I think, thanks to everybody to attend and to Pete and Rosina.
• 49:50
  Speaker 2: Talk to you to the next time.
• 49:53
  Speaker 2: Thank you, ciao.
• 50:01
  Speaker 1: Thank you.