Latest / EDGE AI POD / Optimization Techniques for Powerful yet Tiny Machine Learning Models

Transcript

Optimization Techniques for Powerful yet Tiny Machine Learning Models · EDGE AI POD

• 0:00
  Speaker 1: Hello everybody, thanks to join us today.
• 0:03
  Speaker 1: You are connected to the TinyML Talks series.
• 0:08
  Speaker 1: My name is Danilo Pau and I will be the session chair of today's talks by Russell Clayne, technical Director, siemens EDA.
• 0:21
  Speaker 1: The title of his presentation is Making ML Tiny with High-Level Synthesis.
• 0:28
  Speaker 1: Before to leave the floor to Russell, let me remind a couple of things.
• 0:33
  Speaker 1: First of all, these events, like all the events organized by TinyML Foundation, are possible thanks to the great support of the TinyML strategic partners, which are strongly committed to support all the operations of TinyML, so that the community takes a lot of benefit from what we share, like today's presentation from Russell.
• 1:02
  Speaker 1: So let me remind you what are the names of the strategic partners AIZ, arduino, arm, brainchip, edge, impulse, embedur, efficient Greenways Technology, gravity, imax, imagemob, infineon, innaterra, notarai, nxp, p&g, corvo, qualcomm, renesas, schneider Electric, sensimil, sony, silicon Labs, stmicroelectronics, synaptics, cintiand TDK.
• 1:33
  Speaker 1: The community share a lot of material, a lot of information.
• 1:39
  Speaker 1: There are a couple of things I need to underline.
• 1:42
  Speaker 1: First of all, the meetup.
• 1:44
  Speaker 1: The meetup are a great way to meet in-person people, even at local level, at national level, depending where the groups are based geographically.
• 1:58
  Speaker 1: And I've been experiencing the meetup personally and it's been a great occasion to interconnect, to network and to share progress and learn what also the other community members do.
• 2:16
  Speaker 1: So meetups are very spread in many countries, very much populated, and please, if you didn't yet join, join a meetup at the meetup closer or even help to organize the meetup at your premises.
• 2:33
  Speaker 1: The second thing which is a great tool, a great asset for the community, is the LinkedIn TinyML community, which is very populated by content, by information, well followed.
• 2:49
  Speaker 1: So follow and subscribe, if you didn't do it yet.
• 2:55
  Speaker 1: The LinkedIn community, because it's another way to stay updated with all the operations and all the information the community shares through the TinyML Foundation.
• 3:05
  Speaker 1: Youtube YouTube channel from TinyML is a great asset.
• 3:11
  Speaker 1: The amount of content shared in it is impressive, really awesome.
• 3:18
  Speaker 1: Just to make an example, different working groups have been established in the past.
• 3:25
  Speaker 1: They organized forums, they organized online events.
• 3:32
  Speaker 1: These have been recorded and are available there, so you can get a lot of stuff to learn what's around and what is at the heart of TinyML and even the last innovation, innovation which are brought forward.
• 3:49
  Speaker 1: One to name is the generative AI initiative that TinyML started at the beginning of this year and, through the forum and through the summit, provide a lot of information, a lot of progress, a lot of insight to the community, so it's a perfect asset to learn and to play back what maybe you have missed.
• 4:15
  Speaker 1: Tiny MLMEA is very rapidly approaching.
• 4:18
  Speaker 1: It will be held in Milan, in Italy, north of Italy, june 24th to 26th.
• 4:25
  Speaker 1: Please register now If you didn't register yet.
• 4:29
  Speaker 1: You shall attend to this event.
• 4:32
  Speaker 1: That's the flagship event in Europe after the summit.
• 4:37
  Speaker 1: It will be certainly a great event, very exciting.
• 4:42
  Speaker 1: A lot of information will be shared.
• 4:44
  Speaker 1: A great event, very exciting.
• 4:45
  Speaker 1: A lot of information will be shared.
• 4:44
  Speaker 1: You can get a sense of it through the program, which is available online.
• 4:50
  Speaker 1: So, indeed, plan your attendance we are a month about from it and you can enjoy also Milan during this time, as well as all the events that will be organized by 10 EML.
• 5:07
  Speaker 1: So let me, before to leave the floor to Russell Klein, let me read his bio.
• 5:13
  Speaker 1: Russell is Technical Director with Siemens CDA, working in the high-level synthesis group.
• 5:19
  Speaker 1: His work focuses on optimizing hardware software system performance and efficiency through repartitioning with HLS.
• 5:27
  Speaker 1: He also is an adjunct professor in the master program at Portland State University, teaching in the College of Electrical and Computer Engineering.
• 5:39
  Speaker 1: He has been awarded by six patents in the area of hardware software system design and verification.
• 5:46
  Speaker 1: So thanks a lot, russell, for your great availability, and to you the floor.
• 5:52
  Speaker 2: All right, thank you.
• 5:53
  Speaker 2: So I'm going to be talking today about making ML tiny with high-level synthesis.
• 5:59
  Speaker 2: Here's a little bit of an agenda of what I'm going to be describing describing so we're going to start by talking about pruning the networks pruning convolutional or deep neural networks.
• 6:13
  Speaker 2: I won't go into too much detail on pruning, as I'm guessing this audience is probably pretty well familiar with it, and then I'll go in to talk about quantization, and this is going to be both post-training quantization as well as quantized aware training.
• 6:29
  Speaker 2: We'll talk about saturation and rounding, which have particular influences when we're doing aggressive quantization, and then we'll talk about creating a tiny accelerator using high-level synthesis and a program that we've been working with here at Siemens EDA and the high-level synthesis group.
• 6:49
  Speaker 2: It's a project called HLS for ML, so let's dive into it.
• 6:58
  Speaker 2: So I am specifically going to be talking about creating a hardware implementation of a bespoke accelerator, so a custom bit of hardware that is going to be used to efficiently accelerate the processing of an inferencing algorithm.
• 7:19
  Speaker 2: And I know that there's a bunch of tiny ML that will get run on CPUs and GPUs and TPUs and really anything that reduces the overall computation and the size of the data that we're processing is going to make about custom hardware or bespoke accelerators for accelerating machine learning algorithms, and these would be targeted either at FPGAs or ASICs.
• 7:57
  Speaker 2: Now, in order to make machine learning tiny as well as fast, to make inferencing hardware smaller and to make the footprint of the algorithm smaller, one of the things that we need to do primarily is to reduce the number of weights and features, and then also to reduce the size of the weights and features.
• 8:17
  Speaker 2: And in order to make the algorithm go faster, we want to reduce the number of calculations that we need to do, and, interestingly, this is the same as trying to make the infrastructure smaller.
• 8:30
  Speaker 2: So by shrinking the number of weights and features that we're operating on, we can both shrink the size of the network and help it go faster.
• 8:38
  Speaker 2: Also, to go faster, one of the things we want to do is perform more calculations in parallel, so that's going to increase the amount we want to do is perform more calculations in parallel, so that's going to increase the amount of work we can get done per unit of time.
• 8:49
  Speaker 2: That's kind of at odds with making the inferencing hardware smaller, because as we add parallel execution units, we're going to make the design larger.
• 8:58
  Speaker 2: The other thing we want to look at is reducing data movement.
• 9:02
  Speaker 2: Data movement takes time, it takes energy, and so by customizing the implementation of the accelerator, we can cache data strategically to be able to reduce that data movement.
• 9:17
  Speaker 2: So let's start by talking about pruning.
• 9:20
  Speaker 2: I actually went to try and find a very crisp definition of pruning, and it seems that it's a really broadly defined term.
• 9:30
  Speaker 2: So my use of it here in today's talk is really going to be about simply removing data and computations from a neural network, kind of.
• 9:38
  Speaker 2: However you do it, and there's a lot of different algorithms for it, and within the context of pruning, I am considering the removal of layers, channels, individual filters and then individual weights within the filters.
• 9:55
  Speaker 2: And removing filters or weights could involve actually removing them from the network or just zeroing them out, which means they'd have no more influence on the computations that are taking place.
• 10:08
  Speaker 2: Now, as you prune a network, one of the things that you need to keep in mind is that it can create irregular data access patterns or regular data access patterns, and some hardware implementations are going to work really well with regular access patterns.
• 10:26
  Speaker 2: Other hardware implementations are going to work really well with the irregular data access patterns.
• 10:32
  Speaker 2: So you need to keep in mind as you do the pruning what type of hardware are you targeting and do you want to maintain the regularity of that data access?
• 10:42
  Speaker 2: Because that can have a big impact on the performance and the efficiency of the algorithm that's being built.
• 10:51
  Speaker 2: Now, one of the things that we typically do when we're trying to reduce the size of the network is to go through each of the layers and do a sensitivity analysis of the accuracy of the network versus the number of channels that we have within each of the layers.
• 11:13
  Speaker 2: And here's an example, and it actually looks very typical for the types of algorithms that we look at.
• 11:20
  Speaker 2: This is based on MNIST.
• 11:21
  Speaker 2: Look at, this is based on MNIST, and what we did was took an implementation and this is the first dense layer.
• 11:31
  Speaker 2: As this was authored, it had 500 channels, and so what we did was simply a sweep where we went from 500 down to one and reduced the number of layers, retrained the network and calculated the accuracy, and what we find is that, going from 500 down to about 100, there's really no meaningful drop in the accuracy of the network.
• 11:55
  Speaker 2: As we go from about 50, from 100 down to about 50, it starts to drop off just a little bit, and somewhere right around 25 is where it goes sort of over the waterfall, where it drops down really dramatically.
• 12:10
  Speaker 2: And so if we really want to aggressively reduce this network, we'd be looking at a total channels of 25 or 30.
• 12:20
  Speaker 2: And you actually notice it's a little hard to see on this graph, but right before it goes over the waterfall we actually get a slight increase in accuracy.
• 12:28
  Speaker 2: And this is something we commonly see is that you just get a little bump there and then it falls off, and this is due to the quantization effect within the algorithm.
• 12:41
  Speaker 2: So, as I said, there's lots of algorithms for pruning.
• 12:44
  Speaker 2: I'm not going to spend a lot of time on all the different potential opportunities because I think that you guys probably know this pretty well, but this is just one example of the type of pruning that we do prior to going into quantization and the generation of the hardware.
• 13:02
  Speaker 2: And so this was an MNIST network that we found that had a particular implementation, the MAC operations as it was defined for the data center in the traditional machine learning framework.
• 13:16
  Speaker 2: We had about 12 million MAC operations and about 5 million parameters, and I think this is pretty typical for an MNIST network After we had gone in and very aggressively pruned it.
• 13:27
  Speaker 2: We were down to about half a million MAC operations and about 150,000 parameters.
• 13:34
  Speaker 2: One of the things I also keep track of is the minimum data transfer, which means this is the amount of data that, no matter how efficiently we cache things, no matter if we do things in exactly the right order, how much data is going to need to be moved from our weight store into our computational units.
• 13:51
  Speaker 2: And what we see in this case is that we had more than a 95% decrease in all of these metrics and our accuracy went down by 0.3%.
• 14:04
  Speaker 2: So for very little loss of accuracy we were able to save a lot of computation and area.
• 14:11
  Speaker 2: Now we do pruning typically manually and iteratively, but I recently saw some work by Dr Han over at MIT, who runs a lab there, and he's started to do some automated or AI-driven search for the optimal network architectures as well as doing model compression and handlemitedu.
• 14:35
  Speaker 2: So he's got a bunch of tiny machine learning work being done there.
• 14:41
  Speaker 2: At Embedded World this year there was a group that had furthered, that had taken that work and added in the notion of arbitrary quantization to further reduce the model size and built on top of this AI-driven network search, and that's actually some really interesting work.
• 15:00
  Speaker 2: So the next step, after we've printed the network, is to go into quantization.
• 15:08
  Speaker 2: If we're using an off-the-shelf machine learning accelerator, a TPU or GPU, we're probably going to end up using floating point.
• 15:17
  Speaker 2: Some of the TPUs and NPUs do support a couple of different levels of quantization, but in building a bespoke implementation we can build it to anything that we want and we can then customize the hardware to exactly what's going to be used.
• 15:35
  Speaker 2: So the first thing we'll usually do is go from a floating point to a fixed point, and this saves us about 50% of the area of the multiplier.
• 15:45
  Speaker 2: So it's just going from a fixed point to a floating point without reducing the size.
• 15:51
  Speaker 2: Going from floating point to fixed point without reducing the size of the data, we can get a significant savings in our multiplier area.
• 15:59
  Speaker 2: But the multipliers are proportional to the squares of their input, and so if we can shrink down the operands we can really get some big savings on the overall size of our operators in the hardware.
• 16:12
  Speaker 2: So a 32-bit floating point multiplier is about 32 times larger than an 8-bit fixed point multiplier.
• 16:20
  Speaker 2: That means we're using less than 3% of the area to do the same computations, and what we find is that we also get a benefit from the data storage and movement and those are going to scale linearly with the size of the operands.
• 16:36
  Speaker 2: Now, when we look at the numeric representation for a fixed point number, that goes from minus 3.4 times 10 to the 38th, up to positive 3.4 times 10 to the 38th.
• 16:47
  Speaker 2: And if we look at machine learning algorithms, we can typically get into 10 to 12 or 14-bit numbers and that stores between positive and negative 64.
• 17:01
  Speaker 2: And what we find is that with floating point there's a huge range of possible numbers and within an inferencing algorithm most of that space is completely unused and so dragging around hardware and data be able to support that range of numbers that's simply not used is just a very wasteful both in terms of area, energy and time.
• 17:26
  Speaker 2: So kind of a visual representation.
• 17:30
  Speaker 2: If we had a single floating point multiplier 32 by 32, and we replace it by 10 by 10 multipliers, we can fit 20 in less than the area that was in that single 32 bit multiplier.
• 17:43
  Speaker 2: And for moving the data we're going to find that we can get three times as much data stored and moved across the bus in the same area that we would get one data value.
• 17:55
  Speaker 2: So, as we approach quantization, there's really two ways that we can go about it.
• 18:00
  Speaker 2: One is called post-training quantization and in post-training quantization.
• 18:04
  Speaker 2: One is called post-training quantization, and in post-training quantization what we're doing is training the network and training and doing the network design as usual, without any change from what we typically would do with a floating point network.
• 18:17
  Speaker 2: And after the training is complete, we're going to take the values for the weights and, as we do in inference, we're going to take the values of the features and we're going to shoehorn them into the fixed point values.
• 18:34
  Speaker 2: And doing this gives us a lot of benefit.
• 18:35
  Speaker 2: But an alternative is to do what's called training aware quantization, and this is where we perform training using both the quantized values and the smaller operators to be able to create the weight set.
• 18:47
  Speaker 2: And the key thing here is that this requires that the quantization and the operators used in the training exactly match the quantization and the operators that are going to be used in inferencing.
• 18:59
  Speaker 2: If we do this, we can get even better design characteristics than we do in post-training quantization.
• 19:06
  Speaker 2: But this involves a lot more work.
• 19:10
  Speaker 2: Also, when we look at quantization, we need to figure out are we going to be doing quantization on a layer by layer basis.
• 19:16
  Speaker 2: Will we create a different quantization for each layer within the network or are we going to quantize the entire network or the different layer types to a common quantization?
• 19:28
  Speaker 2: So if we're going to create a piece of hardware where we've unrolled the hardware, we've taken the algorithm and spread it out over the hardware, we have a physical multiplier for every multiplication operation in the algorithm.
• 19:44
  Speaker 2: So each multiplier is physically instantiated in the hardware.
• 19:50
  Speaker 2: This is where we want to uniquely quantize each layer and this is going to give us the smallest possible design for that type of implementation.
• 20:00
  Speaker 2: If we're going to reuse that hardware, for each layer, between the different layers, we want a common quantization so that we've got one accelerator that's going to work for all the layers.
• 20:13
  Speaker 2: So if we have a large network that's billions of multipliers, we're not ever going to be able to create a piece of hardware that has billions of multipliers in it.
• 20:21
  Speaker 2: We've got to reuse that hardware and there we want to have a common quantization so we have one consistent implementation and that would create the smallest possible design for those larger types of networks.
• 20:38
  Speaker 2: So choosing the right word size within quantization is done by looking at the integer portion and the fractional portion, now the integer portion.
• 20:48
  Speaker 2: Basically, what we're going to do is throw away the high order bits that are constantly zero, and so what we can do is go into the network.
• 20:57
  Speaker 2: We can add some tracing to it.
• 20:59
  Speaker 2: It's going to record the largest value that's in any of the weights or any of the intermediate values and anything larger than that largest value.
• 21:08
  Speaker 2: Those end up being constant zeros within the design.
• 21:12
  Speaker 2: We can just throw those away and that won't have any impact on the design at all.
• 21:17
  Speaker 2: When we look at the fractional portion, what we're going to find is that as we remove fractional bits from the tail end of the representation, we're going to be reducing the precision.
• 21:28
  Speaker 2: And the way that we do that is we simply take off the fractional bits, retrain and look at where the accuracy falls off.
• 21:37
  Speaker 2: So again, we're going to have to do this sensitivity analysis falls off.
• 21:43
  Speaker 2: So again, we're going to have to do this sensitivity analysis, and through the combination of those two approaches we can arrive at the right word size.
• 21:51
  Speaker 2: This is done for a wake word algorithm.
• 21:53
  Speaker 2: We looked at the bit width, from 32 all the way down to two, keeping a balanced integer and fractional bits, and what we see is that as we go from 32 down to 15, or actually 14, we really see very little decrease.
• 22:11
  Speaker 2: It has no impact on the accuracy of the algorithm.
• 22:14
  Speaker 2: And then we fall off here and also again we see, right before it falls off, we get slightly better accuracy, and so this is the area we want to be, and so this is the sensitivity analysis that you'd go through to find the correct implementation.
• 22:32
  Speaker 2: Here, what we did was took a sweep from 16-bit numbers, all the different combinations of integer bits and fractional bits.
• 22:41
  Speaker 2: This was for an MNIST calculation, and the floating point accuracy for this network was 98.05.
• 22:50
  Speaker 2: And what we find is that as we reduce the integer bits, of course the accuracy goes down.
• 22:55
  Speaker 2: As we reduce fractional bits, it goes down as well.
• 22:58
  Speaker 2: And what we find is that the optimal point in terms of bits of word size versus accuracy is here, at six integer bits and four fractional bits, where we lose about 0.3% in terms of accuracy.
• 23:16
  Speaker 2: But we've shrunk down.
• 23:18
  Speaker 2: This is the 98.05 is what we get on a full floating point implementation, and so we can see that we can dramatically shrink the size of the hardware needed to support this inference by going to a 6-4 fixed point number when we do quantized aware training.
• 23:37
  Speaker 2: We actually do much better.
• 23:39
  Speaker 2: This is the same network that we did in a quantized aware training approach and we can actually get down to three bits of implementation to be able to get still an even better accuracy of 97.83.
• 23:56
  Speaker 2: A guy named Claudio Norcoello, who used to work at Google, has done a lot of work on this quantized aware training and generally in his work what he's found is that you're able to reduce the size of the fixed point number down in half when you do quantized aware training.
• 24:18
  Speaker 2: So here he's got.
• 24:19
  Speaker 2: The folks at Google have created a QKARIS, or quantized KARIS, for being able to do the training and analysis within the KARIS environment.
• 24:32
  Speaker 2: Now it is very important to understand that your modeling within the training environment has to exactly match your hardware on a bit-for-bit basis, and this is what the Google guys have done between QKeras and the TPUs that Google produces.
• 24:50
  Speaker 2: When you're creating your own bespoke hardware, qkeras doesn't have enough configurability to create an arbitrary bit of hardware, and so you really can't aggressively quantize using QKeras.
• 25:03
  Speaker 2: This is something you have to kind of put together in your own environment.
• 25:10
  Speaker 2: Now, when we aggressively quantize these numbers, there's a couple of things that we need to watch out for.
• 25:16
  Speaker 2: One of them is overflow.
• 25:18
  Speaker 2: So when we're working with floating point numbers in a neural network setting, there's really no need to worry about checking for overflow or worrying about overflow.
• 25:28
  Speaker 2: In the algorithm, the numbers usually are ranging between zero minus one and one, especially if we're normalizing between layers, and so the probability of going greater than 10 to the 38th power is effectively zero.
• 25:43
  Speaker 2: But when we're working with aggressively quantized fixed point numbers, we're going to be shaving down the size of that representation to the point where, it is much more likely, we're going to overflow.
• 25:56
  Speaker 2: And once an overflow takes place, all of our subsequent operations are incorrect.
• 26:00
  Speaker 2: We need to throw them out.
• 26:01
  Speaker 2: Once an overflow takes place, all of our subsequent operations are incorrect, we need to throw them out.
• 26:04
  Speaker 2: And the way that we can mitigate this is with something called saturating map.
• 26:08
  Speaker 2: That is, when the computation exceeds the value that can be held by the representation, we just go to the biggest possible value and it turns out that in inferencing algorithms, when a number is large we really don't need to be very precise on it.
• 26:24
  Speaker 2: We need to be precise right around zero.
• 26:26
  Speaker 2: But if a number is 100 or 110, and we misrepresent it by a few places, it is not going to have much of an impact on the algorithm.
• 26:37
  Speaker 2: So by using saturating operators we can usually take one or two more bits off the representation of our integer portion of these numbers.
• 26:49
  Speaker 2: This is a sensitivity analysis we did and here we can see that with the fixed point numbers we drop off here and our accuracy falls away rather dramatically.
• 26:59
  Speaker 2: Here and our accuracy falls away rather dramatically.
• 27:01
  Speaker 2: Where we use saturating numbers, we can have almost no loss of accuracy and shave one more integer bit off the implementation.
• 27:14
  Speaker 2: Now on the other side, on the fractional side, we've got a rounding problem.
• 27:16
  Speaker 2: So again, when working with floating point numbers, the rounding is going to be such a small error that we really don't have to worry about.
• 27:27
  Speaker 2: And when we go to integer math, which is typically what's used with fixed point, the integer math adder is usually just truncate.
• 27:35
  Speaker 2: The hardware is smaller and it's simpler.
• 27:39
  Speaker 2: So you know that in the C programming language, if we take 21 divided by 11, we're going to get one which is incorrect and in fact it's pretty far in error.
• 27:51
  Speaker 2: If we did 21.0 over 11.0, we get 1.9.
• 27:55
  Speaker 2: So as we aggressively quantize these fixed point numbers and take off the fractional bit, the error that we're going to get from truncation gets larger and larger.
• 28:05
  Speaker 2: And as we aggregate those over multiple operations, like we commonly see in a vector matrix multiplier conclusions, we're going to see a compounding of that error.
• 28:16
  Speaker 2: And so by introducing a rounding model we get much better accuracy and we're going to be able to shrink the size of the representation even further.
• 28:27
  Speaker 2: Here we've got a model that we tested this is again the weight word and we look at the model accuracy using a rounding or truncating map or truncating now and what we find is that we actually get a rather pronounced increase in accuracy as we get close to falling off the waterfall here.
• 28:49
  Speaker 2: But we can see that the rounding algorithm lets us take a couple more bits off of the low end of our representation.
• 28:59
  Speaker 2: So to make ML tiny, what we've talked about so far is we aggressively prune that model.
• 29:05
  Speaker 2: If it's a large network, we want to retain the regular memory access.
• 29:09
  Speaker 2: If it's a small network, we want to remove filters and weights as much as possible, even if we end up with irregular accesses.
• 29:17
  Speaker 2: Then we want to convert to fixed point and quantize that.
• 29:21
  Speaker 2: We're going to remove the leading unused bits.
• 29:24
  Speaker 2: We're going to remove the low order bits and then we're going to introduce saturating, math and rounding, and this enables us to get the most optimal and the smallest implementation for our accelerators.
• 29:38
  Speaker 2: But the question is how do we then check the accuracy?
• 29:42
  Speaker 2: Remember that in order to do the training and in order to validate the accuracy of this network, we've got to have a bit accurate representation of all the operators and all the operands that are being used, and that has to include the saturation and the rounding methods that we've included in the hardware implementation that we're going to go build.
• 30:02
  Speaker 2: Build Now.
• 30:03
  Speaker 2: Most machine learning frameworks only support 32-bit floating point.
• 30:07
  Speaker 2: We're seeing a few that are adding quantized inputs, but these are usually targeted at specific TPUs.
• 30:16
  Speaker 2: We want something targeted at our specific implementation.
• 30:21
  Speaker 2: Now we could build this in Verilog, but Verilog is.
• 30:25
  Speaker 2: It takes a really long time to code up a machine learning accelerator in Verilog and also the runtimes are impractically slow.
• 30:34
  Speaker 2: We did an RTL implementation of a YOLO tiny object recognition algorithm fully in Verilog.
• 30:41
  Speaker 2: It took five hours to do one inference.
• 30:43
  Speaker 2: You might need to do 10,000 inferences to figure out the accuracy.
• 30:48
  Speaker 2: That's too much time.
• 30:51
  Speaker 2: Another option is potentially to go run on an FPGA board, but that also has long development times associated with it.
• 31:00
  Speaker 2: So what we do is use a bit accurate C data type which has support for arbitrarily sized floats and fixed point numbers, and this is usable in a C++ or system C environment.
• 31:13
  Speaker 2: It's available at this website here, http//hlslibsorg, and these have bit accurate operators with user definable saturating and rounding algorithms for all of those operations and this enables a bit accurate running and validation of that network architecture, including the quantization, the operators and the effects of rounding and saturation.
• 31:38
  Speaker 2: This is quick to write.
• 31:40
  Speaker 2: It's a much higher level of abstraction than Verilog and it runs about a thousand times faster than Verilog.
• 31:46
  Speaker 2: So this is something where we can run lots of inferences, prove the correctness and then perhaps readjust the network.
• 31:55
  Speaker 2: Now, once we've created that implementation of the algorithm with the right quantization and the right network architecture for all the different layers, we now have to translate this into RTL.
• 32:09
  Speaker 2: Of course, one option is to go ahead and use handcrafted Verilog.
• 32:15
  Speaker 2: If you really enjoy writing Verilog, knock yourself out.
• 32:19
  Speaker 2: High-level synthesis is a tool that we work on that can create a synthesizable RTL description from an algorithm that's written with those AC data types that I talked about on the last slide, and so the developer would then define, as we migrate from that algorithm into RTL, how much parallelism we introduce, we can bring in caching, we can define the data movement and so forth, the order of the operations.
• 32:50
  Speaker 2: All of these impact how well our accelerator works.
• 32:54
  Speaker 2: And not only do we have the ability to create this RTL implementation, but the HLS tool can give you estimates for performance and area before we go through RTL synthesis, and we also have the ability, within that high-level synthesis tool, to validate that the RTL that's generated through that synthesis process matches the C++, where we've created that algorithm and validated everything.
• 33:23
  Speaker 2: So we take that validated algorithm and we can prove the correctness between the C++ and the RTL, which is an important step that we need to go through.
• 33:34
  Speaker 2: So, fundamentally, we've got this cool tool, high-level synthesis algorithm that takes that C++ or system C that we've got for the representation of the algorithm and it creates the full RTL implementation that we've got.
• 33:50
  Speaker 2: So our tiny AI design flow starts off in a traditional machine learning framework TensorFlow Cafe and we create a neural network architecture this is going to be Python, tensorflow, keras, what have you and we're then going to take that and create this C++ bit accurate algorithm to be able to create the smallest implementation that we've got.
• 34:20
  Speaker 2: And here's where we introduce the quantization.
• 34:23
  Speaker 2: And as we do that, we might realize oh, we need a couple more channels, or we can do with a few, we can do with fewer channels in a particular layer, and so we might be going back to our machine learning framework changing the network.
• 34:39
  Speaker 2: So there's a cycle here where we've got a feedback loop of what we're doing in the bit accurate implementation and quantization to the network architecture itself, in the data movement, the caching, the order of operations, and look at the impact on the throughput and the area for the implementation.
• 35:08
  Speaker 2: And again, at this point we might want to go back to the machine learning framework and change the architecture and once we've been through this cycle, we then get to a point where we can take it through our high-level synthesis tool and build the synthesizable RTL for that implementation and then we can take that into the full backend process targeting an ASIC or FPGA.
• 35:34
  Speaker 2: So that's just standard hardware development tools.
• 35:37
  Speaker 2: But of course I know you guys all want to go from Python directly into RTL and this is something that we're working on.
• 35:46
  Speaker 2: So there's a facility called HLS4ML and this is you can find this at fastmachinelearningorg HLS4ML and this is something that's actually going to convert the Python, cnn or DNN into C++.
• 36:02
  Speaker 2: That is specifically suitable for running through high-level synthesis in an automated flow.
• 36:09
  Speaker 2: This was originally targeted at FPGA, primarily using the Xilinx devices, but Siemens EDA, within the high-level synthesis group, has extended this capability to be able to support an ASIC flow through catapult high-level synthesis, and this can be done either through either targeting an ASIC or FPGA.
• 36:31
  Speaker 2: This is something that's in production at Fermilabs.
• 36:34
  Speaker 2: We've been working with those folks and it creates the folks at Fermilab are using it to create a very high speed inference that's used to identify subatomic particles that are exiting a nuclear accelerator in real time.
• 36:52
  Speaker 2: So they need to have an extremely fast inference.
• 36:54
  Speaker 2: So they take the neural network, they completely unroll it into FPGA fabric.
• 37:00
  Speaker 2: Every individual multiplier has a multiplier represented in the hardware and they take their features off of the output of the nuclear accelerator.
• 37:13
  Speaker 2: They quickly figure out do they have the subatomic particle they're looking for?
• 37:16
  Speaker 2: If so, they save all that data.
• 37:19
  Speaker 2: If not, they don't need to save it, and it saves them a bunch of time.
• 37:23
  Speaker 2: Really interesting approach.
• 37:27
  Speaker 2: And so, just to finish up, I want to talk about an example implementation.
• 37:33
  Speaker 2: This was presented at DAC a couple of years ago.
• 37:38
  Speaker 2: This was a joint project between Siemens, eda, columbia University and Global Foundries.
• 37:43
  Speaker 2: The metrics that we're looking at here we're targeting a 12 nanometer low power ASIC library with a target frequency of 500 megahertz, and this is post-placing route for the power numbers.
• 37:56
  Speaker 2: What we did was started with a software floating point configuration.
• 38:03
  Speaker 2: This was running on a rocket core, took a little over a quarter of a second to be able to perform this wake word analysis and used an enormous amount of energy about 90% accurate on hitting the wake word.
• 38:17
  Speaker 2: And so we took three different blocks of this WakeWord implementation and moved them into bespoke hardware accelerators one for audio preprocessing, one for the convolution and one for the vector matrix.
• 38:35
  Speaker 2: Multiply the dense layers, each of these, once we moved them into hardware, dramatically faster than they were running in the original software.
• 38:43
  Speaker 2: Power significantly better, energy consumption per inference dramatically reduced, and the accuracy is still remaining quite high.
• 38:56
  Speaker 2: The bits of the inference that we left in software total performance of 18 milliseconds versus the 284 in software.
• 39:08
  Speaker 2: Now, within the high-level synthesis environment, we've got the capability of taking the algorithm as we've described it in C++ and targeting it to different architectures.
• 39:21
  Speaker 2: And so within the tool, the high-level synthesis tool, we can identify that we would like to introduce more parallelism or introduce more efficiency and create a smaller circuit.
• 39:33
  Speaker 2: And so for this particular convolution, the easy design points were 1 multiplier, 7 multipliers, 20 multipliers or 140.
• 39:44
  Speaker 2: And so we could create each one of these, not by going in and changing that original C++ implementation, but by changing the settings on the high-level synthesis tool and very small changes to pound define or sharp defines in the C++ code.
• 40:03
  Speaker 2: And so, again in the software implementation, about 60 milliseconds, and we had a range of implementations that we could select from, running from five milliseconds down to about 0.2 milliseconds, with corresponding areas getting larger each time.
• 40:24
  Speaker 2: So we ended up, for our final implementation, picking the seven multipliers pretty close to the smallest implementation and used that in the overall acceleration of that way.
• 40:39
  Speaker 2: So, to wrap things up, making ML tiny with high-level synthesis, the high-level synthesis algorithmic C data types give you a way to model and validate those deep neural networks and convolutional neural networks that are aggressively quantized and pruned, and it allows you to include the effects of grounding and saturation.
• 41:03
  Speaker 2: And it's a practical way to create a bit-level, accurate model that's going to match the hardware that you're ultimately going to implement, and by doing this you've got the ability to try out a number of different architectures and be able to validate those without going all the way down into a Verilog implementation.
• 41:27
  Speaker 2: The high-level synthesis tool can then convert that implementation into the synthesizable RTL, and we do this targeting a very specific ASIC or FPGA technology, specific ASIC or FPGA technology.
• 41:44
  Speaker 2: So the high-level synthesis tool not only takes in your algorithm as input, but also what ASIC or FPGA are you targeting, what frequency are you trying to reach, and it will build the entire RTL implementation specifically to target that, and so it closes timing really nicely Again, because we're doing this at a high level where we can change the implementations quickly.
• 42:09
  Speaker 2: We can look at a variety of different implementations, understand their suitability, how well do they work, how large are they.
• 42:16
  Speaker 2: We can also validate that the ultimate RTL implementation fully matches the C++ implementation, which fully matches our original Python.
• 42:29
  Speaker 2: And the high-level synthesis tool does give us links to downstream synthesis and power analysis tools so that we can figure out not only how big is this thing, how much power will it consume once it's through, put going to be all before we've gone and gone to complete heart.
• 42:47
  Speaker 2: All right, and with that I'll take questions, comments, discussions.
• 42:57
  Speaker 1: Thank you so much, Marcel, for a very interesting presentation.
• 43:00
  Speaker 1: There are a couple of questions waiting for your replies in the chat.
• 43:06
  Speaker 1: The first one.
• 43:08
  Speaker 1: Others are appearing, so let's start with the first one.
• 43:12
  Speaker 1: Does HLS Lib work on Vivado HLS, since the latter is used a lot for FPGA development?
• 43:22
  Speaker 2: since the latter is used a lot for FPGA development, vivado HLS has a corresponding.
• 43:30
  Speaker 2: Ac data type.
• 43:31
  Speaker 2: They're slightly different but it's very similar, so moving between the two is not particularly challenging.
• 43:38
  Speaker 2: We've been actually working with Xilinx to try and standardize this, so everybody's using exactly the same format.
• 43:46
  Speaker 2: But right now there's just a little bit of transition work that you might need to do going between those two environments, but they're really analogous between the two implementations.
• 43:59
  Speaker 1: There is another question.
• 44:00
  Speaker 1: Maybe Matilde is typing an answer.
• 44:04
  Speaker 1: Essentially, the question was about the Python to RTL tool.
• 44:11
  Speaker 1: Luca or Binati would like to get more details about what are the differences with HLS-M.
• 44:32
  Speaker 2: So HLS for ML.
• 44:32
  Speaker 2: It goes through a step that isn't covered by our high-level synthesis tool.
• 44:34
  Speaker 2: So our high-level synthesis tool is going to be taking C++ and turning it into RTL.
• 44:40
  Speaker 2: In the machine learning world we of course start with Python, and so HLS for ML is going to do a conversion between the Python and the C++.
• 44:52
  Speaker 2: So it does that step that you might go through manually.
• 44:56
  Speaker 2: That I kind of described in the talk, and so, yes, we already have done a bunch of work to integrate with it so that you can use HLS for ML to convert your Python and move that into C++, and the C++ is fully suitable for going into catapult, high-level synthesis and then targeting the downstream devices.
• 45:26
  Speaker 2: So it is doing a step prior to high-level synthesis, moving that neural network from the Python environment to C++.
• 45:35
  Speaker 2: And take a look at their website.
• 45:38
  Speaker 2: They've got a lot of good information there, or reach out to me.
• 45:41
  Speaker 2: I'd be happy to go into a bunch more detail on this, but we have integrated Catapult as one of the high-level synthesis tools that HLS4ML does support.
• 45:54
  Speaker 2: We've done some direct work on that and contributed to the project.
• 45:58
  Speaker 3: Exactly, and we just released today Catapult AI NN that does just that.
• 46:03
  Speaker 3: So I added some links to the press release as well as the landing page for that tool.
• 46:10
  Speaker 3: There for you, thank you.
• 46:15
  Speaker 1: Thank you, Russell.
• 46:16
  Speaker 1: Thank you, Matilde.
• 46:17
  Speaker 1: Another question Can you tell us more about the Catapult architecture?
• 46:23
  Speaker 1: It's available for open use.
• 46:25
  Speaker 1: What kind of throughput performances are you able to get running on different devices?
• 46:31
  Speaker 1: How scalable is the architecture to be able to trade off size versus speed?
• 46:39
  Speaker 2: Yeah, so Catapult is a commercial tool.
• 46:41
  Speaker 2: It's not open source.
• 46:44
  Speaker 2: We do make it available to universities through our university program and for research.
• 46:51
  Speaker 2: So if you're involved in that type of thing, we can make it available to folks.
• 46:56
  Speaker 2: If you're doing commercial development for it, it's a commercial product available for sale.
• 47:01
  Speaker 2: Development for it it's a commercial product, available for sale.
• 47:06
  Speaker 2: In terms of performance, it really depends on the algorithm that you're taking into it.
• 47:09
  Speaker 2: I mean, it's a compiler, right?
• 47:11
  Speaker 2: So you're sort of asking what's the performance of GCC on C code?
• 47:14
  Speaker 2: It just completely depends on the algorithm that you're trying to get into it.
• 47:21
  Speaker 2: In terms of scaling, the architecture we can build, it's a commercial tool.
• 47:26
  Speaker 2: It's been around for about 20 years now.
• 47:30
  Speaker 2: We've made a lot of advancements on it.
• 47:32
  Speaker 2: It is being used in a lot of commercial settings.
• 47:35
  Speaker 2: Nvidia actually uses this in the creation of their AI accelerators.
• 47:43
  Speaker 2: They did a DAC talk back in 2017.
• 47:46
  Speaker 2: And that's exactly what they use it for is being able to trade off size versus speed and optimizing the algorithm.
• 47:54
  Speaker 2: And that's something that's really challenging to do within an RTL environment, because there's so much detail that you need to put into creating an implementation in RTL that it takes weeks or even months to create a meaningfully different architecture by moving up to a higher level of abstraction.
• 48:15
  Speaker 2: High-level synthesis lets you make just very tiny changes and you can end up with really different architectures and you can exercise those and analyze them and figure out how they're going to work.
• 48:27
  Speaker 2: So high-level synthesis has been around for a long time.
• 48:30
  Speaker 2: It's a fairly mature technology.
• 48:33
  Speaker 2: It is being used in commercial and industrial settings to create very significant pieces of hardware by lots of big companies that you've heard of.
• 48:47
  Speaker 1: Thank you so much.
• 48:48
  Speaker 1: Another question is there any ML computation integrated with Catapult HLS flow?
• 48:59
  Speaker 2: So Catapult does have the ability to take any algorithm that you that you might like to use.
• 49:06
  Speaker 2: It could.
• 49:07
  Speaker 2: Them is an implementation of Yolo Tiny, which is an object recognition algorithm, and so there's a complete implementation of that included.
• 49:34
  Speaker 2: As an example, going to have an MNIST design kit that goes into all kinds of detail about taking the MNIST all the way from Python, configuring it various ways, doing the quantization.
• 49:49
  Speaker 2: So the whole step that I went through this is going to be available with the product shortly and then, as Mathilde mentioned, we've got our Catapult NN that's available now and that is specifically targeted at machine learning algorithms.
• 50:07
  Speaker 2: So that's the type of support that we're bringing into the mix.
• 50:17
  Speaker 1: Thank you to share reference resources links so that the people can learn more about the topic that you described Before the end of this session.
• 50:32
  Speaker 1: Since I don't see any other questions in the Q&A chat, maybe I would like myself to pose some questions.
• 50:43
  Speaker 1: For example, you mentioned clearly in the design flow the need for quantization-aware training and the need to investigate different bid-dap compromises depending on the implementation cost.
• 51:02
  Speaker 1: So, this is a very interesting when it comes to training neural networks with respect to traditional training or of a floating point network.
• 51:14
  Speaker 1: Can you share some insights or tips or curious experiences you had when managing such type of networks, so the deeply quantized one?
• 51:32
  Speaker 1: I'm not quite sure I understand the question, so the question is essentially, you know, when we train a neural network which is a full floating point, there are, for example, knowledge distillation.
• 51:51
  Speaker 1: It may apply, but sometime, not strictly, when designing and training neural network which are deeply quantized In your experience.
• 52:05
  Speaker 1: Are there tricks that one should consider when going through the methodology that you explained, or the same methodology to design neural networks in floating point applies.
• 52:21
  Speaker 1: So there are no particular tricks or advices.
• 52:25
  Speaker 2: Yeah, there's no particular tricks.
• 52:27
  Speaker 2: There's nothing different that we do in training quantized aware.
• 52:33
  Speaker 2: The only thing that what makes it really challenging is that this is not yet supported by the machine learning frameworks.
• 52:42
  Speaker 2: So within a machine learning framework, all you're going to do is floating point 32.
• 52:49
  Speaker 2: If you use QKeras or some of the other quantized packages, you can get 8-bit quantization or whatever they've introduced, but you really don't have the ability to specify the size of the intermediate values, the size of the operators, the rounding model that you're going to use, whether you're going to use saturation or not.
• 53:16
  Speaker 2: All of those types of details just aren't programmable in the machine learning framework quantization packages, and I know there's five or six different ones out there.
• 53:27
  Speaker 2: Again, those are all targeted towards specific TPU implementations.
• 53:33
  Speaker 2: So the real challenge on doing quantized aware training is that you need to kind of build this, build the entire infrastructure in C++ using the AC data types, and that's not a trivial exercise, right, that is, you're kind of rebuilding things that people have built.
• 53:52
  Speaker 2: But the training exercise itself is going to be extremely similar to what you do in the floating point level.
• 53:59
  Speaker 2: You're doing the same tasks, you're looking for the same tasks, you're looking for the same results.
• 54:09
  Speaker 2: The same tricks that apply at the floating point level would apply in the quantized level.
• 54:16
  Speaker 2: It's really exactly the same math that you're doing throughout that process.
• 54:22
  Speaker 2: Like I said, the challenge is we don't have a really flexible Python environment for being able to do that today.
• 54:29
  Speaker 2: Doing the post training quantization is dead easy.
• 54:33
  Speaker 2: Right, this is very simple and you know there.
• 54:37
  Speaker 2: You just do all of your training in the framework as usual and then take those numbers and scrunch them down.
• 54:42
  Speaker 2: But you get much, much better results if you can do the quantized aware training.
• 54:51
  Speaker 1: Very good, there is a very last minute question from an attendee.
• 54:56
  Speaker 1: Is there any other model optimization methods that you would like to mention or underline?
• 55:06
  Speaker 2: Just in terms of time.
• 55:08
  Speaker 2: We didn't talk very much about data movement optimizations and so, in addition to shrinking things down, the ability to use the data most efficiently as you read it in from the outside world whether you're getting it from a weight database or the input of features, but moving that data around is extremely important in terms of the efficiency of that network and the performance of it.
• 55:44
  Speaker 2: So being able to cache that data and make sure that you're able to fully utilize it before discarding it is very important, and there's an entire sort of analysis that you go through of you know when are we using these particular data elements, how long do they have to live within the accelerator, we using these particular data elements, how long do they have to live within the accelerator, and how can I most efficiently use those together?
• 56:07
  Speaker 2: So, data movement and data caching we didn't go into a lot of detail, but there's a really rich set of optimization that can be done there that has a really big impact on the performance and the size of the ultimate accelerator implementation impact on the performance and the size of the ultimate accelerator implementation Right right.
• 56:32
  Speaker 2: And these in this technique have also a strong impact on power consumption.
• 56:35
  Speaker 2: Yes, especially if you're going off chip for your weights, if you've got them stored on chip, that can help quite a bit.
• 56:41
  Speaker 2: If you've got to go off chip, an extremely large amount of energy to go get that.
• 56:45
  Speaker 2: So you don't want to repetitively get that data.
• 56:48
  Speaker 2: You want to pull it in once and fully utilize it, cache it very efficiently.
• 56:53
  Speaker 2: You want to pay very close attention to that.
• 56:56
  Speaker 1: Are you performing any preliminary even preliminary study on the applicability of such techniques to tiny generative AI or models which are in the range of 100 to few billions parameters?
• 57:14
  Speaker 2: We have customers that are looking at that.
• 57:16
  Speaker 2: I haven't looked into that.
• 57:18
  Speaker 2: As you go to the larger networks, of course you're going to have to reuse the hardware quite a bit, and so keeping regular data accesses is generally really important in that.
• 57:30
  Speaker 2: And then introducing a lot of parallelism, because as you get to those very large numbers, you don't want four multipliers or seven multipliers, you want hundreds of multipliers and you want to be able to effectively use all of those multipliers and bring in as much parallelism as possible, because in order to get that throughput up, you want to fully take advantage of that parallelism.
• 57:56
  Speaker 2: It's extremely important.
• 57:59
  Speaker 1: Thank you very much, russell, for your availability also to address all the questions that have been posed so far and for your great presentation.
• 58:10
  Speaker 1: Thanks to Matilda also for her comments during the QA session.
• 58:15
  Speaker 1: So, as a reminder, tinyml talk series is possible, as well as the other great events organized by TinyML Foundation, thanks to the huge support from the strategic partners.
• 58:31
  Speaker 1: So let me go through the list, the executive strategic partners, which are Qualcomm, syntient.
• 58:41
  Speaker 1: The Platinum strategic partners, embedur, sony.
• 58:51
  Speaker 1: The Gold strategy partners, arm Edge, impulse, infineon, renesas, stmicroelectronics.
• 59:05
  Speaker 1: And the Silver Strategic Partners, aiz, parduino, brainchip, efficient, greenwaves Technologies, gravity, imax, imagebob, innatera, nottaai, nxp, p&g, corvo, snyder, sensimil, silicon Labs, synaptics, tdk.
• 59:23
  Speaker 1: Thanks again, russell, and see you all to the next episode of TinyML Talks series.
• 59:34
  Speaker 1: Have a great day.
• 59:36
  Speaker 1: Bye.